perlhack - How to hack at the Perl internals
DESCRIPTION
This document attempts to explain how Perl development
takes place, and ends with some suggestions for people
wanting to become bona fide porters.
The perl5-porters mailing list is where the Perl standard
distribution is maintained and developed. The list can
get anywhere from 10 to 150 messages a day, depending on
the heatedness of the debate. Most days there are two or
three patches, extensions, features, or bugs being dis
cussed at a time.
A searchable archive of the list is at:
http://www.xray.mpe.mpg.de/mailing-lists/perl5-porters/
The list is also archived under the usenet group name
`perl.porters-gw' at:
http://www.deja.com/
List subscribers (the porters themselves) come in several
flavours. Some are quiet curious lurkers, who rarely
pitch in and instead watch the ongoing development to
ensure they're forewarned of new changes or features in
Perl. Some are representatives of vendors, who are there
to make sure that Perl continues to compile and work on
their platforms. Some patch any reported bug that they
know how to fix, some are actively patching their pet area
(threads, Win32, the regexp engine), while others seem to
do nothing but complain. In other words, it's your usual
mix of technical people.
Over this group of porters presides Larry Wall. He has
the final word in what does and does not change in the
Perl language. Various releases of Perl are shepherded by
a ``pumpking'', a porter responsible for gathering
patches, deciding on a patch-by-patch feature-by-feature
basis what will and will not go into the release. For
instance, Gurusamy Sarathy is the pumpking for the 5.6
release of Perl.
In addition, various people are pumpkings for different
things. For instance, Andy Dougherty and Jarkko
Hietaniemi share the Configure pumpkin, and Tom Chris
tiansen is the documentation pumpking.
Larry sees Perl development along the lines of the US gov
ernment: there's the Legislature (the porters), the Execu
tive branch (the pumpkings), and the Supreme Court
(Larry). The legislature can discuss and submit patches
branch is free to veto them. Rarely, the Supreme Court
will side with the executive branch over the legislature,
or the legislature over the executive branch. Mostly,
however, the legislature and the executive branch are sup
posed to get along and work out their differences without
impeachment or court cases.
You might sometimes see reference to Rule 1 and Rule 2.
Larry's power as Supreme Court is expressed in The Rules:
1 Larry is always by definition right about how Perl
should behave. This means he has final veto power on
the core functionality.
2 Larry is allowed to change his mind about any matter
at a later date, regardless of whether he previously
invoked Rule 1.
Got that? Larry is always right, even when he was wrong.
It's rare to see either Rule exercised, but they are often
alluded to.
New features and extensions to the language are con
tentious, because the criteria used by the pumpkings,
Larry, and other porters to decide which features should
be implemented and incorporated are not codified in a few
small design goals as with some other languages. Instead,
the heuristics are flexible and often difficult to fathom.
Here is one person's list, roughly in decreasing order of
importance, of heuristics that new features have to be
weighed against:
Does concept match the general goals of Perl?
These haven't been written anywhere in stone, but one
approximation is:
1. Keep it fast, simple, and useful.
2. Keep features/concepts as orthogonal as possible.
3. No arbitrary limits (platforms, data sizes, cultures).
4. Keep it open and exciting to use/patch/advocate Perl everywhere.
5. Either assimilate new technologies, or build bridges to them.
Where is the implementation?
All the talk in the world is useless without an imple
mentation. In almost every case, the person or people
who argue for a new feature will be expected to be the
ones who implement it. Porters capable of coding new
features have their own agendas, and are not available
to implement your (possibly good) idea.
Backwards compatibility
It's a cardinal sin to break existing Perl programs.
that emits warnings is not broken, while others say it
is. Adding keywords has the potential to break pro
grams, changing the meaning of existing token
sequences or functions might break programs.
Could it be a module instead?
Perl 5 has extension mechanisms, modules and XS,
specifically to avoid the need to keep changing the
Perl interpreter. You can write modules that export
functions, you can give those functions prototypes so
they can be called like built-in functions, you can
even write XS code to mess with the runtime data
structures of the Perl interpreter if you want to
implement really complicated things. If it can be
done in a module instead of in the core, it's highly
unlikely to be added.
Is the feature generic enough?
Is this something that only the submitter wants added
to the language, or would it be broadly useful? Some
times, instead of adding a feature with a tight focus,
the porters might decide to wait until someone imple
ments the more generalized feature. For instance,
instead of implementing a ``delayed evaluation'' fea
ture, the porters are waiting for a macro system that
would permit delayed evaluation and much more.
Does it potentially introduce new bugs?
Radical rewrites of large chunks of the Perl inter
preter have the potential to introduce new bugs. The
smaller and more localized the change, the better.
Does it preclude other desirable features?
A patch is likely to be rejected if it closes off
future avenues of development. For instance, a patch
that placed a true and final interpretation on proto
types is likely to be rejected because there are still
options for the future of prototypes that haven't been
addressed.
Is the implementation robust?
Good patches (tight code, complete, correct) stand
more chance of going in. Sloppy or incorrect patches
might be placed on the back burner until the pumpking
has time to fix, or might be discarded altogether
without further notice.
Is the implementation generic enough to be portable?
The worst patches make use of a system-specific fea
tures. It's highly unlikely that nonportable addi
tions to the Perl language will be accepted.
Patches without documentation are probably ill-thought
out or incomplete. Nothing can be added without docu
mentation, so submitting a patch for the appropriate
manpages as well as the source code is always a good
idea. If appropriate, patches should add to the test
suite as well.
Is there another way to do it?
Larry said ``Although the Perl Slogan is There's More
Than One Way to Do It, I hesitate to make 10 ways to
do something''. This is a tricky heuristic to navi
gate, though--one man's essential addition is another
man's pointless cruft.
Does it create too much work?
Work for the pumpking, work for Perl programmers, work
for module authors, ... Perl is supposed to be easy.
Patches speak louder than words
Working code is always preferred to pie-in-the-sky
ideas. A patch to add a feature stands a much higher
chance of making it to the language than does a random
feature request, no matter how fervently argued the
request might be. This ties into ``Will it be use
ful?'', as the fact that someone took the time to make
the patch demonstrates a strong desire for the fea
ture.
If you're on the list, you might hear the word ``core''
bandied around. It refers to the standard distribution.
``Hacking on the core'' means you're changing the C source
code to the Perl interpreter. ``A core module'' is one
that ships with Perl.
The source code to the Perl interpreter, in its different
versions, is kept in a repository managed by a revision
control system (which is currently the Perforce program,
see http://perforce.com/). The pumpkings and a few others
have access to the repository to check in changes. Peri
odically the pumpking for the development version of Perl
will release a new version, so the rest of the porters can
see what's changed. The current state of the main trunk
of repository, and patches that describe the individual
changes that have happened since the last public release
are available at this location:
ftp://ftp.linux.activestate.com/pub/staff/gsar/APC/
Selective parts are also visible via the rsync protocol.
To get all the individual changes to the mainline since
the last development release, use the following command:
rsync -avuz rsync://ftp.linux.activestate.com/perl-diffs perl-diffs
rsync -avuz rsync://ftp.linux.activestate.com/perl-current perl-current
Needless to say, the source code in perl-current is usu
ally in a perpetual state of evolution. You should expect
it to be very buggy. Do not use it for any purpose other
than testing and development.
Always submit patches to perl5-porters@perl.org. This
lets other porters review your patch, which catches a sur
prising number of errors in patches. Either use the diff
program (available in source code form from
ftp://ftp.gnu.org/pub/gnu/), or use Johan Vromans'
makepatch (available from CPAN/authors/id/JV/). Unified
diffs are preferred, but context diffs are accepted. Do
not send RCS-style diffs or diffs without context lines.
More information is given in the Porting/patching.pod file
in the Perl source distribution. Please patch against the
latest development version (e.g., if you're fixing a bug
in the 5.005 track, patch against the latest 5.005_5x ver
sion). Only patches that survive the heat of the develop
ment branch get applied to maintenance versions.
Your patch should update the documentation and test suite.
To report a bug in Perl, use the program perlbug which
comes with Perl (if you can't get Perl to work, send mail
to the address perlbug@perl.com or perlbug@perl.org).
Reporting bugs through perlbug feeds into the automated
bug-tracking system, access to which is provided through
the web at http://bugs.perl.org/. It often pays to check
the archives of the perl5-porters mailing list to see
whether the bug you're reporting has been reported before,
and if so whether it was considered a bug. See above for
the location of the searchable archives.
The CPAN testers (http://testers.cpan.org/) are a group of
volunteers who test CPAN modules on a variety of plat
forms. Perl Labs (http://labs.perl.org/) automatically
tests Perl source releases on platforms and gives feedback
to the CPAN testers mailing list. Both efforts welcome
volunteers.
To become an active and patching Perl porter, you'll need
to learn how Perl works on the inside. Chip Salzenberg, a
pumpking, has written articles on Perl internals for The
Perl Journal (http://www.tpj.com/) which explain how vari
ous parts of the Perl interpreter work. The `perlguts'
manpage explains the internal data structures. And, of
course, the C source code (sometimes sparsely commented,
sometimes commented well) is a great place to start (begin
with `perl.c' and see where it goes from there). A lot of
the style of the Perl source is explained in the
It is essential that you be comfortable using a good
debugger (e.g. gdb, dbx) before you can patch perl. Step
ping through perl as it executes a script is perhaps the
best (if sometimes tedious) way to gain a precise under
standing of the overall architecture of the language.
If you build a version of the Perl interpreter with `-DDE
BUGGING', Perl's -D command line flag will cause copious
debugging information to be emitted (see the `perlrun'
manpage). If you build a version of Perl with compiler
debugging information (e.g. with the C compiler's `-g'
option instead of `-O') then you can step through the exe
cution of the interpreter with your favourite C symbolic
debugger, setting breakpoints on particular functions.
It's a good idea to read and lurk for a while before chip
ping in. That way you'll get to see the dynamic of the
conversations, learn the personalities of the players, and
hopefully be better prepared to make a useful contribution
when do you speak up.
If after all this you still think you want to join the
perl5-porters mailing list, send mail to perl5-porters-
subscribe@perl.org. To unsubscribe, send mail to
perl5-porters-unsubscribe@perl.org.
AUTHOR
This document was written by Nathan Torkington, and is
maintained by the perl5-porters mailing list.
Man(1) output converted with
man2html