238 lines
16 KiB
Plaintext
238 lines
16 KiB
Plaintext
Hacking and Refactoring
|
|
<p>In 2001, there was a history-making conference of software-engineering<br />
|
|
thinkers in Snowbird, Colorado. The product of that meeting was a remarkable<br />
|
|
document called the <a href="http://agilemanifesto.org/">Agile Manifesto</a>,<br />
|
|
a call to overturn many of the assumptions of traditional software development.<br />
|
|
I was invited to be at Snowbird, but couldn’t make it.</p>
|
|
<p>Ever since, though, I’ve been sensing a growing convergence between<br />
|
|
agile programming and the open-source movement. I’ve seen agile<br />
|
|
concepts and terminology being adopted rapidly and enthusiastically by<br />
|
|
my colleagues in open-source-land—especially ideas like<br />
|
|
refactoring, unit testing, and design from stories and personas. From<br />
|
|
the other side, key agile-movement figures like Kent Beck and Martin<br />
|
|
Fowler have expressed strong interest in open source both in published<br />
|
|
works and to me personally. Fowler has gone so far as to <a href="http://www.martinfowler.com/articles/newMethodology.html#N400254">include</a><br />
|
|
open source on his list of agile-movement schools.</p>
|
|
<p>I agree that we belong on that list. But I also agree with<br />
|
|
Fowler’s description of of open source as a style, rather than a<br />
|
|
process. I think his reservations as to whether open source can be<br />
|
|
described as just another agile school are well-founded. There is<br />
|
|
something more complicated and interesting going on here. and I<br />
|
|
realized when I read Fowler’s description of open source that at some<br />
|
|
point I was going to have to do some hard thinking and writing in an<br />
|
|
effort to sort it all out.</p>
|
|
<p>While doing research for my forthcoming book, <a href="http://www.catb.org/esr/writings/taoup/">The Art of Unix<br />
|
|
Programming</a>, I read one particular passage in Fowler’s<br />
|
|
<cite>Refactoring</cite> that finally brought it all home. He<br />
|
|
writes:</p>
|
|
<blockquote>
|
|
<p>One argument is that refactoring can be an alternative to up-front<br />
|
|
design. In this scenario, you don’t do any design at all. You just<br />
|
|
code the first approach that comes into your head, get it working, and<br />
|
|
then refactor it into shape. Actually, this approach can work. I’ve<br />
|
|
seen people do this and come out with a very well-defined piece of<br />
|
|
software. Those who support Extreme Programming often are portrayed<br />
|
|
as advocating this approach.</p>
|
|
</blockquote>
|
|
<p>I read this, and had one of those moments where everything comes<br />
|
|
together in your head with a great ringing crash and the world assumes<br />
|
|
a new shape—a moment not unlike the one I had in late 1996<br />
|
|
when I got the central insight that turned into <cite>The Cathedral<br />
|
|
and the Bazaar</cite>. In the remainder of this essay I’m going to<br />
|
|
try to articulate what I now think I understand about open source,<br />
|
|
agile programming, how they are related, and why the connection should<br />
|
|
be interesting even to programmers with no stake in either movement.</p>
|
|
<p>Now I need to set a little background here, because I’m going<br />
|
|
to need to have to talk about several different categories which are<br />
|
|
contingently but not necessarily related.</p>
|
|
<p>First, there is <em>Unix programmer</em>. Unix is the operating<br />
|
|
system with the longest living tradition of programming and design.<br />
|
|
It has an unusually strong and mature technical culture around it, a<br />
|
|
culture which originated or popularized many of the core ideas and<br />
|
|
tools of modern software design. <cite>The Art of Unix<br />
|
|
Programming</cite> is a concerted attempt to capture the craft wisdom<br />
|
|
of this culture, one to which I have successfully enlisted quite a few<br />
|
|
of its founding elders.</p>
|
|
<p>Second, there is <em>hacker</em>. This is a very complex term, but<br />
|
|
more than anything else, it describes an attitude—an<br />
|
|
intentional stance that relates hackers to programming and other<br />
|
|
disciplines in a particular way. I have described the hacker stance<br />
|
|
and its cultural correlates in detail in <a href="http://www.catb.org/esr/faqs/hacker-howto.html">How To Become A<br />
|
|
Hacker</a>.</p>
|
|
<p>Third, there is <em>open-source programmer</em>. Open source is a<br />
|
|
programming style with strong roots in the Unix tradition and the<br />
|
|
hacker culture. I wrote the modern manifesto for it in 1997, <a href="http://www.catb.org/~esr/writings/cathedral-bazaar/">The<br />
|
|
Cathedral and the Bazaar</a>, building on earlier thinking by<br />
|
|
Richard Stallman and others.</p>
|
|
<p>These three categories are historically closely related. It is<br />
|
|
significant that a single person (accidentally, me) wrote touchstone<br />
|
|
documents for the second and third and is attempting a <em>summum<br />
|
|
bonum</em> of the first. That personal coincidence reflects a larger<br />
|
|
social reality that in 2003 these categories are becoming increasingly<br />
|
|
merged — essentially, the hacker community has become the core<br />
|
|
of the open-source community, which is rapidly re-assimilating the<br />
|
|
parts of the Unix culture that got away from the hackers during<br />
|
|
the ten bad years after the AT&T divestiture in 1984.</p>
|
|
<p>But the relationship is not logically entailed; we can imagine<br />
|
|
a hacker culture speaking a common tongue other than Unix and C (in<br />
|
|
the far past its common tongue was Lisp), and we can imagine an<br />
|
|
explicit ideology of open source developing within a cultural and<br />
|
|
technical context other than Unix (as indeed nearly happened several<br />
|
|
different times).</p>
|
|
<p>With this scene-setting done, I can explain that my first take on<br />
|
|
Fowler’s statement was to think “Dude, you’ve just described<br />
|
|
<em>hacking</em>!”</p>
|
|
<p>I mean something specific and powerful by this. Throwing together<br />
|
|
a prototype and refactoring it into shape is a rather precise<br />
|
|
description of the normal working practice of hackers since that<br />
|
|
culture began to self-define in the 1960s. Not a complete one, but it<br />
|
|
captures the most salient feature of how hackers relate to code. The<br />
|
|
open-source community has inherited and elaborated this practice,<br />
|
|
building on similar tendencies within the Unix tradition.</p>
|
|
<p>The way Fowler writes about design-by-refactoring has two huge<br />
|
|
implications for the relationship between open source and agile<br />
|
|
programming:</p>
|
|
<p>First, Fowler writes as though he <em>didn’t know he was describing<br />
|
|
hacking</em>. In the passage, he appears unaware that design by<br />
|
|
repeated refactoring is not just a recent practice semi-accidentally<br />
|
|
stumbled on by a handful of agile programmers, but one which hundreds<br />
|
|
of thousands of hackers have accumulated experience with for over three<br />
|
|
decades and have in their bones. There is a substantial folklore, an<br />
|
|
entire craft practice, around this!</p>
|
|
<p>Second, in that passage Fowler described the practice of hacking<br />
|
|
<em>better than hackers themselves have done</em>. Now, admittedly,<br />
|
|
the hacker culture has simply not had that many theoreticians, and if<br />
|
|
you list the ones that are strongly focused on development methodology<br />
|
|
you lose Richard Stallman and are left with, basically, myself and<br />
|
|
maybe Larry Wall (author of Perl and occasional funny and illuminating<br />
|
|
ruminations on the art of hacking). But the fact that we don’t have a<br />
|
|
lot of theoreticians is itself an important datum; we have always<br />
|
|
tended to develop our most important wisdoms as unconscious and<br />
|
|
unarticulated craft practice.</p>
|
|
<p>These two observations imply an enormous mutual potential, a gap<br />
|
|
across which an arc of enlightenment may be beginning to blaze. It<br />
|
|
implies two things:</p>
|
|
<p>First, <em>people who are excited by agile-programming ideas can<br />
|
|
look to open source and the Unix tradition and the hackers for the<br />
|
|
lessons of experience</em>. We’ve been doing a lot of the stuff the<br />
|
|
agile movement is talking about for a <em>long</em> time. Doing it in a<br />
|
|
clumsy, unconscious, learned-by-osmosis way, but doing it<br />
|
|
nevertheless. I believe that we have learned things that you agile<br />
|
|
guys need to know to give your methodologies groundedness. Things<br />
|
|
like (as Fowler himself observes) how to manage communication and<br />
|
|
hierarchy issues in distributed teams.</p>
|
|
<p>Second, <em>open-source hackers can learn from agile programmers<br />
|
|
how to wake up</em>. The terminology and conceptual framework of<br />
|
|
agile programming sharpens and articulates our instincts. Learning to<br />
|
|
speak the language of open source, peer review, many eyeballs, and<br />
|
|
rapid iterations gave us a tremendous unifying boost in the late<br />
|
|
1990s; I think becoming similarly conscious about agile-movement ideas<br />
|
|
like refactoring, unit testing, and story-centered design could be<br />
|
|
just as important for us in the new century.</p>
|
|
<p>I’ve already given an example of what the agile movement has to<br />
|
|
teach the hackers, in pointing out that repeated redesign by<br />
|
|
refactoring is a precise description of hacking. Another thing we can<br />
|
|
stand to learn from agile-movement folks is how to behave so that we<br />
|
|
can actually develop requirements and deliver on them when the<br />
|
|
customer isn’t, ultimately, ourselves.</p>
|
|
<p>For the flip side, consider Fowler’s anecdote on page 68-69, which<br />
|
|
ends “Even if you know exactly what is going on in your system,<br />
|
|
measure performance, don’t speculate. You’ll learn something, and<br />
|
|
nine times out of ten it won’t be that you were right.” The Unix guy<br />
|
|
in me wants to respond “Well, <em>duh!</em>“. In my tribe, profiling<br />
|
|
<em>before</em> you speculate is DNA; we have a strong tradition of<br />
|
|
this that goes back to the 1970s. From the point of view of any old<br />
|
|
Unix hand, the fact that Fowler thought he had to write this down is a<br />
|
|
sign of severe naivete in either Fowler or his readership or both.</p>
|
|
<p>In reading <cite>Refactoring</cite>, I several times had the<br />
|
|
experience of thinking “What!?! That’s obvious!” closely followed<br />
|
|
by “But Fowler explains it better than Unix traditions do…” This may<br />
|
|
be because he relies less on the very rich shared explanatory context<br />
|
|
that Unix provides.</p>
|
|
<p>How deep do the similarities run? Let’s take a look at what the<br />
|
|
Agile Manifesto says:</p>
|
|
<p><em>Individuals and interactions over processes and tools.</em> Yeah,<br />
|
|
that sounds like us, all right. Open-source developers will toss out<br />
|
|
a process that isn’t working in a nanosecond, and frequently do, and take<br />
|
|
gleeful delight in doing so. In fact, the reaction against heavyweight<br />
|
|
process has a key part of our self-identification as hackers for<br />
|
|
at least the last quarter century, if not longer.</p>
|
|
<p><em>Working software over comprehensive documentation.</em> That’s<br />
|
|
us, too. In fact, the radical hacker position is that source code of<br />
|
|
a working system <em>is</em> its documentation. We, more than any<br />
|
|
other culture of software engineering, emphasize program source code as<br />
|
|
human-to-human communication that is expected to bind together<br />
|
|
communities of cooperation and understanding distributed through time<br />
|
|
and space. In this, too, we build on and amplify Unix tradition.</p>
|
|
<p><em>Customer collaboration over contract negotiation.</em> In the<br />
|
|
open-source world, the line between “developer” and “customer” blurs<br />
|
|
and often disappears. Non-technical end users are represented by<br />
|
|
developers who are proxies for their interests—as when, for<br />
|
|
example, companies that run large websites second developers to<br />
|
|
work on Apache Software Foundation projects.</p>
|
|
<p><em>Responding to change over following a plan.</em> Absolutely.<br />
|
|
Our whole development style encourages this. It’s fairly unusual for<br />
|
|
any of our projects to <em>have</em> any plan more elaborate than “fix<br />
|
|
the current bugs and chase the next shiny thing we see”.</p>
|
|
<p>With these as main points, it’s hardly surprising that so many of<br />
|
|
the <a href="http://agilemanifesto.org/principles.html">Principles<br />
|
|
behind the Agile Manifesto</a> read like Unix-tradition and hacker<br />
|
|
gospel. “Deliver working software frequently, from a couple of weeks<br />
|
|
to a couple of months, with a preference to the shorter timescale.<br />
|
|
Well, yeah—we <em>pioneered</em> this. Or “Simplicity—the art of<br />
|
|
maximizing the amount of work not done—is essential.” That’s<br />
|
|
Unix-tradition holy writ, there. Or “The best architectures,<br />
|
|
requirements, and designs emerge from self-organizing teams.”</p>
|
|
<p>This is stone-obvious stuff to any hacker, and exactly the sort of<br />
|
|
subversive thinking that most panics managers attached to big plans,<br />
|
|
big budgets, big up-front design, and big rigid command-and-control<br />
|
|
structures. Which may, in fact, be a key part of its appeal to<br />
|
|
hackers and agile developers—because at least one thing that points<br />
|
|
agile-movement and open-source people in the same direction is a drive<br />
|
|
to take control of our art back from the suits and get out from under<br />
|
|
big dumb management.</p>
|
|
<p>The most important difference I see between the hackers and the<br />
|
|
agile-movement crowd is this: the hackers are the people who never<br />
|
|
surrendered to big dumb management — they either bailed out of the<br />
|
|
system or forted up in academia or industrial R&D labs or<br />
|
|
technical-specialty areas where pointy-haired bosses weren’t permitted<br />
|
|
to do as much damage. The agile crowd, on the other hand, seems to be<br />
|
|
composed largely of people who were swallowed into the belly of the<br />
|
|
beast (waterfall-model projects, Windows, the entire conventional<br />
|
|
corporate-development hell so vividly described in Edward Yourdon’s<br />
|
|
books) and have been smart enough not just to claw their way out but<br />
|
|
to formulate an ideology to justify not getting sucked back in.</p>
|
|
<p>Both groups are in revolt against the same set of organizational<br />
|
|
assumptions. And both are winning because those assumptions are<br />
|
|
obsolete, yesterday’s adaptations to a world of expensive machines and<br />
|
|
expensive communications. But software development doesn’t need big<br />
|
|
concentrations of capital and resources anymore, and doesn’t need the<br />
|
|
control structures and hierarchies and secrecy and elaborate rituals<br />
|
|
that go with managing big capital concentrations either. In fact, in<br />
|
|
a world of rapid change, these things are nothing but a drag. Thus<br />
|
|
agile techniques. Thus, open source. Converging paths to the same<br />
|
|
destination, which is not just software that doesn’t suck but a<br />
|
|
software-development <em>process</em> that doesn’t suck.</p>
|
|
<p>When I think about how the tribal wisdom of the hackers and the<br />
|
|
sharp cut-the-bullshit insights of the agile movement seem to be<br />
|
|
coming together, my mind keeps circling back to Phil Greenspun’s brief<br />
|
|
but trenchant essay <a href="http://tinyplanet.ca/projects/professionalism.html">Redefining<br />
|
|
Professionalism for Software Engineers</a>. Greenspun proposes,<br />
|
|
provocatively but I think correctly, that the shift towards<br />
|
|
open-source development is a key part of the transformation of<br />
|
|
software engineering into a mature profession, with the dedication to<br />
|
|
excellence and ethos of service that accompanies professionalism. I<br />
|
|
have elsewhere suggested that we are seeing a close historical analog<br />
|
|
of the transition from alchemy to chemistry. Secrets leak out, but<br />
|
|
skill sustains; the necessity to stop relying on craft secrecy is one<br />
|
|
of the crises that occupational groups normally face as they attain<br />
|
|
professional standing.</p>
|
|
<p>I’m beginning to think that from the wreckage of the software<br />
|
|
industry big dumb management made, I can see the outline of a mature,<br />
|
|
<em>humane</em> discipline of software engineering emerging — and<br />
|
|
that it will be in large part a fusion of the responsiveness and<br />
|
|
customer focus of the agile movement with the wisdom and groundedness<br />
|
|
of the Unix tradition, expressed in open source.</p>
|
|
<p><a href="http://enetation.co.uk/comments.php?user=esr&commentid=105563829929887753">Blogspot<br />
|
|
comments</a></p>
|