blog_post_tests/20030614175100.blog

Hacking and Refactoring
<p>In 2001, there was a history-making conference of software-engineering<br />
thinkers in Snowbird, Colorado.  The product of that meeting was a remarkable<br />
document called the <a href="http://agilemanifesto.org/">Agile Manifesto</a>,<br />
a call to overturn many of the assumptions of traditional software development.<br />
I was invited to be at Snowbird, but couldn&#8217;t make it.</p>
<p>Ever since, though, I&#8217;ve been sensing a growing convergence between<br />
agile programming and the open-source movement.  I&#8217;ve seen agile<br />
concepts and terminology being adopted rapidly and enthusiastically by<br />
my colleagues in open-source-land&mdash;especially ideas like<br />
refactoring, unit testing, and design from stories and personas.  From<br />
the other side, key agile-movement figures like Kent Beck and Martin<br />
Fowler have expressed strong interest in open source both in published<br />
works and to me personally.  Fowler has gone so far as to <a href="http://www.martinfowler.com/articles/newMethodology.html#N400254">include</a><br />
open source on his list of agile-movement schools.</p>
<p>I agree that we belong on that list.  But I also agree with<br />
Fowler&#8217;s description of of open source as a style, rather than a<br />
process.  I think his reservations as to whether open source can be<br />
described as just another agile school are well-founded.  There is<br />
something more complicated and interesting going on here. and I<br />
realized when I read Fowler&#8217;s description of open source that at some<br />
point I was going to have to do some hard thinking and writing in an<br />
effort to sort it all out.</p>
<p>While doing research for my forthcoming book, <a href="http://www.catb.org/esr/writings/taoup/">The Art of Unix<br />
Programming</a>, I read one particular passage in Fowler&#8217;s<br />
<cite>Refactoring</cite> that finally brought it all home. He<br />
writes:</p>
<blockquote>
<p>One argument is that refactoring can be an alternative to up-front<br />
design.  In this scenario, you don&#8217;t do any design at all.  You just<br />
code the first approach that comes into your head, get it working, and<br />
then refactor it into shape.  Actually, this approach can work.  I&#8217;ve<br />
seen people do this and come out with a very well-defined piece of<br />
software.  Those who support Extreme Programming often are portrayed<br />
as advocating this approach.</p>
</blockquote>
<p>I read this, and had one of those moments where everything comes<br />
together in your head with a great ringing crash and the world assumes<br />
a new shape&mdash;a moment not unlike the one I had in late 1996<br />
when I got the central insight that turned into <cite>The Cathedral<br />
and the Bazaar</cite>.  In the remainder of this essay I&#8217;m going to<br />
try to articulate what I now think I understand about open source,<br />
agile programming, how they are related, and why the connection should<br />
be interesting even to programmers with no stake in either movement.</p>
<p>Now I need to set a little background here, because I&#8217;m going<br />
to need to have to talk about several different categories which are<br />
contingently but not necessarily related.</p>
<p>First, there is <em>Unix programmer</em>.  Unix is the operating<br />
system with the longest living tradition of programming and design.<br />
It has an unusually strong and mature technical culture around it, a<br />
culture which originated or popularized many of the core ideas and<br />
tools of modern software design. <cite>The Art of Unix<br />
Programming</cite> is a concerted attempt to capture the craft wisdom<br />
of this culture, one to which I have successfully enlisted quite a few<br />
of its founding elders.</p>
<p>Second, there is <em>hacker</em>.  This is a very complex term, but<br />
more than anything else, it describes an attitude&mdash;an<br />
intentional stance that relates hackers to programming and other<br />
disciplines in a particular way.  I have described the hacker stance<br />
and its cultural correlates in detail in <a href="http://www.catb.org/esr/faqs/hacker-howto.html">How To Become A<br />
Hacker</a>.</p>
<p>Third, there is <em>open-source programmer</em>.  Open source is a<br />
programming style with strong roots in the Unix tradition and the<br />
hacker culture.  I wrote the modern manifesto for it in 1997, <a href="http://www.catb.org/~esr/writings/cathedral-bazaar/">The<br />
Cathedral and the Bazaar</a>, building on earlier thinking by<br />
Richard Stallman and others.</p>
<p>These three categories are historically closely related. It is<br />
significant that a single person (accidentally, me) wrote touchstone<br />
documents for the second and third and is attempting a <em>summum<br />
bonum</em> of the first.  That personal coincidence reflects a larger<br />
social reality that in 2003 these categories are becoming increasingly<br />
merged &mdash; essentially, the hacker community has become the core<br />
of the open-source community, which is rapidly re-assimilating the<br />
parts of the Unix culture that got away from the hackers during<br />
the ten bad years after the AT&amp;T divestiture in 1984.</p>
<p>But the relationship is not logically entailed; we can imagine<br />
a hacker culture speaking a common tongue other than Unix and C (in<br />
the far past its common tongue was Lisp), and we can imagine an<br />
explicit ideology of open source developing within a cultural and<br />
technical context other than Unix (as indeed nearly happened several<br />
different times).</p>
<p>With this scene-setting done, I can explain that my first take on<br />
Fowler&#8217;s statement was to think &#8220;Dude, you&#8217;ve just described<br />
<em>hacking</em>!&#8221;</p>
<p>I mean something specific and powerful by this.  Throwing together<br />
a prototype and refactoring it into shape is a rather precise<br />
description of the normal working practice of hackers since that<br />
culture began to self-define in the 1960s.  Not a complete one, but it<br />
captures the most salient feature of how hackers relate to code.  The<br />
open-source community has inherited and elaborated this practice,<br />
building on similar tendencies within the Unix tradition.</p>
<p>The way Fowler writes about design-by-refactoring has two huge<br />
implications for the relationship between open source and agile<br />
programming:</p>
<p>First, Fowler writes as though he <em>didn&#8217;t know he was describing<br />
hacking</em>.  In the passage, he appears unaware that design by<br />
repeated refactoring is not just a recent practice semi-accidentally<br />
stumbled on by a handful of agile programmers, but one which hundreds<br />
of thousands of hackers have accumulated experience with for over three<br />
decades and have in their bones. There is a substantial folklore, an<br />
entire craft practice, around this!</p>
<p>Second, in that passage Fowler described the practice of hacking<br />
<em>better than hackers themselves have done</em>.  Now, admittedly,<br />
the hacker culture has simply not had that many theoreticians, and if<br />
you list the ones that are strongly focused on development methodology<br />
you lose Richard Stallman and are left with, basically, myself and<br />
maybe Larry Wall (author of Perl and occasional funny and illuminating<br />
ruminations on the art of hacking).  But the fact that we don&#8217;t have a<br />
lot of theoreticians is itself an important datum; we have always<br />
tended to develop our most important wisdoms as unconscious and<br />
unarticulated craft practice.</p>
<p>These two observations imply an enormous mutual potential, a gap<br />
across which an arc of enlightenment may be beginning to blaze.  It<br />
implies two things:</p>
<p>First, <em>people who are excited by agile-programming ideas can<br />
look to open source and the Unix tradition and the hackers for the<br />
lessons of experience</em>.  We&#8217;ve been doing a lot of the stuff the<br />
agile movement is talking about for a <em>long</em> time.  Doing it in a<br />
clumsy, unconscious, learned-by-osmosis way, but doing it<br />
nevertheless.  I believe that we have learned things that you agile<br />
guys need to know to give your methodologies groundedness.  Things<br />
like (as Fowler himself observes) how to manage communication and<br />
hierarchy issues in distributed teams.</p>
<p>Second, <em>open-source hackers can learn from agile programmers<br />
how to wake up</em>.  The terminology and conceptual framework of<br />
agile programming sharpens and articulates our instincts.  Learning to<br />
speak the language of open source, peer review, many eyeballs, and<br />
rapid iterations gave us a tremendous unifying boost in the late<br />
1990s; I think becoming similarly conscious about agile-movement ideas<br />
like refactoring, unit testing, and story-centered design could be<br />
just as important for us in the new century.</p>
<p>I&#8217;ve already given an example of what the agile movement has to<br />
teach the hackers, in pointing out that repeated redesign by<br />
refactoring is a precise description of hacking.  Another thing we can<br />
stand to learn from agile-movement folks is how to behave so that we<br />
can actually develop requirements and deliver on them when the<br />
customer isn&#8217;t, ultimately, ourselves.</p>
<p>For the flip side, consider Fowler&#8217;s anecdote on page 68-69, which<br />
ends &#8220;Even if you know exactly what is going on in your system,<br />
measure performance, don&#8217;t speculate.  You&#8217;ll learn something, and<br />
nine times out of ten it won&#8217;t be that you were right.&#8221;  The Unix guy<br />
in me wants to respond &#8220;Well, <em>duh!</em>&#8220;.  In my tribe, profiling<br />
<em>before</em> you speculate is DNA; we have a strong tradition of<br />
this that goes back to the 1970s. From the point of view of any old<br />
Unix hand, the fact that Fowler thought he had to write this down is a<br />
sign of severe naivete in either Fowler or his readership or both.</p>
<p>In reading <cite>Refactoring</cite>, I several times had the<br />
experience of thinking &#8220;What!?!  That&#8217;s obvious!&#8221; closely followed<br />
by &#8220;But Fowler explains it better than Unix traditions do&#8230;&#8221; This may<br />
be because he relies less on the very rich shared explanatory context<br />
that Unix provides.</p>
<p>How deep do the similarities run? Let&#8217;s take a look at what the<br />
Agile Manifesto says:</p>
<p><em>Individuals and interactions over processes and tools.</em>  Yeah,<br />
that sounds like us, all right.  Open-source developers will toss out<br />
a process that isn&#8217;t working in a nanosecond, and frequently do, and take<br />
gleeful delight in doing so.  In fact, the reaction against heavyweight<br />
process has a key part of our self-identification as hackers for<br />
at least the last quarter century, if not longer.</p>
<p><em>Working software over comprehensive documentation.</em> That&#8217;s<br />
us, too.  In fact, the radical hacker position is that source code of<br />
a working system <em>is</em> its documentation.  We, more than any<br />
other culture of software engineering, emphasize program source code as<br />
human-to-human communication that is expected to bind together<br />
communities of cooperation and understanding distributed through time<br />
and space.  In this, too, we build on and amplify Unix tradition.</p>
<p><em>Customer collaboration over contract negotiation.</em> In the<br />
open-source world, the line between &#8220;developer&#8221; and &#8220;customer&#8221; blurs<br />
and often disappears.  Non-technical end users are represented by<br />
developers who are proxies for their interests&mdash;as when, for<br />
example, companies that run large websites second developers to<br />
work on Apache Software Foundation projects.</p>
<p><em>Responding to change over following a plan.</em> Absolutely.<br />
Our whole development style encourages this.  It&#8217;s fairly unusual for<br />
any of our projects to <em>have</em> any plan more elaborate than &#8220;fix<br />
the current bugs and chase the next shiny thing we see&#8221;.</p>
<p>With these as main points, it&#8217;s hardly surprising that so many of<br />
the <a href="http://agilemanifesto.org/principles.html">Principles<br />
behind the Agile Manifesto</a> read like Unix-tradition and hacker<br />
gospel.  &#8220;Deliver working software frequently, from a couple of weeks<br />
to a couple of months, with a preference to the shorter timescale.<br />
Well, yeah&mdash;we <em>pioneered</em> this.  Or &#8220;Simplicity&mdash;the art of<br />
maximizing the amount of work not done&mdash;is essential.&#8221;  That&#8217;s<br />
Unix-tradition holy writ, there.  Or &#8220;The best architectures,<br />
requirements, and designs emerge from self-organizing teams.&#8221;</p>
<p>This is stone-obvious stuff to any hacker, and exactly the sort of<br />
subversive thinking that most panics managers attached to big plans,<br />
big budgets, big up-front design, and big rigid command-and-control<br />
structures.  Which may, in fact, be a key part of its appeal to<br />
hackers and agile developers&mdash;because at least one thing that points<br />
agile-movement and open-source people in the same direction is a drive<br />
to take control of our art back from the suits and get out from under<br />
big dumb management.</p>
<p>The most important difference I see between the hackers and the<br />
agile-movement crowd is this: the hackers are the people who never<br />
surrendered to big dumb management &mdash; they either bailed out of the<br />
system or forted up in academia or industrial R&amp;D labs or<br />
technical-specialty areas where pointy-haired bosses weren&#8217;t permitted<br />
to do as much damage.  The agile crowd, on the other hand, seems to be<br />
composed largely of people who were swallowed into the belly of the<br />
beast (waterfall-model projects, Windows, the entire conventional<br />
corporate-development hell so vividly described in Edward Yourdon&#8217;s<br />
books) and have been smart enough not just to claw their way out but<br />
to formulate an ideology to justify not getting sucked back in.</p>
<p>Both groups are in revolt against the same set of organizational<br />
assumptions.  And both are winning because those assumptions are<br />
obsolete, yesterday&#8217;s adaptations to a world of expensive machines and<br />
expensive communications.  But software development doesn&#8217;t need big<br />
concentrations of capital and resources anymore, and doesn&#8217;t need the<br />
control structures and hierarchies and secrecy and elaborate rituals<br />
that go with managing big capital concentrations either.  In fact, in<br />
a world of rapid change, these things are nothing but a drag.  Thus<br />
agile techniques.  Thus, open source.  Converging paths to the same<br />
destination, which is not just software that doesn&#8217;t suck but a<br />
software-development <em>process</em> that doesn&#8217;t suck.</p>
<p>When I think about how the tribal wisdom of the hackers and the<br />
sharp cut-the-bullshit insights of the agile movement seem to be<br />
coming together, my mind keeps circling back to Phil Greenspun&#8217;s brief<br />
but trenchant essay <a href="http://tinyplanet.ca/projects/professionalism.html">Redefining<br />
Professionalism for Software Engineers</a>.  Greenspun proposes,<br />
provocatively but I think correctly, that the shift towards<br />
open-source development is a key part of the transformation of<br />
software engineering into a mature profession, with the dedication to<br />
excellence and ethos of service that accompanies professionalism.  I<br />
have elsewhere suggested that we are seeing a close historical analog<br />
of the transition from alchemy to chemistry. Secrets leak out, but<br />
skill sustains; the necessity to stop relying on craft secrecy is one<br />
of the crises that occupational groups normally face as they attain<br />
professional standing.</p>
<p>I&#8217;m beginning to think that from the wreckage of the software<br />
industry big dumb management made, I can see the outline of a mature,<br />
<em>humane</em> discipline of software engineering emerging &mdash; and<br />
that it will be in large part a fusion of the responsiveness and<br />
customer focus of the agile movement with the wisdom and groundedness<br />
of the Unix tradition, expressed in open source.</p>
<p><a href="http://enetation.co.uk/comments.php?user=esr&amp;commentid=105563829929887753">Blogspot<br />
comments</a></p>