Right back at ya, Captain
<p>Last Saturday morning in San Diego I had breakfast with Steven den<br />
Beste, the redoubtable captain of <a href="http://www.denbeste.nu/">U.S.S. Clueless</a>.  One of the<br />
side-effects of that meeting was a long <a href="http://www.denbeste.nu/cd_log_entries/2002/07/OpenSourcepart1.shtml"><br />
critique</a> of open-source development.  Herewith my response.</p>
<p>Steve and I agree on the scaling problem that has pushed software<br />
development efforts to the ragged edge of what is sustainable even by<br />
corporations with lots of money.  Software project sizes are roughly<br />
doubling every eighteen months, and for reasons Steve alluded to the<br />
expected bug count per thousand lines is actually rising.</p>
<p>My assertion is that software development has reached a scale at<br />
which (a) even large corporations can often no longer afford to field<br />
enough developers to be effective at today&#8217;s project scales, and (b)<br />
traditional methods of software quality assurance (ranging from formal<br />
methods to internal walkthroughs) are no longer effective.  The only<br />
development organizations that seem to thrive on today&#8217;s complexity<br />
regime are open-source teams.</p>
<p>Note that I am not claiming that open source is a silver bullet for<br />
the software-complexity problem.  There are no silver bullets, no<br />
permanent solutions. What I <em>am</em> claiming is that at the<br />
leading edge of large-scale software, closed-source development<br />
doesn&#8217;t work any more.  The future belongs to open source plus<br />
whatever other practices and technologies we learn to use with<br />
it to develop at ever-higher scales of complexity.</p>
<p>Steve&#8217;s analysis of the open-source phenomenon is very intelligent,<br />
but doesn&#8217;t quite understand either the mode of organization, the<br />
associated technology, or the factional politics within the movement.<br />
Diagnostic of the slight disconnect is when he writes &#8220;For [the<br />
zealots], the only true &#8220;Open Source&#8221; is governed by the strong form<br />
of the GPL, and all other forms and licenses are harmful dilution of<br />
the concept.&#8221;  In fact, the people he&#8217;s talking about reject the term<br />
&#8220;open source&#8221; entirely and insist on the ideologically-loaded term<br />
&#8220;free software&#8221;.</p>
<p>A more serious error is when he writes &#8220;It is plausible that an OSS<br />
project would require each participant to sign an NDA before being<br />
given access to the source.&#8221;  It is <em>not</em> plausible.  The licenses<br />
and community values of the open-source community would not permit this.<br />
His two bullet points characterizing open source are missing its most<br />
important characteristic: the entire practice is designed to facilitate<br />
scrutiny by people with no institutional or contract relationship to the core<br />
development team.  The astringent effect of peer review by people who<br />
have <em>nothing to lose</em> by reporting bugs is precisely the<br />
point of the whole game.</p>
<p>Steve doesn&#8217;t undertand the importance or the power of this effect. This<br />
slightly skews his whole essay; much of it is talking past what open-source<br />
people do, rather than addressing us.  He&#8217;s also unaware of a lot of the<br />
real-world evidence for the success of the method.  Some of the things he<br />
thinks are technologically or economically impossible are actually being<br />
done, routinely.</p>
<p>He&#8217;s correct when he says that most contributors are self-selected and<br />
self-motivated.  He overestimates the cost of training newbies, though.  They<br />
self-train; normally, the first time a core developer hears from a newbie<br />
is typically when the newbie sends a patch &#8212; self-evidence that the newbie<br />
has <em>already</em> acquired a critical level of knowledge about the<br />
software.  The &#8220;sink or swim&#8221; method turns out to work, and work well.</p>
<p>It&#8217;s incorrect to imply, as he does, that open-source development<br />
is unsustainable because the people doing it are flaky amateurs.<br />
Steve hasn&#8217;t absorbed the implications of the Boston Consulting<br />
Group study that shows that about 40% of contributors to core projects<br />
are professionals getting paid for working on open source by patrons<br />
who need to use the results.  In fact, what the open-source community<br />
is evolving into is something very like a huge machine for bringing<br />
newbies into apprenticeship contact with experienced developers and<br />
professionalizing both groups.</p>
<p>He also writes &#8220;OSS by its nature tends to be reactive rather than<br />
predictive. It doesn&#8217;t look into the future, try to predict a problem<br />
which doesn&#8217;t exist now but will exist then, and be ready with a<br />
solution. Rather, it tends to see problems that exist now and work on<br />
solutions for them.&#8221;  This is false &#8212; or, at any rate, no more true<br />
than it is for closed-source development.</p>
<p>The open-source community built the Web and the Internet before it<br />
had acquired a name for itself and full consciousness of its own<br />
practices.  Today, the cutting-edge work in operating systems<br />
languages, desktop user interfaces, relational databases and many<br />
other areas is being done either within the open-source community or<br />
in cooperation with it by academics.  These prodigious efforts of<br />
imagination dwarf any &#8220;prediction&#8221; produced by closed-source software<br />
development in the last two decades.</p>
<p>Steve&#8217;s &#8220;open source is reactive&#8221; claim strikes me as ironically<br />
funny, because I can remember when the standard knock on my crowd was<br />
that we&#8217;re great at innovation but can&#8217;t actually field product.  How<br />
quickly they forget&#8230;</p>
<p>He&#8217;s right enough about the difficulty of planning and high cost<br />
of face-to-face meetings, though.  These are real problems.  It&#8217;s<br />
a testimony to the power of our practices that we manage to ship large<br />
volumes of high-quality software despite these obstacles.</p>
<p>What Steve called &#8220;player-killer&#8221; tactics have been tried &#8212; there<br />
was a famous incident a few years back in which a TCP-wrappers<br />
distribution was Trojaned.  The crack was detected and the community<br />
warned within hours.  The black hats don&#8217;t seem to bother trying this<br />
any more; our reaction time is too fast for that game to be very<br />
rewarding.  The technical design of Linux helps here in ways that<br />
I won&#8217;t go into here &#8212; suffice it to say that it&#8217;s intrinsically<br />
much harder to get a Trojan to do anything interesting than it<br />
is under Windows or other single-user operating systems.</p>
<p>So far, the supply of open-source developers seems to be pretty<br />
elastic &#8212; we&#8217;re not limited much by lacking bodies.  Other factors<br />
loom much larger; patents, the DMCA, intrinsically hard technical<br />
problems.  I don&#8217;t understand why this is as well as I&#8217;d like to, but<br />
the facts are undeniable; the community is ten times the size my<br />
wildest high-end scenarios predicted a decade ago and seems to be<br />
growing <em>faster</em> as it gets larger.</p>
<p>Steve&#8217;s whole argument that open-source can&#8217;t win in embedded<br />
systems is very curious, since it predicts exactly the opposite of<br />
what is actually happening out there.  Linux is taking over in<br />
embedded systems &#8212; in fact, many observers would say it has already<br />
won that space.  If Steve had worked in the field within the last<br />
three years he would probably know this.</p>
<p>Here are some data about the demand; the only non-general-purpose<br />
open-source software magazine in existence is the Linux Embedded<br />
Systems Journal.  Open-source embedded developers like Monta Vista<br />
Software are bucking the recession by growing like crazy.  The first<br />
cell-phone prototype running entirely open-source software just<br />
entered beta testing.</p>
<p>I was in California to meet Steve partly because Real Networks<br />
wanted me to be on stage when they announced the open-sourcing of<br />
their RTSP engine. Their CEO, Rob Glaser, was quite frank about the<br />
immediate business reasons: they needed to get ports to forty<br />
different Nokia cellphones and just couldn&#8217;t figure out how to muster<br />
the resources for that short of inviting every stakeholder on the<br />
planet to hack the problem. Scaling bites.  Hard.</p>
<p>In fact, some of the very characteristics that Steve thinks make<br />
embedded systems like cellphones safe for closed development seems to<br />
be the factors that are driving increased open-sourcing.  The close<br />
tie to hardware actually <em>decreases</em> the value of secrecy,<br />
because it means the software is typically not easily re-usable by<br />
hardware competitors.  Thus open sourcing is often a great way to<br />
recruit help from customer engineers without a real downside risk of<br />
plagiarism.</p>
<p>In fact, it&#8217;s an open secret in the industry that the most<br />
important reason most closed-source embedded and driver software<br />
remains closed is not nerves about plagiarism but fear of patent<br />
audits on the source code.  Graphics-card manufacturers, in<br />
particular, routinely swipe patented techniques from their competitors<br />
and bury them in binaries.  (This is generally believed to be the<br />
reason nVidia&#8217;s drivers aren&#8217;t open.)</p>
<p>Another trend that&#8217;s driving Linux and open-sourcing in embedded<br />
stuff is the shift from specialty embedded 8-bit processors to 32-bit<br />
chips with general-purpose architectures.  Turns out the development<br />
costs for getting stuff to run on the 8-bit chips are sickeningly high<br />
and rising &#8212; partly because the few wizards who can do good work on<br />
that hardware are <em>expensive</em>.  The incremental cost for<br />
smarter hardware has dropped a lot; it&#8217;s now cheaper to embed<br />
general-purpose chips running Linux because it means you have a<br />
larger, less expensive talent pool that can program them.  Also,<br />
when your developers aren&#8217;t fighting hardware limits as hard,<br />
you get better time to market (which, as Steve observes, is<br />
critical).</p>
<p>Steve is right about the comparative difficulty of applying<br />
open-source methods to vertical applications.  But the difficulty is<br />
only comparative; it&#8217;s happening anyway.  The metalab archive carries<br />
a point-of-sale system for pizza parlors.  I know of another case in<br />
which a Canadian auto dealership built specialized accounting software<br />
for their business and open-sourced it.  The reasons?  Same as usual;<br />
they wanted to lay off as much as possible of the development and<br />
maintainance cost on their competitors.</p>
<p>This is the same co-opetition logic that makes the Apache Software<br />
Foundation work &#8212; it&#8217;s just as powerful for vertical apps, though<br />
less obviously so.  Each sponsoring company sees a higher payoff from<br />
having the software at a small fraction of the manpower cost for a<br />
complete in-house development.  The method spreads risk in a way<br />
beneficial to all parties, too, because the ability of separate<br />
companies to sustain development tends to be uncorrelated &#8212; unless<br />
they <em>all</em> sink, the project endures.</p>
<p>The way to solve the problem of not exposing your business logic to<br />
competitors is to separate your app into an open-source engine and a<br />
bunch of declarative business-rule schemas that you keep secret.<br />
Databases work this way, and websites (the web pages and CGIs are the<br />
schema).  Many vertical apps can be partitioned this way too &#8212; in<br />
fact, for things like tax-preparation software they almost have to be,<br />
because the complexity overhead of hacking executable code every time<br />
the rules change is too high.</p>
<p>Steve thinks the differences between Apache and Mozilla are bigger<br />
than they are.  In fact, the core groups of <em>both</em> projects are<br />
full-time pros being funded by large users of the software.</p>
<p>So, let&#8217;s address Steve&#8217;s objections point by point:</p>
<p><em>For embedded software, OSS has the following problems:</em></p>
<ul>
<li>
<p><em>It can&#8217;t be scheduled; timely delivery can&#8217;t be relied<br />
on.</em></p>
<p>Timely delivery can&#8217;t be relied on for <em>any</em> software; see<br />
De Marco and Lister&#8217;s excellent book <cite><a href="http://www.dorsethouse.com/books/pw.html">Peopleware: Productive<br />
Projects and Teams</a></cite> on the delusion of deadlines, especially<br />
the empirical evidence that the &#8220;wake me up when it&#8217;s done&#8221; strategy<br />
of not setting them actually gets your project done faster (also the<br />
implication of a recent Harvard Business School study of software<br />
project outcomes).</p>
<p>Open source is at least not noticeably worse than closed-source on this<br />
axis. Arguably it&#8217;s better, because the rapid release cycles allow users<br />
to pick up on project results as soon as they&#8217;re good enough.</p>
</li>
<li>
<p><em>Debugging requires access to custom hardware which usually<br />
can&#8217;t easily be accessed across the net.</em></p>
<p>There aren&#8217;t good solutions to this problem yet, but the increasing<br />
use of &#8220;overpowered&#8221; 32-bit processors using standard busses is<br />
tending to reduce it in scope.  The development tools and interface<br />
hardware used in embedded stuff are rapidly getting more generic and closer<br />
to what&#8217;s used in general-purpose computers.</p>
</li>
<li>
<p><em>Active participation even for junior people requires substantial<br />
amounts of project-specific knowledge which isn&#8217;t easily acquired,<br />
especially remotely.</em></p>
<p>This one puzzles me, because I think Steve ought to be right about<br />
it &#8212; but I&#8217;m not hearing the kinds of noises that I&#8217;d hear if it were<br />
slowing down the move to Linux and open source significantly.</p>
<p>At least part of the answer is that embedded-systems work is<br />
getting de-skilled in a particular sense &#8212; more of it&#8217;s being done by<br />
application specialists who are training up to the required level of<br />
programming, rather than programmers who have acquired expensive<br />
application-specific knowledge.</p>
</li>
<li>
<p><em>A great deal of proprietary information is usually involved in<br />
the process, and if that&#8217;s released the company can be seriously<br />
harmed.</em></p>
<p>It&#8217;s a question of tradeoffs.  As RealNetworks found out when<br />
costing its Nokia contract, the choice is increasingly between giving<br />
up control of some of your proprietary IP and being too resource-bound<br />
to ship at all.</p>
<p>There is no market for secrecy.  There&#8217;s a market for product.  If<br />
you can&#8217;t ship product, or your customers aren&#8217;t confident that you<br />
can maintain it after shipping, all that proprietary IP amounts to is<br />
a millstone around your neck.</p>
<p>There will be more stories like RTSP in the future.  Count on it.<br />
In fact, the day will come when most of your contract partners simply<br />
won&#8217;t accept the business risks of having someone else hold<br />
proprietary rights on the embedded software they use.</p>
</li>
<li>
<p><em>It&#8217;s nearly impossible to do embedded software without<br />
common impromptu face-to-face meetings with co-workers, either to ask<br />
questions or to brainstorm. Doing this electronically is sufficiently<br />
different as to not be practical.</em></p>
<p>Yeah.  They used to think that about operating systems, too.  Obviously<br />
the Linux kernel is impossible, and therefore doesn&#8217;t exist.</p>
<p>(At which point Oolon Colluphid disappeared in a puff of logic.)</p>
</li>
</ul>
<p><em>For vertical apps, the objections are:</em></p>
<ul>
<li>
<p><em>Security, security, security. You want me to trust my<br />
billing system to code written by anyone who happens to come along and<br />
volunteer to work on it, without any kind of check of credentials or<br />
checks on trustworthiness?</em></p>
<p>One of the lessons the business world has been absorbing is that<br />
open-source projects are dramatically <em>more</em> secure than their<br />
closed-source competition &#8212; anybody who compares the Bugtraq records<br />
on Apache vs. ISS defacements, or Linux vs. Windows remote exploits,<br />
will notice that real fast.</p>
<p>It&#8217;s not hard to understand why this is &#8212; I&#8217;ve found that even<br />
corporate executives grok the theory pretty quickly.  I won&#8217;t do the whole<br />
argument here, but this article on <a href="http://www.wikipedia.com/wiki/Kerckhoffs'+law">Kerckhoff&#8217;s<br />
Law</a> holds the crucial clue.  When you rely on the obscurity of source<br />
code for security, it means that the bad guys find the bugs faster than<br />
you can plug them &#8212; there are more of them, and they have entropy on<br />
their side.  Open source evens the odds for the good guys.</p>
</li>
<li>
<p><em>Recruitment: for most of the kind of people involved in<br />
OSS, vertical apps are boring. (Unless they want to figure out how to<br />
steal from it.)</em></p>
<p>This remains a problem.  On the other hand, open source makes it<br />
easier to train domain specialists to be good enough programmers to<br />
get the job done.  It&#8217;s easier for physicists to learn to hack than<br />
it is for hackers to learn physics.</p>
</li>
<li>
<p><em>It takes a lot of knowledge of the specific aspects of the<br />
problem to make a significant contribution, which means things like<br />
observing the actual process of guests checking in at the front desk<br />
of the hotel.</em></p>
<p>This just reinforces the tendency for vertical-app developers to be<br />
obsessives about something else who learn to program, rather than obsessives<br />
about programming who learn something else.</p>
<p>Professional programmers tend to bridle at this thought.  Well, better<br />
learn to live with it.  As software becomes more pervasive, the amount<br />
of it done by application-specialist &#8220;amateurs&#8221; is going to increase.</p>
</li>
<li>
<p><em>The industry is full of horror stories of vertical apps<br />
which ran badly over budget and over schedule; the idea scares the<br />
hell out of business people. They&#8217;re unlikely to be very enthused by<br />
the use of a process which by its nature *cannot* be reliably<br />
scheduled. (Remember that Mozilla ran two years long.)</em></p>
<p>Schedules &#8212; and the belief that deadlines make software happen<br />
faster &#8212; are a delusion in the mind of management, one not supported<br />
by the actual evidence about project outcomes.  This delusion is<br />
so entrenched that managers fail to interpret the 70% rate of<br />
project failures correctly.  It&#8217;s as if people were so determined<br />
to believe the Earth is flat that they ignore what their eyes tell<br />
them when ships sink over the horizon.</p>
<p><em>No</em> software larger than toy programs can be scheduled.<br />
Tactics aimed at doing so normally have the actual effect of<br />
<em>increasing</em> the time to market.  `Aggressive&#8217; schedules<br />
effectively guarantee failure.  The sooner we learn these objective<br />
truths, and that the illusion of control that schedules give is not<br />
worth the real costs, the sooner rates of outright project failure<br />
will dip below 70%.</p>
<p>Go read <cite>Peopleware</cite>. <em>Now</em>.</p>
</li>
</ul>
<p><em>For short life apps:</em></p>
<ul>
<li>
<p><em>Schedule is everything. If you&#8217;re six months late, you&#8217;re dead.</em></p>
<p>See above.  There are reasons open sourcing is less applicable to short-life<br />
applications, but this turns out not to be one of them.</p>
</li>
<li>
<p><em>Secrecy is everything else. If you&#8217;re on time but your<br />
competitor knows what you&#8217;re doing a year ahead, he&#8217;ll wipe you<br />
out.</em></p>
<p>This argument has more force for short-life apps than for Steve&#8217;s other<br />
categories, but remember that increasingly the alternative to open source<br />
is not being able to ship at all.  Your competitor is in the same boat<br />
you are.</p>
</li>
<li>
<p><em>How do you make money selling what anyone can get for free<br />
from any developer? If your product was developed out in the open, who<br />
exactly buys it afterwards?</em></p>
<p>Steve has a stronger point here.  It&#8217;s one that people used to<br />
think applied to almost all software, but which turns out to be mainly<br />
a problem for short-life apps.  Actually the distinguishing<br />
characteristic isn&#8217;t expected lifetime per se, but something<br />
correlated with it &#8212; whether the product needs continued downstream<br />
work (maintainance and upgrades) or not.</p>
<p>Long-life, high-maintainance apps create niches for service businesses.<br />
That&#8217;s the main way you make money in an open-source world.  It&#8217;s<br />
harder to make that work with a short-life app.  Sometimes it&#8217;s<br />
impossible.  Life is hard.</p>
</li>
</ul>
<p><em>For long life apps:</em></p>
<ul>
<li>
<p><em>Will the participants be willing to work on what our<br />
marketing analysis says we need, or will they insist that they know<br />
what is required and try to add that instead? We don&#8217;t need feature<br />
creep, or people trying to change the direction we&#8217;re moving.</em></p>
<p>In open-source projects, the function of &#8220;marketing analysis&#8221; tends to<br />
be taken be direct interaction with the user community.  We find we<br />
do better work without a bunch of marketroids getting between us and<br />
our customers.</p>
</li>
<li>
<p><em>There is major learning curve involved in making a<br />
reasonable contribution to these kinds of programs; you don&#8217;t learn<br />
how a circuit board router works in a few days of study. In most cases<br />
you have to be conversant with the way that the package&#8217;s customers do<br />
what they do, and most programmers don&#8217;t know these things and can&#8217;t<br />
easily learn them.</em></p>
<p>See my previous remarks about application specialists and the<br />
democratization of programming.  And every time you&#8217;re tempted to<br />
say &#8220;But they couldn&#8217;t possibly get away with that in application<br />
area X&#8221; remember that they once said that about all the areas where<br />
open source now dominates.</p>
<p>It&#8217;s just not smart to bet against the hackers.  Not smart at all.<br />
We generally end up having the last laugh on the naysayers. As recently<br />
as 1990, &#8220;serious analysts&#8221; laughed at the idea of ubiquitous Internet.<br />
As late as 1996, they said Unix was dead.  We showed them &#8212; and there<br />
are more of us now, with better tools, than ever.</p>
</li>
</ul>
<p>Steve is right that one of the most effective ways to head off bugs<br />
is to have a core group of professional engineers do a clean design.<br />
Where he&#8217;s mistaken is in believing this truth has anything to tell<br />
us about open vs. closed development.  Us open-source guys, it turns<br />
out, are <em>really good</em> at clean design.</p>
<p>This something to do with the fact that, as individuals, we tend to<br />
be exceptionally capable and self-motivated &#8212; an elite selected by<br />
dedication to the art of programming. It has more to do with not<br />
having managers and marketroids pissing in the soup constantly,<br />
telling us what tools to use, imposing insane deadlines, demanding<br />
endless checklist features that don&#8217;t actually benefit anyone.</p>
<p>But mostly it has to do with the ruthless, invaluable pressure of<br />
peer review &#8212; the knowledge that every design decision we make will<br />
be examined by thousands of people who may well be smarter than we<br />
are, and if we fail the test our effort will be pitilessly<br />
discarded. In that kind of environment, you get good or you get<br />
gone.</p>
<p><a href="http://enetation.co.uk/comments.php?user=esr&amp;commentid=79585067">Blogspot comments</a></p>
