387 lines
24 KiB
Plaintext
387 lines
24 KiB
Plaintext
Right back at ya, Captain
|
|
<p>Last Saturday morning in San Diego I had breakfast with Steven den<br />
|
|
Beste, the redoubtable captain of <a href="http://www.denbeste.nu/">U.S.S. Clueless</a>. One of the<br />
|
|
side-effects of that meeting was a long <a href="http://www.denbeste.nu/cd_log_entries/2002/07/OpenSourcepart1.shtml"><br />
|
|
critique</a> of open-source development. Herewith my response.</p>
|
|
<p>Steve and I agree on the scaling problem that has pushed software<br />
|
|
development efforts to the ragged edge of what is sustainable even by<br />
|
|
corporations with lots of money. Software project sizes are roughly<br />
|
|
doubling every eighteen months, and for reasons Steve alluded to the<br />
|
|
expected bug count per thousand lines is actually rising.</p>
|
|
<p>My assertion is that software development has reached a scale at<br />
|
|
which (a) even large corporations can often no longer afford to field<br />
|
|
enough developers to be effective at today’s project scales, and (b)<br />
|
|
traditional methods of software quality assurance (ranging from formal<br />
|
|
methods to internal walkthroughs) are no longer effective. The only<br />
|
|
development organizations that seem to thrive on today’s complexity<br />
|
|
regime are open-source teams.</p>
|
|
<p>Note that I am not claiming that open source is a silver bullet for<br />
|
|
the software-complexity problem. There are no silver bullets, no<br />
|
|
permanent solutions. What I <em>am</em> claiming is that at the<br />
|
|
leading edge of large-scale software, closed-source development<br />
|
|
doesn’t work any more. The future belongs to open source plus<br />
|
|
whatever other practices and technologies we learn to use with<br />
|
|
it to develop at ever-higher scales of complexity.</p>
|
|
<p>Steve’s analysis of the open-source phenomenon is very intelligent,<br />
|
|
but doesn’t quite understand either the mode of organization, the<br />
|
|
associated technology, or the factional politics within the movement.<br />
|
|
Diagnostic of the slight disconnect is when he writes “For [the<br />
|
|
zealots], the only true “Open Source” is governed by the strong form<br />
|
|
of the GPL, and all other forms and licenses are harmful dilution of<br />
|
|
the concept.” In fact, the people he’s talking about reject the term<br />
|
|
“open source” entirely and insist on the ideologically-loaded term<br />
|
|
“free software”.</p>
|
|
<p>A more serious error is when he writes “It is plausible that an OSS<br />
|
|
project would require each participant to sign an NDA before being<br />
|
|
given access to the source.” It is <em>not</em> plausible. The licenses<br />
|
|
and community values of the open-source community would not permit this.<br />
|
|
His two bullet points characterizing open source are missing its most<br />
|
|
important characteristic: the entire practice is designed to facilitate<br />
|
|
scrutiny by people with no institutional or contract relationship to the core<br />
|
|
development team. The astringent effect of peer review by people who<br />
|
|
have <em>nothing to lose</em> by reporting bugs is precisely the<br />
|
|
point of the whole game.</p>
|
|
<p>Steve doesn’t undertand the importance or the power of this effect. This<br />
|
|
slightly skews his whole essay; much of it is talking past what open-source<br />
|
|
people do, rather than addressing us. He’s also unaware of a lot of the<br />
|
|
real-world evidence for the success of the method. Some of the things he<br />
|
|
thinks are technologically or economically impossible are actually being<br />
|
|
done, routinely.</p>
|
|
<p>He’s correct when he says that most contributors are self-selected and<br />
|
|
self-motivated. He overestimates the cost of training newbies, though. They<br />
|
|
self-train; normally, the first time a core developer hears from a newbie<br />
|
|
is typically when the newbie sends a patch — self-evidence that the newbie<br />
|
|
has <em>already</em> acquired a critical level of knowledge about the<br />
|
|
software. The “sink or swim” method turns out to work, and work well.</p>
|
|
<p>It’s incorrect to imply, as he does, that open-source development<br />
|
|
is unsustainable because the people doing it are flaky amateurs.<br />
|
|
Steve hasn’t absorbed the implications of the Boston Consulting<br />
|
|
Group study that shows that about 40% of contributors to core projects<br />
|
|
are professionals getting paid for working on open source by patrons<br />
|
|
who need to use the results. In fact, what the open-source community<br />
|
|
is evolving into is something very like a huge machine for bringing<br />
|
|
newbies into apprenticeship contact with experienced developers and<br />
|
|
professionalizing both groups.</p>
|
|
<p>He also writes “OSS by its nature tends to be reactive rather than<br />
|
|
predictive. It doesn’t look into the future, try to predict a problem<br />
|
|
which doesn’t exist now but will exist then, and be ready with a<br />
|
|
solution. Rather, it tends to see problems that exist now and work on<br />
|
|
solutions for them.” This is false — or, at any rate, no more true<br />
|
|
than it is for closed-source development.</p>
|
|
<p>The open-source community built the Web and the Internet before it<br />
|
|
had acquired a name for itself and full consciousness of its own<br />
|
|
practices. Today, the cutting-edge work in operating systems<br />
|
|
languages, desktop user interfaces, relational databases and many<br />
|
|
other areas is being done either within the open-source community or<br />
|
|
in cooperation with it by academics. These prodigious efforts of<br />
|
|
imagination dwarf any “prediction” produced by closed-source software<br />
|
|
development in the last two decades.</p>
|
|
<p>Steve’s “open source is reactive” claim strikes me as ironically<br />
|
|
funny, because I can remember when the standard knock on my crowd was<br />
|
|
that we’re great at innovation but can’t actually field product. How<br />
|
|
quickly they forget…</p>
|
|
<p>He’s right enough about the difficulty of planning and high cost<br />
|
|
of face-to-face meetings, though. These are real problems. It’s<br />
|
|
a testimony to the power of our practices that we manage to ship large<br />
|
|
volumes of high-quality software despite these obstacles.</p>
|
|
<p>What Steve called “player-killer” tactics have been tried — there<br />
|
|
was a famous incident a few years back in which a TCP-wrappers<br />
|
|
distribution was Trojaned. The crack was detected and the community<br />
|
|
warned within hours. The black hats don’t seem to bother trying this<br />
|
|
any more; our reaction time is too fast for that game to be very<br />
|
|
rewarding. The technical design of Linux helps here in ways that<br />
|
|
I won’t go into here — suffice it to say that it’s intrinsically<br />
|
|
much harder to get a Trojan to do anything interesting than it<br />
|
|
is under Windows or other single-user operating systems.</p>
|
|
<p>So far, the supply of open-source developers seems to be pretty<br />
|
|
elastic — we’re not limited much by lacking bodies. Other factors<br />
|
|
loom much larger; patents, the DMCA, intrinsically hard technical<br />
|
|
problems. I don’t understand why this is as well as I’d like to, but<br />
|
|
the facts are undeniable; the community is ten times the size my<br />
|
|
wildest high-end scenarios predicted a decade ago and seems to be<br />
|
|
growing <em>faster</em> as it gets larger.</p>
|
|
<p>Steve’s whole argument that open-source can’t win in embedded<br />
|
|
systems is very curious, since it predicts exactly the opposite of<br />
|
|
what is actually happening out there. Linux is taking over in<br />
|
|
embedded systems — in fact, many observers would say it has already<br />
|
|
won that space. If Steve had worked in the field within the last<br />
|
|
three years he would probably know this.</p>
|
|
<p>Here are some data about the demand; the only non-general-purpose<br />
|
|
open-source software magazine in existence is the Linux Embedded<br />
|
|
Systems Journal. Open-source embedded developers like Monta Vista<br />
|
|
Software are bucking the recession by growing like crazy. The first<br />
|
|
cell-phone prototype running entirely open-source software just<br />
|
|
entered beta testing.</p>
|
|
<p>I was in California to meet Steve partly because Real Networks<br />
|
|
wanted me to be on stage when they announced the open-sourcing of<br />
|
|
their RTSP engine. Their CEO, Rob Glaser, was quite frank about the<br />
|
|
immediate business reasons: they needed to get ports to forty<br />
|
|
different Nokia cellphones and just couldn’t figure out how to muster<br />
|
|
the resources for that short of inviting every stakeholder on the<br />
|
|
planet to hack the problem. Scaling bites. Hard.</p>
|
|
<p>In fact, some of the very characteristics that Steve thinks make<br />
|
|
embedded systems like cellphones safe for closed development seems to<br />
|
|
be the factors that are driving increased open-sourcing. The close<br />
|
|
tie to hardware actually <em>decreases</em> the value of secrecy,<br />
|
|
because it means the software is typically not easily re-usable by<br />
|
|
hardware competitors. Thus open sourcing is often a great way to<br />
|
|
recruit help from customer engineers without a real downside risk of<br />
|
|
plagiarism.</p>
|
|
<p>In fact, it’s an open secret in the industry that the most<br />
|
|
important reason most closed-source embedded and driver software<br />
|
|
remains closed is not nerves about plagiarism but fear of patent<br />
|
|
audits on the source code. Graphics-card manufacturers, in<br />
|
|
particular, routinely swipe patented techniques from their competitors<br />
|
|
and bury them in binaries. (This is generally believed to be the<br />
|
|
reason nVidia’s drivers aren’t open.)</p>
|
|
<p>Another trend that’s driving Linux and open-sourcing in embedded<br />
|
|
stuff is the shift from specialty embedded 8-bit processors to 32-bit<br />
|
|
chips with general-purpose architectures. Turns out the development<br />
|
|
costs for getting stuff to run on the 8-bit chips are sickeningly high<br />
|
|
and rising — partly because the few wizards who can do good work on<br />
|
|
that hardware are <em>expensive</em>. The incremental cost for<br />
|
|
smarter hardware has dropped a lot; it’s now cheaper to embed<br />
|
|
general-purpose chips running Linux because it means you have a<br />
|
|
larger, less expensive talent pool that can program them. Also,<br />
|
|
when your developers aren’t fighting hardware limits as hard,<br />
|
|
you get better time to market (which, as Steve observes, is<br />
|
|
critical).</p>
|
|
<p>Steve is right about the comparative difficulty of applying<br />
|
|
open-source methods to vertical applications. But the difficulty is<br />
|
|
only comparative; it’s happening anyway. The metalab archive carries<br />
|
|
a point-of-sale system for pizza parlors. I know of another case in<br />
|
|
which a Canadian auto dealership built specialized accounting software<br />
|
|
for their business and open-sourced it. The reasons? Same as usual;<br />
|
|
they wanted to lay off as much as possible of the development and<br />
|
|
maintainance cost on their competitors.</p>
|
|
<p>This is the same co-opetition logic that makes the Apache Software<br />
|
|
Foundation work — it’s just as powerful for vertical apps, though<br />
|
|
less obviously so. Each sponsoring company sees a higher payoff from<br />
|
|
having the software at a small fraction of the manpower cost for a<br />
|
|
complete in-house development. The method spreads risk in a way<br />
|
|
beneficial to all parties, too, because the ability of separate<br />
|
|
companies to sustain development tends to be uncorrelated — unless<br />
|
|
they <em>all</em> sink, the project endures.</p>
|
|
<p>The way to solve the problem of not exposing your business logic to<br />
|
|
competitors is to separate your app into an open-source engine and a<br />
|
|
bunch of declarative business-rule schemas that you keep secret.<br />
|
|
Databases work this way, and websites (the web pages and CGIs are the<br />
|
|
schema). Many vertical apps can be partitioned this way too — in<br />
|
|
fact, for things like tax-preparation software they almost have to be,<br />
|
|
because the complexity overhead of hacking executable code every time<br />
|
|
the rules change is too high.</p>
|
|
<p>Steve thinks the differences between Apache and Mozilla are bigger<br />
|
|
than they are. In fact, the core groups of <em>both</em> projects are<br />
|
|
full-time pros being funded by large users of the software.</p>
|
|
<p>So, let’s address Steve’s objections point by point:</p>
|
|
<p><em>For embedded software, OSS has the following problems:</em></p>
|
|
<ul>
|
|
<li>
|
|
<p><em>It can’t be scheduled; timely delivery can’t be relied<br />
|
|
on.</em></p>
|
|
<p>Timely delivery can’t be relied on for <em>any</em> software; see<br />
|
|
De Marco and Lister’s excellent book <cite><a href="http://www.dorsethouse.com/books/pw.html">Peopleware: Productive<br />
|
|
Projects and Teams</a></cite> on the delusion of deadlines, especially<br />
|
|
the empirical evidence that the “wake me up when it’s done” strategy<br />
|
|
of not setting them actually gets your project done faster (also the<br />
|
|
implication of a recent Harvard Business School study of software<br />
|
|
project outcomes).</p>
|
|
<p>Open source is at least not noticeably worse than closed-source on this<br />
|
|
axis. Arguably it’s better, because the rapid release cycles allow users<br />
|
|
to pick up on project results as soon as they’re good enough.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>Debugging requires access to custom hardware which usually<br />
|
|
can’t easily be accessed across the net.</em></p>
|
|
<p>There aren’t good solutions to this problem yet, but the increasing<br />
|
|
use of “overpowered” 32-bit processors using standard busses is<br />
|
|
tending to reduce it in scope. The development tools and interface<br />
|
|
hardware used in embedded stuff are rapidly getting more generic and closer<br />
|
|
to what’s used in general-purpose computers.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>Active participation even for junior people requires substantial<br />
|
|
amounts of project-specific knowledge which isn’t easily acquired,<br />
|
|
especially remotely.</em></p>
|
|
<p>This one puzzles me, because I think Steve ought to be right about<br />
|
|
it — but I’m not hearing the kinds of noises that I’d hear if it were<br />
|
|
slowing down the move to Linux and open source significantly.</p>
|
|
<p>At least part of the answer is that embedded-systems work is<br />
|
|
getting de-skilled in a particular sense — more of it’s being done by<br />
|
|
application specialists who are training up to the required level of<br />
|
|
programming, rather than programmers who have acquired expensive<br />
|
|
application-specific knowledge.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>A great deal of proprietary information is usually involved in<br />
|
|
the process, and if that’s released the company can be seriously<br />
|
|
harmed.</em></p>
|
|
<p>It’s a question of tradeoffs. As RealNetworks found out when<br />
|
|
costing its Nokia contract, the choice is increasingly between giving<br />
|
|
up control of some of your proprietary IP and being too resource-bound<br />
|
|
to ship at all.</p>
|
|
<p>There is no market for secrecy. There’s a market for product. If<br />
|
|
you can’t ship product, or your customers aren’t confident that you<br />
|
|
can maintain it after shipping, all that proprietary IP amounts to is<br />
|
|
a millstone around your neck.</p>
|
|
<p>There will be more stories like RTSP in the future. Count on it.<br />
|
|
In fact, the day will come when most of your contract partners simply<br />
|
|
won’t accept the business risks of having someone else hold<br />
|
|
proprietary rights on the embedded software they use.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>It’s nearly impossible to do embedded software without<br />
|
|
common impromptu face-to-face meetings with co-workers, either to ask<br />
|
|
questions or to brainstorm. Doing this electronically is sufficiently<br />
|
|
different as to not be practical.</em></p>
|
|
<p>Yeah. They used to think that about operating systems, too. Obviously<br />
|
|
the Linux kernel is impossible, and therefore doesn’t exist.</p>
|
|
<p>(At which point Oolon Colluphid disappeared in a puff of logic.)</p>
|
|
</li>
|
|
</ul>
|
|
<p><em>For vertical apps, the objections are:</em></p>
|
|
<ul>
|
|
<li>
|
|
<p><em>Security, security, security. You want me to trust my<br />
|
|
billing system to code written by anyone who happens to come along and<br />
|
|
volunteer to work on it, without any kind of check of credentials or<br />
|
|
checks on trustworthiness?</em></p>
|
|
<p>One of the lessons the business world has been absorbing is that<br />
|
|
open-source projects are dramatically <em>more</em> secure than their<br />
|
|
closed-source competition — anybody who compares the Bugtraq records<br />
|
|
on Apache vs. ISS defacements, or Linux vs. Windows remote exploits,<br />
|
|
will notice that real fast.</p>
|
|
<p>It’s not hard to understand why this is — I’ve found that even<br />
|
|
corporate executives grok the theory pretty quickly. I won’t do the whole<br />
|
|
argument here, but this article on <a href="http://www.wikipedia.com/wiki/Kerckhoffs'+law">Kerckhoff’s<br />
|
|
Law</a> holds the crucial clue. When you rely on the obscurity of source<br />
|
|
code for security, it means that the bad guys find the bugs faster than<br />
|
|
you can plug them — there are more of them, and they have entropy on<br />
|
|
their side. Open source evens the odds for the good guys.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>Recruitment: for most of the kind of people involved in<br />
|
|
OSS, vertical apps are boring. (Unless they want to figure out how to<br />
|
|
steal from it.)</em></p>
|
|
<p>This remains a problem. On the other hand, open source makes it<br />
|
|
easier to train domain specialists to be good enough programmers to<br />
|
|
get the job done. It’s easier for physicists to learn to hack than<br />
|
|
it is for hackers to learn physics.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>It takes a lot of knowledge of the specific aspects of the<br />
|
|
problem to make a significant contribution, which means things like<br />
|
|
observing the actual process of guests checking in at the front desk<br />
|
|
of the hotel.</em></p>
|
|
<p>This just reinforces the tendency for vertical-app developers to be<br />
|
|
obsessives about something else who learn to program, rather than obsessives<br />
|
|
about programming who learn something else.</p>
|
|
<p>Professional programmers tend to bridle at this thought. Well, better<br />
|
|
learn to live with it. As software becomes more pervasive, the amount<br />
|
|
of it done by application-specialist “amateurs” is going to increase.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>The industry is full of horror stories of vertical apps<br />
|
|
which ran badly over budget and over schedule; the idea scares the<br />
|
|
hell out of business people. They’re unlikely to be very enthused by<br />
|
|
the use of a process which by its nature *cannot* be reliably<br />
|
|
scheduled. (Remember that Mozilla ran two years long.)</em></p>
|
|
<p>Schedules — and the belief that deadlines make software happen<br />
|
|
faster — are a delusion in the mind of management, one not supported<br />
|
|
by the actual evidence about project outcomes. This delusion is<br />
|
|
so entrenched that managers fail to interpret the 70% rate of<br />
|
|
project failures correctly. It’s as if people were so determined<br />
|
|
to believe the Earth is flat that they ignore what their eyes tell<br />
|
|
them when ships sink over the horizon.</p>
|
|
<p><em>No</em> software larger than toy programs can be scheduled.<br />
|
|
Tactics aimed at doing so normally have the actual effect of<br />
|
|
<em>increasing</em> the time to market. `Aggressive’ schedules<br />
|
|
effectively guarantee failure. The sooner we learn these objective<br />
|
|
truths, and that the illusion of control that schedules give is not<br />
|
|
worth the real costs, the sooner rates of outright project failure<br />
|
|
will dip below 70%.</p>
|
|
<p>Go read <cite>Peopleware</cite>. <em>Now</em>.</p>
|
|
</li>
|
|
</ul>
|
|
<p><em>For short life apps:</em></p>
|
|
<ul>
|
|
<li>
|
|
<p><em>Schedule is everything. If you’re six months late, you’re dead.</em></p>
|
|
<p>See above. There are reasons open sourcing is less applicable to short-life<br />
|
|
applications, but this turns out not to be one of them.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>Secrecy is everything else. If you’re on time but your<br />
|
|
competitor knows what you’re doing a year ahead, he’ll wipe you<br />
|
|
out.</em></p>
|
|
<p>This argument has more force for short-life apps than for Steve’s other<br />
|
|
categories, but remember that increasingly the alternative to open source<br />
|
|
is not being able to ship at all. Your competitor is in the same boat<br />
|
|
you are.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>How do you make money selling what anyone can get for free<br />
|
|
from any developer? If your product was developed out in the open, who<br />
|
|
exactly buys it afterwards?</em></p>
|
|
<p>Steve has a stronger point here. It’s one that people used to<br />
|
|
think applied to almost all software, but which turns out to be mainly<br />
|
|
a problem for short-life apps. Actually the distinguishing<br />
|
|
characteristic isn’t expected lifetime per se, but something<br />
|
|
correlated with it — whether the product needs continued downstream<br />
|
|
work (maintainance and upgrades) or not.</p>
|
|
<p>Long-life, high-maintainance apps create niches for service businesses.<br />
|
|
That’s the main way you make money in an open-source world. It’s<br />
|
|
harder to make that work with a short-life app. Sometimes it’s<br />
|
|
impossible. Life is hard.</p>
|
|
</li>
|
|
</ul>
|
|
<p><em>For long life apps:</em></p>
|
|
<ul>
|
|
<li>
|
|
<p><em>Will the participants be willing to work on what our<br />
|
|
marketing analysis says we need, or will they insist that they know<br />
|
|
what is required and try to add that instead? We don’t need feature<br />
|
|
creep, or people trying to change the direction we’re moving.</em></p>
|
|
<p>In open-source projects, the function of “marketing analysis” tends to<br />
|
|
be taken be direct interaction with the user community. We find we<br />
|
|
do better work without a bunch of marketroids getting between us and<br />
|
|
our customers.</p>
|
|
</li>
|
|
<li>
|
|
<p><em>There is major learning curve involved in making a<br />
|
|
reasonable contribution to these kinds of programs; you don’t learn<br />
|
|
how a circuit board router works in a few days of study. In most cases<br />
|
|
you have to be conversant with the way that the package’s customers do<br />
|
|
what they do, and most programmers don’t know these things and can’t<br />
|
|
easily learn them.</em></p>
|
|
<p>See my previous remarks about application specialists and the<br />
|
|
democratization of programming. And every time you’re tempted to<br />
|
|
say “But they couldn’t possibly get away with that in application<br />
|
|
area X” remember that they once said that about all the areas where<br />
|
|
open source now dominates.</p>
|
|
<p>It’s just not smart to bet against the hackers. Not smart at all.<br />
|
|
We generally end up having the last laugh on the naysayers. As recently<br />
|
|
as 1990, “serious analysts” laughed at the idea of ubiquitous Internet.<br />
|
|
As late as 1996, they said Unix was dead. We showed them — and there<br />
|
|
are more of us now, with better tools, than ever.</p>
|
|
</li>
|
|
</ul>
|
|
<p>Steve is right that one of the most effective ways to head off bugs<br />
|
|
is to have a core group of professional engineers do a clean design.<br />
|
|
Where he’s mistaken is in believing this truth has anything to tell<br />
|
|
us about open vs. closed development. Us open-source guys, it turns<br />
|
|
out, are <em>really good</em> at clean design.</p>
|
|
<p>This something to do with the fact that, as individuals, we tend to<br />
|
|
be exceptionally capable and self-motivated — an elite selected by<br />
|
|
dedication to the art of programming. It has more to do with not<br />
|
|
having managers and marketroids pissing in the soup constantly,<br />
|
|
telling us what tools to use, imposing insane deadlines, demanding<br />
|
|
endless checklist features that don’t actually benefit anyone.</p>
|
|
<p>But mostly it has to do with the ruthless, invaluable pressure of<br />
|
|
peer review — the knowledge that every design decision we make will<br />
|
|
be examined by thousands of people who may well be smarter than we<br />
|
|
are, and if we fail the test our effort will be pitilessly<br />
|
|
discarded. In that kind of environment, you get good or you get<br />
|
|
gone.</p>
|
|
<p><a href="http://enetation.co.uk/comments.php?user=esr&commentid=79585067">Blogspot comments</a></p>
|