Right back at ya, Captain

Last Saturday morning in San Diego I had breakfast with Steven den
Beste, the redoubtable captain of U.S.S. Clueless. One of the
side-effects of that meeting was a long
critique of open-source development. Herewith my response.

Steve and I agree on the scaling problem that has pushed software
development efforts to the ragged edge of what is sustainable even by
corporations with lots of money. Software project sizes are roughly
doubling every eighteen months, and for reasons Steve alluded to the
expected bug count per thousand lines is actually rising.

My assertion is that software development has reached a scale at
which (a) even large corporations can often no longer afford to field
enough developers to be effective at today’s project scales, and (b)
traditional methods of software quality assurance (ranging from formal
methods to internal walkthroughs) are no longer effective. The only
development organizations that seem to thrive on today’s complexity
regime are open-source teams.

Note that I am not claiming that open source is a silver bullet for
the software-complexity problem. There are no silver bullets, no
permanent solutions. What I am claiming is that at the
leading edge of large-scale software, closed-source development
doesn’t work any more. The future belongs to open source plus
whatever other practices and technologies we learn to use with
it to develop at ever-higher scales of complexity.

Steve’s analysis of the open-source phenomenon is very intelligent,
but doesn’t quite understand either the mode of organization, the
associated technology, or the factional politics within the movement.
Diagnostic of the slight disconnect is when he writes “For [the
zealots], the only true “Open Source” is governed by the strong form
of the GPL, and all other forms and licenses are harmful dilution of
the concept.” In fact, the people he’s talking about reject the term
“open source” entirely and insist on the ideologically-loaded term
“free software”.

A more serious error is when he writes “It is plausible that an OSS
project would require each participant to sign an NDA before being
given access to the source.” It is not plausible. The licenses
and community values of the open-source community would not permit this.
His two bullet points characterizing open source are missing its most
important characteristic: the entire practice is designed to facilitate
scrutiny by people with no institutional or contract relationship to the core
development team. The astringent effect of peer review by people who
have nothing to lose by reporting bugs is precisely the
point of the whole game.

Steve doesn’t undertand the importance or the power of this effect. This
slightly skews his whole essay; much of it is talking past what open-source
people do, rather than addressing us. He’s also unaware of a lot of the
real-world evidence for the success of the method. Some of the things he
thinks are technologically or economically impossible are actually being
done, routinely.

He’s correct when he says that most contributors are self-selected and
self-motivated. He overestimates the cost of training newbies, though. They
self-train; normally, the first time a core developer hears from a newbie
is typically when the newbie sends a patch — self-evidence that the newbie
has already acquired a critical level of knowledge about the
software. The “sink or swim” method turns out to work, and work well.

It’s incorrect to imply, as he does, that open-source development
is unsustainable because the people doing it are flaky amateurs.
Steve hasn’t absorbed the implications of the Boston Consulting
Group study that shows that about 40% of contributors to core projects
are professionals getting paid for working on open source by patrons
who need to use the results. In fact, what the open-source community
is evolving into is something very like a huge machine for bringing
newbies into apprenticeship contact with experienced developers and
professionalizing both groups.

He also writes “OSS by its nature tends to be reactive rather than
predictive. It doesn’t look into the future, try to predict a problem
which doesn’t exist now but will exist then, and be ready with a
solution. Rather, it tends to see problems that exist now and work on
solutions for them.” This is false — or, at any rate, no more true
than it is for closed-source development.

The open-source community built the Web and the Internet before it
had acquired a name for itself and full consciousness of its own
practices. Today, the cutting-edge work in operating systems
languages, desktop user interfaces, relational databases and many
other areas is being done either within the open-source community or
in cooperation with it by academics. These prodigious efforts of
imagination dwarf any “prediction” produced by closed-source software
development in the last two decades.

Steve’s “open source is reactive” claim strikes me as ironically
funny, because I can remember when the standard knock on my crowd was
that we’re great at innovation but can’t actually field product. How
quickly they forget…

He’s right enough about the difficulty of planning and high cost
of face-to-face meetings, though. These are real problems. It’s
a testimony to the power of our practices that we manage to ship large
volumes of high-quality software despite these obstacles.

What Steve called “player-killer” tactics have been tried — there
was a famous incident a few years back in which a TCP-wrappers
distribution was Trojaned. The crack was detected and the community
warned within hours. The black hats don’t seem to bother trying this
any more; our reaction time is too fast for that game to be very
rewarding. The technical design of Linux helps here in ways that
I won’t go into here — suffice it to say that it’s intrinsically
much harder to get a Trojan to do anything interesting than it
is under Windows or other single-user operating systems.

So far, the supply of open-source developers seems to be pretty
elastic — we’re not limited much by lacking bodies. Other factors
loom much larger; patents, the DMCA, intrinsically hard technical
problems. I don’t understand why this is as well as I’d like to, but
the facts are undeniable; the community is ten times the size my
wildest high-end scenarios predicted a decade ago and seems to be
growing faster as it gets larger.

Steve’s whole argument that open-source can’t win in embedded
systems is very curious, since it predicts exactly the opposite of
what is actually happening out there. Linux is taking over in
embedded systems — in fact, many observers would say it has already
won that space. If Steve had worked in the field within the last
three years he would probably know this.

Here are some data about the demand; the only non-general-purpose
open-source software magazine in existence is the Linux Embedded
Systems Journal. Open-source embedded developers like Monta Vista
Software are bucking the recession by growing like crazy. The first
cell-phone prototype running entirely open-source software just
entered beta testing.

I was in California to meet Steve partly because Real Networks
wanted me to be on stage when they announced the open-sourcing of
their RTSP engine. Their CEO, Rob Glaser, was quite frank about the
immediate business reasons: they needed to get ports to forty
different Nokia cellphones and just couldn’t figure out how to muster
the resources for that short of inviting every stakeholder on the
planet to hack the problem. Scaling bites. Hard.

In fact, some of the very characteristics that Steve thinks make
embedded systems like cellphones safe for closed development seems to
be the factors that are driving increased open-sourcing. The close
tie to hardware actually decreases the value of secrecy,
because it means the software is typically not easily re-usable by
hardware competitors. Thus open sourcing is often a great way to
recruit help from customer engineers without a real downside risk of
plagiarism.

In fact, it’s an open secret in the industry that the most
important reason most closed-source embedded and driver software
remains closed is not nerves about plagiarism but fear of patent
audits on the source code. Graphics-card manufacturers, in
particular, routinely swipe patented techniques from their competitors
and bury them in binaries. (This is generally believed to be the
reason nVidia’s drivers aren’t open.)

Another trend that’s driving Linux and open-sourcing in embedded
stuff is the shift from specialty embedded 8-bit processors to 32-bit
chips with general-purpose architectures. Turns out the development
costs for getting stuff to run on the 8-bit chips are sickeningly high
and rising — partly because the few wizards who can do good work on
that hardware are expensive. The incremental cost for
smarter hardware has dropped a lot; it’s now cheaper to embed
general-purpose chips running Linux because it means you have a
larger, less expensive talent pool that can program them. Also,
when your developers aren’t fighting hardware limits as hard,
you get better time to market (which, as Steve observes, is
critical).

Steve is right about the comparative difficulty of applying
open-source methods to vertical applications. But the difficulty is
only comparative; it’s happening anyway. The metalab archive carries
a point-of-sale system for pizza parlors. I know of another case in
which a Canadian auto dealership built specialized accounting software
for their business and open-sourced it. The reasons? Same as usual;
they wanted to lay off as much as possible of the development and
maintainance cost on their competitors.

This is the same co-opetition logic that makes the Apache Software
Foundation work — it’s just as powerful for vertical apps, though
less obviously so. Each sponsoring company sees a higher payoff from
having the software at a small fraction of the manpower cost for a
complete in-house development. The method spreads risk in a way
beneficial to all parties, too, because the ability of separate
companies to sustain development tends to be uncorrelated — unless
they all sink, the project endures.

The way to solve the problem of not exposing your business logic to
competitors is to separate your app into an open-source engine and a
bunch of declarative business-rule schemas that you keep secret.
Databases work this way, and websites (the web pages and CGIs are the
schema). Many vertical apps can be partitioned this way too — in
fact, for things like tax-preparation software they almost have to be,
because the complexity overhead of hacking executable code every time
the rules change is too high.

Steve thinks the differences between Apache and Mozilla are bigger
than they are. In fact, the core groups of both projects are
full-time pros being funded by large users of the software.

So, let’s address Steve’s objections point by point:

For embedded software, OSS has the following problems:

For vertical apps, the objections are:

For short life apps:

For long life apps:

Steve is right that one of the most effective ways to head off bugs
is to have a core group of professional engineers do a clean design.
Where he’s mistaken is in believing this truth has anything to tell
us about open vs. closed development. Us open-source guys, it turns
out, are really good at clean design.

This something to do with the fact that, as individuals, we tend to
be exceptionally capable and self-motivated — an elite selected by
dedication to the art of programming. It has more to do with not
having managers and marketroids pissing in the soup constantly,
telling us what tools to use, imposing insane deadlines, demanding
endless checklist features that don’t actually benefit anyone.

But mostly it has to do with the ruthless, invaluable pressure of
peer review — the knowledge that every design decision we make will
be examined by thousands of people who may well be smarter than we
are, and if we fail the test our effort will be pitilessly
discarded. In that kind of environment, you get good or you get
gone.

Blogspot comments