Sneaking up on the forge problem
<p>I&#8217;ve written before about the <a href="http://esr.ibiblio.org/?p=1282">problems with today&#8217;s software-forge sites</a><a> &#8211; how they&#8217;re </a><a href="http://esr.ibiblio.org/?p=1295">craptacular piles of PHP driving direct SQL queries</a> with almost zero scriptability that become data jails for open-source projects.  I&#8217;ve <a href="http://esr.ibiblio.org/?p=1359">hinted</a> that I think there&#8217;s a potential solution based on <a href="http://roundup.sourceforge.net/">Roundup</a>, a brilliantly simple and powerful message queue manager disguised as a mere issue tracker.</p>
<p><span id="more-3931"></span></p>
<p>Two years ago I started an effort to address the data jail called &#8220;forgeplucker&#8221;.  This project has since stalled, partly due to problems (irony alert!) at the hosting site I put it on, and partly because there was a basic problem with it that I acknowledged at the time: it doesn&#8217;t do a lot of good to have working project state extractors if none of the forges are writing injectors.  Nobody has stepped up to do that, leaving forgeplucker a useful learning experience but something of a dead end in itself.</p>
<p>One of my regulars <a href="http://esr.ibiblio.org/?p=3927&#038;cpage=1#comment-338934">recently predicted</a> that I would do something dramatic about the forge problem within two years.  That prediction was correct; in fact, I&#8217;ve already begun the preparation work.</p>
<p>However, I&#8217;m not announcing a new forge.  Yet.  The path I&#8217;m pursuing is sneakier and more interesting than that.  It relies on the fact that there is a mature, stable, and field-tested implementation of Roundup already in existence in fact, the Python development group uses it as their issue tracker.</p>
<p>Roundup has a number of interesting properties.  One is that every capability in it can be scripted via a mail robot or an XML-RPC interface. Another is that the message queues are accessible through either a web-forum-like interface or email &#8211; content is separated from presentation. Most importantly, <em>it already works</em> &#8211; it&#8217;s got ten years of history, tested code, happy users, and a development community.</p>
<p>Some weeks ago I joined the Roundup development list, explaining my interest in using it as a platform for a future forge system.  As a first contribution I explained about <a href="http://www.catb.org/esr/reposurgeon/">reposurgeon</a> and offered to lift the project repo from Subversion to a DVCS; the project founder accepted this offer, and I have done the job.  As a nice  side effect, reposurgeon grew a nice list of new capabilities in the process, essentially everything in releases 1.3 to 1.10.</p>
<p>So now I&#8217;m a dev on the Roundup project, in fact an administrator (which was needed so I could do the repo manipulations).  My plan is to add certain new primitive capabilities to Roundup.  Each of these capabilities is orthogonal to the others and can be functionally justified on its own terms. Here are the big ones:</p>
<p><b>Extract/inject:</b>  Extract will dump the state of a Roundup instance as a big hunk of (yes) JSON; inject will take that JSON and use it to populate a live Roundup instance.  The standalone justification for this is that it can be used for tracker state backups.</p>
<p><b>Namespaces:</b> Presently, each Roundup instance hosts just one tracker (message queue).  It should be possible for a Roundup instance to host multiple named message queues.  The standalone justification for this is&#8230;itself.  </p>
<p><b>Network-wide identities:</b>   Presently, Roundup identities are login-name/password pairs. It should move to being based on email addresses coupled with ssh and PGP keys &#8211; network-wide credentials the way DVCSes handle identity.  The standalone justification for this is to reduce the number of credentials Roundup users need to manage.</p>
<p><b>A repository-management class.</b>  Roundup has a role-based permissions system.  A primitive object type could be added to Roundup that would use that permissions system to control access to a DVCS repository.</p>
<p>There&#8217;s no mystery intended about where this is going.  If all these features were successfully added to Roundup, it would have all the capabilities of a forge.  The message queues would span the functions of issue trackers and mailing lists.  The &#8220;forge&#8221; part would be some management logic for associating a group of message-queue namespaces and repository objects with a project name and giving them all one set of access permissions.</p>
<p> But I don&#8217;t want or need to promise a forge yet.  The plan is to add these primitive orthogonal features to Roundup. At each stage they&#8217;ll get actual testing in real deployments.  Rather than trying to design a forge now, what I aim to do is push these capabilities into Roundup until one of two things happens:</p>
<p>1. I hit a wall.  The underlying architecture isn&#8217;t robust enough to carry these features.  (I regard this as possible but very unlikely.)  If this happens, I have to think up another approach.</p>
<p>2. All these features are successfully integrated. <em>Then</em> a forge could be written as a thin layer over the enhanced Roundup &#8211; and, functionally speaking, it would nuke every existing forge from orbit. Just the complete XML-RPC scriptability alone would make it a category-killer.</p>
<p>I have not started coding yet, mainly because I have a writing project I&#8217;ve promised I&#8217;d finish before I dive into this. But the existence of Roundup will save <em>immense</em> amounts of coding and community-building.  The project has a ten-year history; I estimate it would take four to five person-years to duplicate the capabilities it already has in a stable and tested codebase.</p>
<p>With Roundup to start from, on the other hand, I think we could get to a world-beating platform for forge-building in nine months to a year.  I&#8217;m going to do extract/inject as my first bit, building on experience with ForgePlucker. One of my blog regulars, Susan Stewart aka HedgeMage, has already joined the Roundup project with the intention of doing the network-identity piece.  We&#8217;d love to have somebody working on implementing namespaces; all these things should be doable in parallel.</p>
<p>If you want to help, please join the roundup-devel mailing list, introduce yourself, and explain which part you want to work on. </p>
