20 lines
6.2 KiB
Plaintext
20 lines
6.2 KiB
Plaintext
Ground-truth documents
|
|
<p>Sometimes good terminology, by making a distinction that wasn’t easily articulated before, can be very clarifying. I was in an IRC conversation about software engineering with A&D regular HedgeMage earlier today and found myself inventing a term that I think may be useful: the ground-truth document.</p>
|
|
<p><span id="more-4532"></span></p>
|
|
<p>The context was this: HedgeMage has observed a lot of haphazard practices at software shops that have to deal with hardware interfaces. Very often these interfaces are poorly documented by the hardware vendor, with serious gaps and ambiguities in what little description they give you.</p>
|
|
<p>To cope with this, S.O.P. for most shops is what HedgeMage described as “observe what it’s doing, then throw inputs at it and observe what you get back, and try to puzzle out what the patterns are”. Which is OK, except that it’s also normal to go through this process <em>while trying to write production code</em>. </p>
|
|
<p>And that is a bad, bad mistake. The result is often code that sort of worked, once, but is buggy and unmaintainable because nobody actually remembers what their assumptions were at the time it was written. They’re baked into code in cryptic ways, and trying to fix problems is terrifying because it’s so hard to tell when a change will break an undocumented assumption.</p>
|
|
<p>There is way to avoid this kind of mess. It’s to write down your assumptions <em>before</em> you write code, and treat that document as the authority of which the code is an implementation.</p>
|
|
<p>Later in the IRC thread, HedgeMage explained this to someone else by saying “So given the example of the hearing device from my interview question, [the document] would show all the control codes we’ve been able to pass to the device and what the device does in response to those codes, but would leave out the three that the company insists are there but that the device doesn’t actually respond to.” I added “It would list those as ‘documented, but no response'”.</p>
|
|
<p>At the earlier point in the discussion when I first advocated writing one, I was referring to this thing as a “design document”. But then I realized, and said, that calling it a “design document” is a problem. Programmers often associate that term with waterfall-model practices in which they’re expected to implement a bloated specification that’s wildly out of contact with reality. The point of the kind of document I was trying to describe is that it’s totally in contact with reality and not trying to describe or mandate anything else.</p>
|
|
<p>Here is an example: <a href="http://catb.org/gpsd/AIVDM.html">AIVDM/AIVDO protocol decoding</a>. It describes the behavior of Marine AIS radios; I wrote it as preparation for coding the GPSD project’s AIS driver. It isn’t exactly or completely a hardware-interface specification, and some of its claims are derived from standards documents and not yet tested – but the point is that it <em>tells</em> you which claims have been tested and which have not. It also tells you where the observed behavior of AIS doesn’t match the standards.</p>
|
|
<p>Casting about semi-consciously for a way to distinguish this from a “design document”, I found one. What this is, is a “ground-truth document”.</p>
|
|
<p>The thing about ground-truth documents is that they don’t make promises, don’t erect requirements, and don’t talk about the future. They’re just the facts, ma’am. They describe what is, warts and all. Mine evolved into the best single reference on the AIS protocols anywhere, and has since been used as a spec by at least three decoder projects other than GPSD itself.</p>
|
|
<p>The practice that goes with this term is simple: always put your ground-truth document together <em>before</em> you start on production code (test tools to reverse-engineer the device are not production code). Maintain it with the code, treat it as the authority for how the code should behave, and when the code doesn’t behave that way treat the divergence as a bug. When your knowledge about how the device behaves changes, change the code second; <em>change the ground-truth document first</em>. (Of course you have it under version control, so you also have a history of your knowledge of the device.)</p>
|
|
<p>This a form of knowledge capture that will save you immense amounts of pain, hassle, and rework over the entire life cycle of your project. For even greater gains, write your ground-truth document in a form that can be machine-parsed and then generate as much code as you can directly from the specification tables. (Yes, GPSD does this. So does the X windows project).</p>
|
|
<p>The other thing <em>not</em> to do (besides starting on production code too soon) is to entangle the process of writing the ground-truth document with the process of writing the specifications for your software. Wishes, plans, and hopes don’t belong in this thing.</p>
|
|
<p>Ground-truth documents can also have other, more political uses besides knowledge capture. Having one can help you hold a balky vendor’s feet to the fire, or short-stop an attempt to pass the buck back to your team when it belongs elsewhere. “Yeah? You say that transfer should run at 30MB/s? Well, here’s exactly what happened when we shipped it the control code for high-speed mode.”</p>
|
|
<p>For best effect in this kind of situation you hand the vendor your test-jig software along with the ground-truth document in which you recorded the results of running it. (Yes, this is another reason to write your test tools well before you start on the production code.)</p>
|
|
<p>In extreme cases (and yes, I’ve seen this happen) you can wind up documenting things about the hardware that the vendor’s engineers as a group once knew but have partly or totally forgotten. This is good. It’s great negotiating leverage.</p>
|
|
<p>I broadcast this term and concept so that software development teams can use it to rethink their processes and do better work. Have fun with it, and stay safe out there!</p>
|