Generative models and programming talent
<p>What, if anything, is the essential and unique talent of good programmers?</p>
<p><span id="more-316"></span></p>
<p>The best account of this I&#8217;ve seen was written by one Stuart Reges in 1992.  It has been interestingly corroborated by two more recent papers on CS education.  I think Reges was onto something, but he was missing a couple of concepts he needed and therefore could not state his theory as concisely or effectively as he might have. I&#8217;m going to restate his idea here, and (I think) do a somewhat better job of tying it into other kinds of knowledge than he does.</p>
<p>Reges tried to write an account of a skill that he called &#8220;CS IQ&#8221;. I observe that CS IQ has to do with the ability to construct what philosophers of science call &#8220;generative theories&#8221; (this is the concept Reges is missing).</p>
<p>Here is Reges&#8217;s 1992 account, lightly edited for concision and readability (mainly by adding some paragraph breaks) and with my own analysis inserted:</p>
<blockquote>
<p>[Good] programmers are able to &#8220;play computer&#8221; in their head (sometimes requiring the aid of a scrap of paper).  In other words, we have a model of exactly what the computer does when it executes each statement.  For any given program, we have a mental picture of the state the computer is in when execution begins, and we can simulate how that state changes as each statement executes.  This is rather abstract, so let me try to explain by giving a specific example.</p>
</blockquote>
<p>In the language of philosophy, Reges is saying that (good) programmers have a generative model of computing.  That is, they have a model that is rich in internal causal connections.  They can reason forward about how causal effects will ripple through the rest of the model when the  state of some part of it changes.  They can also reason backward about what sort of state change would be required to produce a specified effect.</p>
<blockquote>
<p>Let me tell a story that is typical of those I heard from the TAs who worked for me at the computing center.  A student comes up to the TA and says that his program isn&#8217;t working.  The numbers it prints out are all wrong.  The first number is twice what it should be, the second is four times what it should be,and the others are even more screwed up.  The student says, &#8220;Maybe I should divide this first number by 2 and the second by 4.  That would help, right?&#8221;</p>
<p>No, it wouldn&#8217;t, the TA explains.  The problem is not in the printing routine.  The problem is with the calculating routine. Modifying the printing routine will produce a program with TWO problems rather than one.  But the student doesn&#8217;t understand this (I claim because he isn&#8217;t reasoning about what state his program should be in as it executes various parts of the program).</p>
<p>The student goes away to work on it.  He comes back half an hour later and says he&#8217;s closer, but the numbers are still wrong.  The TA looks at it and seems puzzled by the fact that the first two numbers are right but the others don&#8217;t match.  &#8220;Oh,&#8221; the student explains, &#8220;I added those 2 lines of code you suggested to divide the first number by 2 and the second by 4.&#8221;  The TA points out that he didn&#8217;t suggest the lines of code, but the student just shrugs his shoulders and says, &#8220;Whatever.&#8221;</p>
<p>The TA endeavors to get the student to think about what change is necessary, but the student obviously doesn&#8217;t get it.  The TA has a long line of similarly confused students, so he suggests that the student go sit down and think through his calculating procedure and exactly what it&#8217;s supposed to be doing.</p>
<p>Half an hour later the student is back again.  &#8220;While I was looking over the calculating procedure, a friend of mine who is a CS major came by and said my loop was all screwed up.  I fixed it the way he suggested, but the numbers are still wrong.  The first number is half what it&#8217;s supposed to be and the second is one-fourth what it&#8217;s supposed to be, but the others are okay.&#8221;</p>
<p>The TA considers for a moment whether he should bring up the student on an honor code charge for receiving inappropriate help, but decides that it isn&#8217;t worth it (especially since that line of similarly confused students is now twice what it was an hour ago).  He asks the student whether he still has those lines of code in the printing routine that divide by 2 and 4 before printing.  &#8220;Oh yeah,&#8221; the student exclaims, &#8220;those lines you said I should put in.  That must be the problem.&#8221;  The TA once more politely points out that he didn&#8217;t suggest the two lines of code, but the student again shrugs and says, &#8220;Whatever.  Thanks, dude!&#8221;</p>
<p>The student in my hypothetical displays the classic mistake of treating symptoms rather than solving problems.  The student knows the program doesn&#8217;t work, so he tries to find a way to make it appear to work a little better.  As in my example, without a proper model of computation, such fixes are likely to make the program worse rather than better.  How can the student fix his program if he can&#8217;t reason in his head about what it is supposed to do versus what it is actually doing?  He can&#8217;t.</p>
</blockquote>
<p>In the language of philosophy, the failing student does not have (or even <em>seek</em>) a generative model of computing, and therefore has very little ability to see how the facts about his particular program are causally connected. His ability to reason forward from causes like the construction of his loop to effects like the observed output is weak; his ability to reason backwards from desired output to the construction of the loop is nonexistent.</p>
<blockquote>
<p>But for many people (I dare say for most people), they simply do not think of their program the way a programmer does.  As a result, it is impossible for a programmer to explain to such a person how to find the problem in their code.  I&#8217;m convinced after years of patiently trying to explain this to novices that most are just not used to thinking this way while a small group of other students seem to think this way automatically, without me having to explain it to them.</p>
</blockquote>
<p>Reges later wrote up his account more formally as <a href="http://www.cs.washington.edu/homes/reges/mystery">The Mystery of b := (b = false)</a>.</p>
<p>What is missing from Reges&#8217;s account of a &#8220;model of computation&#8221; is what the model held in a good programmer&#8217;s mind actually consists of: a rich set of causal relationships between possible states of the model.  (But I am pretty certain Reges would agree with this elaboration instantly.)</p>
<p>Reges&#8217;s implicit theory is this: good programmers, and students  who will become good programmers, have a generative model of computing. Poor programmers, and students who will become poor programmers, don&#8217;t. CS IQ consists of a talent for building generative models of computing.</p>
<p>Some recent work on the teaching of programming reinforce Reges&#8217;s point.  See, for example, <a href="http://www.cs.umu.se/~jubo/Meetings/ECOOP07/Submissions/MaEtAl_rev.pdf">Improving the Viability of Mental Models Held by Novice Programmers</a>. The authors, llike Reges, lack the notion of a generative model. But in considering how students succeed and fail, they note that students may hold different causal theories about the assignment operator in Java &mdash; and of course, only one of these theories is correct.</p>
<p>Another recent paper, <a href="http://www.cs.mdx.ac.uk/research/PhDArea/saeed/paper1.pdf">The camel has two humps</a>, draws a more general conclusion.  In this one, two British researchers correlated responses on an exam used to filter incoming students aiming to study CS.  They boldly claim to be able to predict a student&#8217;s success in CS courses <em>before students have had any contact with any programming language</em>.</p>
<p>Their exam included several questions about how assignments in computer languages will change variables, with multiple-choice answers implying different theories about how assignment works.  Theories included the correct one (a = b copies the value of b to a) and several incorrect ones (a = b moves the value of b to a, zeroing b; or a = b swaps the values of a and b).</p>
<p>The result:  The test-takers who went on to fail in the CS courses were the ones who applied different theories of assignment to different questions.  The ones who succeeded applied the same theory to each question; whether it was the correct theory <em>did not matter</em>!</p>
<p>In other words, the successful CS entrants were the ones who, before learning any CS, responded to the test questions by constructing a generative model of assignment in their heads.  The failures did not.  Whether the result was correct or incorrect, it was the predisposition to seek a generative theory that predicted success.</p>
<p>I have not linked to Reges&#8217;s original post for a reason.  It&#8217;s because Reges was using his argument about &#8220;CS IQ&#8221; as a stepping stone to a more contentious theory, which I&#8217;ll examine in a future post.  In the meantime, Reges&#8217;s theory has a consequence; it is, itself,  generative.  It is exactly the consequence that the authors of <cite>The camel has two humps</cite> actually observed; successful CS students are those who, given a set of facts, will instinctively seek a consistent generative model to connect them.</p>
<p>I&#8217;m going to coin a term.  Successful CS students are &#8220;model-seekers&#8221;.  Reges&#8217;s &#8220;CS IQ&#8221; is a measure of model-seeking tendency.</p>
<p>UPDATE: Stuart Reges has not disappeared; he&#8217;s now a Senior Lecturer in Computer Science and Engineering at the University of Washington.</p>
<p>UPDATE**2: Reges himself informs me that the authors of <cite>The camel has two humps</cite> have now withdrawn their strongest claims.  More experimental results refuted the theory that model-seeking is the only predictor of success, but they still believe it is an important factor and are looking for possible confounding variables.</p>
