30 lines
8.0 KiB
Plaintext
30 lines
8.0 KiB
Plaintext
The Smartphone Wars: More fun with statistics
|
|
<p>Today, a prediction about the timing of Android victory. But – more importantly – a discussion of the uses and perils of statistical extrapolation.</p>
|
|
<p><span id="more-3251"></span></p>
|
|
<p>I’ve used gnuplot to do a linear regression against comScore’s history of Android market share and project it into the future. Here it is:</p>
|
|
<p><a href="http://esr.ibiblio.org/wp-content/uploads/2011/05/android-predict-1.png"><img src="http://esr.ibiblio.org/wp-content/uploads/2011/05/android-predict-1.png" alt="" title="android-predict-1" width="640" height="480" class="alignnone size-full wp-image-3252" /></a></p>
|
|
<p>Looks very neat, doesn’t it? Shows Android crossing 50% U.S. market share right about the end of October. Happy news, if true. </p>
|
|
<p>But there are significant methodological issues about how far we should trust a graph like this. By exploring them, I hope to help my readers become a bit more informed about how to apply rational skepticism to statistical extrapolation. There’s an awful lot of lying with statistics going on out there, much of it on issues weightier than smartphone market share; it is good to learn how not to be fooled.</p>
|
|
<p>I will start by giving you the dead minimum you should accept for confidence in my statistical extrapolation – <a href="http://www.catb.org/esr/comscore/">full access</a> to my dataset and my analysis/visualization code. Deep suspicion would be justified if I did not. I could be hiding dishonest manipulation of the data, or I could simply have made mistakes. Without the ability to check my work, you can’t know – and if I denied you that ability, the safe bet would be that I have something to hide.</p>
|
|
<p>This may seem obvious, but a surprisingly large amount of science (especially politically-loaded science) is done under conditions of nondisclosure that could cover an awful lot of fimflam. Skepticism about such ‘science’ is not merely justified, it’s <em>required</em> – and the higher the public-policy stakes are, the more uncompromising the demand for full disclosure has to be.</p>
|
|
<p>But don’t forget that <em>I</em> am relying on comScore’s primary datasets and code, which I can’t see. The absolute most you can deduce from what I show you is that my extrapolation is correct if comScore’s numbers are correct, and your confidence in that can only be as strong as the value of the business comScore would lose if it got out that its numbers were erroneous or fudged. </p>
|
|
<p>But there are even more fundamental issues that still arise even if you assume that comScore and myself are both honest and without flaw.</p>
|
|
<p>One obvious reason to believe the graph is that the line of extrapolation looks like a pretty clean fit. Indeed, the residuals are quite small, and are comparable to the measurement errors one normally sees in large market surveys. But if we become too seduced by the goodness of the fit we risk missing a more fundamental question: <em>what reason do we have for believing that a linear fit is appropriate?</em></p>
|
|
<p>Suppose we had a theoretical model of why people buy smartphones that predicted Android’s market share would rise linearly over time as y = ax + b, but the theory didn’t give us the coefficients a and b. The rough linearity of the observed data would confirm this theory; we would then have justification for doing a linear fit to the observed data to get a and b and using that to extrapolate into the future.</p>
|
|
<p>But this isn’t our situation. We have no theory of what drives smartphone marketshare, or at least not one that yields an equation. We have no justification for believing that market shares will tend to rise and fall linearly; in fact, we’ve already <a href="http://www.catb.org/esr/comscore/">seen</a> that over the same period RIM’s certainly does not.</p>
|
|
<p>We’re actually worse off than this, because we’ve seen growth curves in natural systems before and know that neat linearity is rare. The most likely model is that customers pick smartphones by what they see others buying, crowdsourcing the job of evaluating products to each other and leading to a growth pattern that looks like the spread of a contagious disease. Growth by contagion in bounded systems tends to be not linear but <a href="http://en.wikipedia.org/wiki/Logistic_function">logistic</a>.</p>
|
|
<p>There is a theory that could explain linear growth, however. That is this: customers would be buying Androids faster if they could, but the available supply is only growing linearly (because it takes constant dollars for each additive increment of manufacturing capacity). We actually get some support for this from the <a href="http://www.catb.org/esr/comscore/#usercounts">userbase growth graph</a>, which could be well approximated by two linear segments joined by a slight change in slope.</p>
|
|
<p>In a slightly more elaborate version of such a theory, J. Random Consumer has set a strike price for getting an Android and is buying as soon as his strike price is crossed going downward. The supply of Androids available at any fixed price X, and thus the number of buyers, will also tend to rise proportionately to total manufacturing capacity, and thus to rise linearly.</p>
|
|
<p>Now we have justification for the following statement: If comScore’s numbers are accurate, and Android sales are mainly constrained by supply chain and manufacturing capacity, then a linear fit is appropriate, and it’s highly likely that Android will cross 50% share in late October.</p>
|
|
<p>Other theoretical models that predict linear growth of sales could be plugged in here. The point is that you really need to have such a model – and confirmation of it that is to some degree independent of a pretty graph – before a statistically-based forecast can be any better than numbers pulled out of a hat. Without such a model, applying linear regression or any other sort of curve fit imposes a shape that may look nice but will have no connection to causal reality. </p>
|
|
<p>Of course it could be that the linearity is an illusion. The <a href="http://www.catb.org/esr/comscore/#usercounts">Android userbase growth graph</a> actually looks like it could be power-law growth. If that’s true, my linear prediction will be too conservative. Alternatively, growth might be about to nose over into the saturation part of a logistic curve. There’s simply no way to know this from the data; you have to have a model underlying your curve-fit, an independently confirmed theory about the future.</p>
|
|
<p>Such a theory needn’t be super-elaborate to make useful predictions. For example, if we consider that fewer than 50% of cellphone users have converted to smartphones, its seems much less likely that smartphone sales are going to reach saturation in the next year. Under these conditions, the entire Android army would have to be afflicted by some Android-specific design or execution failure for sales growth to go sublinear. (Yes, this is possible; worst case, a junk-patent lawsuit leading to a temporary restraining order could be pretty bad in the U.S. market, even if it had little effect intertnationally.)</p>
|
|
<p>Let’s review our premises here:</p>
|
|
<ol>
|
|
<li>ESR has conducted an honest analysis which is not significantly compromised by errors.</li>
|
|
<li>comScore’s numbers are accurate.</li>
|
|
<li>Android sales growth is constrained primarily by supply or some unknown input with linear growth.</li>
|
|
<li>The Android Army is not going to be hit by some kind of huge gotcha like a massive design failure or patent TRO.</li>
|
|
</ol>
|
|
<p>My point right now isn’t to argue for any of these premises, just to point out how much model construction has to go on before a curve-fit means anything real. The data is what the data is, but the curve it’s fit to is laden with assumptions. Whenever someone throws a curve at you without being explicit about the assumptions and their failure points, be very, very wary of it.</p>
|