15 lines
4.1 KiB
Plaintext
15 lines
4.1 KiB
Plaintext
Pandora’s Biases
|
||
<p>Back in February I experimented with Pandora Radio and <a href="http://esr.ibiblio.org/?p=1670">loved it</a>…enough that I bought a subscription within a few days. It’s my background music now; I might never own an analog radio again.</p>
|
||
<p>For a while I ran around telling all my friends about how Pandora was the greatest thing! since sliced bread! you should try it! But I’ve stopped doing that, because I’ve learned that it doesn’t work as well for other people…starting with my wife. I think I know why, and it reveals an interesting failure mode of all such systems.</p>
|
||
<p><span id="more-1909"></span></p>
|
||
<p>Back in February I commented on my original post saying this:</p>
|
||
<blockquote><p>
|
||
From the reactions here I think it’s the case that some seeds and gene clusters are more productive than others under their similarity metric, and I seem to have picked one that’s at the good end of the distribution. I wonder why that is? I have a tentative guess that it’s because the stuff I like is complex and has lots of structure, so there are lots of traits sticking out of it.</p>
|
||
<p>There may also be a selection bias. The classification system was almost certainly designed by musicians and is certainly applied by musicians, so the traits it’s going to represent most effectively will be those that are foreground for people with analytical musical ears. And that describes me; a lot of the stuff I like could be truthfully tagged “only musicians listen to thisâ€.
|
||
</p></blockquote>
|
||
<p>60 days later the feedback I’m getting seems to confirm this pretty strongly. How well Pandora will work for someone seems to correlate closely with the distance of the center of their tastes from “stuff musicians like”. And I think this highlights a likely failure mode of all recommender systems based in a taxonomy.</p>
|
||
<p>That is, if you try to do an equivalent of the Music Genome Project for creative content type X, your natural pool of evaluators is <em>people who make content type X</em>. That pool is much smaller than, and may have different tastes than, most of the the X Genome Project’s potential audience.</p>
|
||
<p>But there’s a subtler and perhaps more important effect – not different tastes, but different feature filters. It’s not just that musicians like somewhat different music than non-musicians do, it’s that they hear and retain things non-musicians miss. As a personal example, my memory of electric-guitar solos I’ve heard more than once or twice is so precise that it includes pick-scrape noises and unintentional quarter-tone off-notes. I can still recall my bemusement when I finally figured how unusual that is – that most people have trouble hearing such things even when they’re cued to the timing and told what to listen for.</p>
|
||
<p>Thus: I think an in-built limitation of Pandora is that it will work well <em>if you have feature filters like a musician’s</em>. Actually, it’s worse than that – because my wife is a musician, but doesn’t hear music in the hyper-analytical way I do, and Pandora doesn’t work well for her. So, maybe, the key group is “musicians listening with their left ears”. Yes, this actually matters – it’s been shown in the lab that left-ear listening activates the analytical left brain. It makes sense; if you’re hiring people to analyze music, you’re likely to find unusually analytical musicians.</p>
|
||
<p>The larger point here is that all recommender systems dependent on hiring evaluators are likely to have the same problem. Even if you work at getting a broad selection of <em>taste</em> in the evaluators (say, by making an extra effort to hire people who understand country & western, or psychotronic films, or 19th-century penny dreadfuls) you’re likely to end up with a pool that has feature filters different from the general population – probably more analytical, finer-grained, pickier. This will leave your classification system with subtle biases, possibly ill-matched to the general population.</p>
|