MSNBC – Does Your iPod Play Favorites?: it’s entirely possible that nothing at all is amiss with the shuffle function. . . .
Life may indeed be random, and the iPod probably is, too. But we humans will always provide our own narratives and patterns to bring chaos under control. The fault, if there is any, lies not in shuffle but in ourselves.

This reminds me of a recent igNobel prize: “when people pay close attention to something, it’s all too easy to overlook anything else — even a woman in a gorilla suit.”

In the case of iPod/iTunes randomness, many people fail to see a pattern. Those people all use the same examples (the Birthday “Paradox” is one) to convince themselves that because human beings often misunderstand probabilities, iTunes (and the iPod) must be random. Kind of a reverse argument, ain’t it?
This has been discussed on blogs, on Slashdot, and on two MacSlash threads [1], [2]. Some people even want this thread to “die” (talk about anthropomorphism) yet add nothing of scientific value to the discussion. Many comments are derisive and condescending, coming from self-appointed experts in human cognition (apparently, as they seem to know so much about why we see patterns; they must have all attended the same PSY101 class or, maybe, had the same instructor for statistics in high school).
Thing is, it’s hard to find anyone doing an actual experiment (with null hypotheses) to in fact determine “how random” the algorithm in iTunes and/or the iPod really is.
As we know that no computer algorithm can generate true randomness (but maybeLava Lites can) and as it’s quite likely that the algorithm used in those devices isn’t the most advanced/complex one, it’s safe to assume that the degree of randomness we’re talking about isn’t as high as those deriders assume it is. I’m saying this without knowing whether or not there is such a measure for randomness but, surely, there must be some measure for entropy. That’d work!

What’s infuriating about this series of personal comments (“you see patterns because you don’t know stats!”) is that, assuming those condescending nay-sayers have any scientific background whatever, they should know better than that, just assuming that the algorithm really produces random series of numbers. Pffish! I’d like to be on their dissertation committees…
Why don’t they apply the scientific method? Ya know: observation, hypothesis, test… Kind of thing. It’s taught in some high schools too…
In this case, people observe patterns in a supposedly random series. One hypothesis is that these patterns are artefacts of the human tendency to see patterns everywhere. Fair enough. These people (or people with an experimental bend) could test this hypothesis and show us the results. If you want to do it, don’t test with 100 songs by a few artists. People who see these patterns have thousands of tracks by hundreds of artists in their collections*. Do it with a real world collection. In fact, do it with a collection to which tracks have been added at different times. Here’s why it matters:

The patterns that people see have to do with tracks from the same artist or, more broadly, with tracks that are similar in a more atypical way. Fine. One thing that might be common with these tracks is that they could have been added at the same time. If they have in fact been added at the same time, it’s probable that they reside on the same part of the hard drive, especially on an iPod to which things have been added in a very limited number of “sessions.” When one adds tracks to iTunes, the import process seems not to follow any obvious order so it’s likely that there are specific locations for tracks added in iTunes (on a computer or on the iPod). In this case, because of the way both iTunes and the iPod work, it’s quite possible that it means that they’re in the same folder. Yes, those “Fnn” folders that are supposed to be invisible but from which we can extract files if we want to. Isn’t it possible that tracks in the same folder have a higher probability of being played within a given sequence because of the way the randomization algorithm has been applied to them?

Now, a disclaimer of sorts. I do observe patterns in the way iTunes and my iPod select some tracks. We’re talking about tracks from two albums (out of several hundreds) playing alternatively (a track from album one, a track from album two, another track from album one…) for fifteen tracks. That’s pretty patterned to me. Not “intelligent.” Just patterned, clustered, not-so-random.
In fact, the other day, I noticed that my iPod played one song from one artist, then a song from another artist, then a song from the first artist. I “made myself a bet” (so to speak) that the track after the next would be a track by the first artist. Turns out that it was a song with the same artist involved but listed under another artist (so the “artist” tag wasn’t the same). I believe that I added all those tracks at the same time. Coincidence? Possibly. But it should be proven. “Real scientists don’t believe in coincidence.” Or some such.

It does feel awkward. In a nice way. Another time, I was listening to a song and, for no apparent reason (at least, not apparent to me), I started thinking about a song by a completely different artist. Turns out, that song was the very next song that played after the one to which I was listening. Very strange effect. Now, it’s possible that I always think about many different songs and that I noticed this occurrence because it appeared to “fulfill my prophecy” (the same way people see a conscious hand at play when one connection occurs between different events that have been associated in many different ways). Or it might be that I had heard these two songs in succession before and that hearing one made me think of the other. I tend to lean toward the second hypothesis because, the way my brain works (when it does work), I usually think about “the next song” based on sequences to which I’m used. For instance, if I made myself a specific playlist and I played it (in linear sequence) enough times, chances are that if I hear one of the tracks, I’ll think of the next track in that playlist even if the context is completely different. I’m probably not the only one who does this type of “stream of thought” association. But, you never know.

Anyhoo… I’m not saying the iTunes/iPod “shuffle” algorithm is smart/intelligent/sentient. But I do think that it generates some kind of cluster effect, the result of which sometimes feels spooky…

*My iPod (2G, 20GB) statistics:
101 Genres
644 Artists
703 Albums
3737 Tracks
Tracks by/with the artist of whom I recently had a cluster of songs: 55