* MP3 vs. CD: The Beer Test

Unless you’re a dog, a whale, or a computer, you’re not going to be able to tell the difference between a good MP3 and CD audio.

Think you’re an audiophile? I’ll bet you beers you can’t tell the difference between 256 Kbps VBR MP3s and CD audio in a blind test.

Track Selection

12 tracks have been selected from commercially sold, mass-produced audio CDs (i.e. “silver” discs – no CD-R, no online purchases, etc.). The primary criteria for track selection are length of track (2-4 minutes), availability (I have to own them), and pure subjectivity.

Selected tracks are:

Song Artist Album Genre
Cats & Dogs Howard Shore The Truth About Cats & Dogs (Soundtrack) Jazz
The Four Seasons: II. Largo Antonio Vivaldi Classical Christmas Favorites Holiday
Goldberg Variations Johann Sebastian Bach Tune Your Brain with Bach: Heal Classical
I Wonder As I Wander The Night Heron Consort A Celtic Celebration Holiday
I’m On Fire Bruce Springsteen Born In The U.S.A. Rock
No! They Might Be Giants No! Alternative, Children’s Music
The Nutcracker Suite – Dance of the Sugar Plum Fairy Peter Ilyich Tchaikovsky Classical Christmas Favorites Holiday
She’s Always A Woman Billy Joel The Stranger Rock
Surfin’ U.S.A. The Beach Boys Endless Summer Rock
Violin Concerto No. 2, K. 211, III Itzhak Perlman,Violine; Wiener Philharmoniker, James Lavine Mozart: Violin Concertos No. 2 & 4 Classical
We Will Rock You Queen ESPN Presents Stadium Anthems Rock
Zombie Zoo Tom Petty Full Moon Fever Rock

Order

Songs are ordered alphabetically by song title.

CD to MP3

MP3 tracks were imported to iTunes with these settings:

  • stereo bit rate = 256 Kbps Variable Bit Rate (VBR)
  • Quality = Highest
  • Sample Rate = 48 kHz
  • Channels = Audio
  • Stereo Mode = Normal
  • Smart Encoding Adjustments = Off
  • Filter Frequencies Below 10 Hz = On

CD tracks were imported to iTunes using the Apple Lossless Encoder.

Randomness

To determine which track comes first in a pairing, I rolled a six-sided die. If odd, the MP3 track comes first. If even, the CD track comes first.

Sound Level

I created a playlist with the 24 tracks and sound-leveled all tracks with iVolume v3.1.4.

CD Burning

A CD-R was burned with my iMac using iTunes. The CD drive is 24x read/write, Matshita DVD-R UJ-875, firmware revision DB09.

Listening

Using any stereo components desired, subject may listen to tracks on audio CD for for as long and as often as desired switching tracks at will, making notes, etc.

Subject must work alone and not seek opinions from others.

To avoid the possibility of finding differences in the original and compressed files, subject must agree not to rip any tracks from the CD or in a any way use a computer to analyze the CD.

Subject must use his/her ears only. No visual cues (including, but not limited to, graphic equalizers) may be used.

Identification

For each pair of tracks representing the same original CD track, the subject must identify whether the A track or the B track is the original.

Scoring

Based on pure chance, the odds for getting a specific number of tracks identified correctly are:

N Percentage
0 0.024%
1 0.293%
2 0.611%
3 5.371%
4 12.085%
5 19.336%
6 22.559%
7 19.336%
8 12.085%
9 5.371%
10 0.611%
11 0.293%
12 0.024%

What the results mean:

Number Correct Conclusion
5, 6, or 7 No better than random chance. Sorry, you’re not an audiophile.*
3, 4, 8, or 9 Better than chance, but not significant enough to make claims about the ability to distinguish between original and compressed files without further testing.*
0, 1, or 2 Statistically significant. However, since the subject identified most tracks incorrectly, there will be much mocking.*
10 or 11 Statistically significant. Subject can reliably distinguish between original and encoded files, may claim such, and may spend time re-ripping entire CD collection at a higher bitrate. Subject will be treated to beers at a mutually agreed upon establishment.
12 Accolades of auditory excellence. Subject may also have a cheeseburger.

*Plus you owe me beers.

Credits

In May 2005, I bragged about my ability to distinguish between 128 Kbps MP3s and CD audio. My MIT friend Dave Kaffine then challenged me with this test. His test was double-blind, used an HP-48SX for ordering the tracks, and use Perl scripts for the randomness. My test is not double-blind, used alphabetical order for the tracks, and used a six-sided die for randomness, so it’s admittedly less rigorous. But I’m no Dave, nobody is. When I took Dave’s test, I got 7 of 12 tracks correct, which is statistically insignificant. However, I maintain that I was certain (and could tell within five seconds) on at least a couple of tracks. I concluded that if I could tell the difference on one track, then I must re-rip my entire CD collection from 128 Kbps CBR to 256 Kbps VBR, which I did. So I declared moral victory but lost the bet.

I think I’ll be getting free beers from anyone that takes me up on this challenge.

Related Posts