Friday, August 14, 2009

Evaluating Weiss’ Top 50 List(s): Hip Hop In Decline?

For years now, lots of folks have been saying that hip hop is dead. I’ve always preferred to use the words “declining” or “dying” (notice the difference) or even “has cancer”. It might be a bit grimmer since it’s describing an ongoing, depressingly downward spiral as opposed to an occurrence of the past; but it’s more accurate in my book. And I might just be able to prove its validity!

I was honored to be a part of Jeff Weiss’ ‘Top 50 Rap Albums of the '00s’ project, featuring a roster of well-regarded hip hop experts whom I read, occasionally converse with, but most of all respect. I certainly didn’t agree with everybody’s lists, but I could understand where they were coming from. Soulja Boy, Hurricane Chris, Gorilla Zoe, Huey, Mims, Rich Boy – none of these guys were on anybody’s list. Not even at #50 just for kicks. Hip hop certainly has a wide spectrum, proof positive by the fact that no two lists looked identical (though of course there were many similarities – top ten entries to be specific). In short, not all of the contributors’ lists looked like my ideal list, but they were all reasonable. And that’s what makes them all so important – as the old saying goes, “the more, the merrier” – for a study I decided on toiling with: proving hip hop’s decline over the decade.

My hypothesis, as you could probably gather by now, was to prove that hip hop has declined in quality over the years. I went about trying to prove this by collecting the data from Weiss’ ‘Complete Voters Ballots’ page. First, I copy-and-pasted everybody’s list and spent several minutes removing the numbers in front of the entries (nearly 1,600 total). It was a strenuous task, but I did it while listening to Pete Rock & CL Smooth, so it wasn’t that bad. Next, I plugged them all into a Microsoft Excel spreadsheet and alphabetized them. The next step was to count each album and remove any duplicate rows on the spreadsheet. For instance, 50 Cent’s Get Rich or Die Tryin’ was chosen by twenty-one people, so I deleted 20 of the rows with identical content and put the number “21” to the right of the sole cell which read “50 Cent – Get Rich or Die Tryin’”. Are you following me? I did this for each album (over 450 albums in total)!

So now what I had was an Excel file with two columns: one for the artist name/album title, the other for the amount of times it was featured on somebody’s list (i.e. how many total “votes” it received). Now I had to do the real tough work. I had to Google search each and every album title to verify the year it was released. It was a strenuous task, but I did it while listening to Poor Righteous Teachers, so it wasn’t that bad. (Not so) soon afterwards, I had the two aforementioned columns, as well as a third column for all of the release date years. Next up, I used the Sort feature to arrange the entries by year.

Next, I created yet another column which served the purpose of counting the total number of the second column (the vote total per album) per year. To explain, here is an example: the year “2001” had 35 selected albums. And the total sum for all votes for all albums in the year 2001 was 138 (Jay-Z’s The Blueprint contributed to nearly one quarter of the total vote count for that year). I moved all of this new data (I no longer needed the artist name/album title column) to a newer, simpler Sheet which looked like this:

As you’ll notice, the years 1999 and 2000 are combined. This is because a few entries (a total of five, to be exact) from 1999 were included on some peoples’ lists. Albums like MF DOOM’s Operation Doomsday and Q-Tip’s Amplified, for instance, fit into this small category. I decided to throw them into the 2000 group. My rationale for doing this was that if I left 1999 as its own group, it wouldn’t be represented well as a full year in and of itself, barring other great 1999 releases that would have been in peoples’ consideration had the ballot explicitly accepted 1999 releases. Besides, the numbers are so marginal, it wouldn’t make much of a difference. I know that doesn’t sound very scientific, but for all intents and purposes it’s the most practical choice. I mention this all purely for the sake of clarity and honesty. Still, you may ask: “Doesn’t this tilt the count in your hypothesis’ favor?” No! I’ll explain soon enough!

Next up, I created two graphs to visualize the data tables above. Here is Figure 1A:

Figure 1A illustrates a steady rise in the amount of albums per year which were included in everyone’s shared/combined lists. Keep in mind that A) 1999 and 2000 are merged, and B) 2009 is not yet complete. A rise in the amount of albums seems to suggest a rise in the quantity of albums released. In other words, more albums, more choices/options to be included on the list. Below is a chart which aims to illustrate levels of quality per year (not quantity) of hip hop releases:

You might be asking: how can you measure “quality” with just numbers? Here’s my methodology for measuring quality in this test: if an album gets a lot of votes as opposed to another album with very few votes, the album with more votes earns a higher “quality” score than the other. Therefore albums like The Blueprint, which earned lots of votes (The Blueprint landed on all but one person’s list), are more “quality” then, say, Jay-Z’s The Dynasty: Roc La Familia which only earned six people’s votes. In other words, “quantity” equals the total number of albums per year, whereas “quality” equals the total votes of all albums per year.

These two graphs served a purpose, yes, but they didn’t fully satisfy me. I wanted to prove beyond a reasonable doubt that there was a strong correlation to be found here. I did. By dividing the decade into two segments, I could create an easier way to view the charts. I split the decade as follows: 1) 1999/2000 through 2004, and 2) 2005-2009. Below is a chart which displays the rise in quantity of releases from decade half one to decade half two:

In essence, Figure 2A is a simplified version of Figure 1A. You’ll notice that the number 218 (the total number of all albums selected between the release dates of 1999/2000 to 2004) equals the first half of numbers on Figure 1A and so on for the second half. The jump from 218 and 236 may seem insignificant, but it’s worth noting that that’s an 8.25% rise. Keep in mind that this is including A) the extra albums from 1999, and B) the fact that 2009 is technically represented as only a half of a year. Fixing this disadvantage would tilt the incline even more. This is what I was talking about a few paragraphs back. But this isn’t the graph that’s most telling. This is:

This is the decline I’ve been talking about and searching for (what can I say, I had a hunch). This chart displays the decline in votes for the releases of the second half of the decade. The collective of all votes for all selected albums released from 1999/2000 to 2004 is 881, compared to 2005 – 2009’s 676 vote total. That’s a 23.27% drop. Quite significant if you ask me! Even if you were to A) remove all votes for albums from 1999, and B) double the votes for 2009 (being generous, since 2009 is roughly only halfway through), we’d still be facing an 875 to 698 “win” for the first half of the decade. That’s a still-significant 20.23% slide. Now we might know why Dr. Dre is apprehensive about dropping Detox: “Never let me slip, ‘cause if I slip, then I'm slippin’”.

Again, let me break it down: the sum total of all votes by all 30+ participants in Weiss’ ballot showed a disparate preference for early-to-mid-2000’s hip hop as opposed to mid-to-late-2000’s hip hop. Hip hop has placed quantity over quality, plain and simple. So let’s be clear, once and for all: Hip Hop Is Dead Declining! What are you gonna do about it?

P.S. Even though I am Jewish and I do wear glasses, I don’t consider myself a statistics expert. Therefore, if you have any questions or concerns about my procedures, or just downright found a flaw with the way I looked at this data, please lemme know! Otherwise, Nate Silver, eat your heart out (j/k)!