8 Websites Control Over 30% Of Digg’s Technology Section

Posted Feb 12, 2009

Despite whatever Digg’s administrators tells us, Digg.com has a bias towards certain websites over others.  Why do I say this?  I have a Digg account where I submitted 477 stories and have gotten about 63 websites homepaged so that automatically gives me experience in knowing how the site works.  Below are several cases why I’m telling you that Digg is biased towards certain websites over others.

Case 1 (A lot of Diggs doesn’t meant you’ll get homepaged if the site isn’t “white listed” by Digg): In one case, one of the stories I submitted received over 280 Diggs and over 15 comments, but still wasn’t homepaged.  About 219 days ago, a story called “The Constitution Dies Tomorrow” received over 1,000 Diggs and over 100 comments, but still wasn’t homepaged.  It didn’t get homepaged until TechCrunch pointed it out and Digg probably didn’t want the negative PR from Arrington so they decided to add it to the homepage.

One time I submitted an article from NBA.com and it was homepaged with about 70-75 diggs and one comment.  Even though I submitted that article, I actually thought it made the Digg.com homepage undeservedly based on those numbers.  Those numbers don’t seem very community-focused to me.  I’ve seen a lot better written articles with a lot more Diggs and comments not get homepaged at all.

Case 2 (Crawl3.digg.internal): There have been several cases where people have noticed a mysterious connection on their server called Crawl3.digg.internal appeared shortly before their stories disappeared from Digg.com’s homepage. Pronet Advertising has pointed out that the internal Digg team is notorious for burying stories that they aren’t too fond of. And HMTK.com pointed out that crawl3.digg.internal showed up on his server around 10AM on March 5 when the story disappeared from the homepage.

Case 3 (Pulse2.com experience): Pulse2.com got homepaged once for less than one minute.  One of the articles I put together about 228 days ago explored the benefits of shouting stories and adding friends on Digg.  It was made popular, but never made it to the homepage.  I’ve never heard of a case where this has happened to someone before.  The story was called “How To Effectively Utilize Digg Shouts 101.”  I have reason to believe that crawl3.digg.internal buried it and blacklisted pulse2.com because this site has never been seen on the Digg frontpage since then.  This makes me believe that if you get touched with the crawl3.digg.internal once, you get blacklisted from making the Digg homepage ever again.

I tried writing all sorts of content that would be interesting and not teach people how to use Digg anymore, but it didn’t matter if I had over 250 diggs on a story and 10 comments, it would never be homepaged.  Even this story I wrote yesterday was submitted by Atomicpoet, got 184 diggs, and 15 comments but didn’t get homepaged at all.  If the same story was written by The New York Times, ReadWriteWeb, or Ars Technica, the odds of it getting homepaged would have been much higher.

And last, but not least:

Case 4 (8 Websites Control Over 30% of the Digg.com Technology page):  When you look at the navigation order at the top of Digg, you’ll notice that the Technology section comes first.  This is why I decided to analyze which websites in the Technology section make the homepage the most based on Digg’s bias towards them.  It turns out that the results were pretty staggering.  There are 8 websites that control over 30% of the Technology section homepage.

How did I figure this out?  I obviously had to pick a sample size first.  I decided about 8 days of data is a good enough sample size to base my statement.  So I went back about 8 days worth of pages and counted how many times a certain story from a specific website was homepaged.  I counted about 270 from the technology section were homepaged in the last 8 days.  94 (or 34.18%) of those stories were from the same 8 websites.  For example Ars Technica was homepaged about 17 times, AppleInsider about 16 times, and TorrentFreak about 16 times in the last 8 days for the Technology section.  Those three websites alone make up 18% of the Technology stories that were homepaged in the last 8 days.

Coincidentally around the same time I wrote this article, the Digg blog was writing an article about an update to the algorithm.  Apparently they thought it was hilarious to say that MC Hammer controls what stories get homepaged and which ones get buried.  Its too bad that me and Digg don’t share the same sense of humor. I think I speak for a lot of users when I demand more transparency from Digg.  How many times is a story getting buried?  When a story gets buried, how much does it set a story back?  Which stories are getting buried by the community and which ones are getting buried by internal staff?