Backlit header graphic

Hyperlink Poison: The 'Nofollow' Story

"If there was no existing mechanism through which Silicon Valley could justify unfair and anti-competitive self-preferencing, Google could be relied upon to invent one."

Today, they're everywhere. Nofollow-attributed hyperlinks have become the web publisher's default means of linking out to external sites. But the "nofollow" property was never meant to be used by all and sundry. It was designed by Google as a self-preferencing scheme for the Silicon Valley collective.

We, the plebs, were not supposed to discover the dark secret of the nofollow link, or use its poison against Silicon Valley as Silicon Valley set out to use it against us. But in the end, we wised up. And our eventual retaliation has rendered the nofollow attribute entirely useless, to the point that Google had to quietly replace it with a new attribute called ugc. The damage, nevertheless, was done...

NOFOLLOW BACKGROUND

Once upon a time, long before anyone considered Google to be evil, Google devised an evil plan. A plan that sought to establish a fantastically biased search ranking tweak that would slowly lock small publishers out of the visible search results, whilst preferencing massive, Silicon Valley silos. The plan was simple: create a two-tier hyperlink landscape, in which the links pointing to large platforms carried a search ranking advantage, and the links pointing to small publishers did not.

To explore this cunning plot, we need to time-travel back to the year 2004, when Google organised the self-preferencing scheme with a collective of ICSPs (Internet Computer Service Providers), behind closed doors. But before we dip into the birth of the scheme, let's see how it works...

HOW NOFOLLOW WORKS

Nofollow is a single-word attribute added to the raw HTML page code of a hyperlink. If you're not a coder you'll probably never see it. It's invisible to page visitors unless they consciously use their browser's technical tools to inspect the source code behind the links. Here's the HTML code for an ordinary link. This default-format link is known as a "dofollow" link, although it contains no actual "dofollow" attribute:

<a href="http://www.example.com/">example</a>

As specified in Google's original plan, that link would carry a search ranking advantage.

And now here's how the same link looks with a nofollow attribute added:

<a href="http://www.example.com/" rel="nofollow">example</a>

As specified in Google's original plan, that link would NOT carry a search ranking advantage.

So, simply by adding rel="nofollow" into a link's HTML code, the domain harbouring that link could effectively poison it. Ensure that the site they were linking to would not receive the search ranking benefit that a regular hyperlink would normally carry.

A spam link in a comments thread was NOT deemed to be spam if it pointed to Silicon Valley's own property. And that is why Big Tech took the apparently illogical step of poisoning individually selectable links, rather than the whole user-generated block. This wasn't anti-spam, it was anti-competition.

RIGGING THE BOARD

So what makes that unfair?

The nofollow attribute was introduced as an invisible, technical modification that people outside the tech industry would seldom know about, let alone understand. By that token alone it gave a search ranking advantage to the tech industry. In Google's release documentation, circa January 2005, there appeared this telling nugget:

"We hope the web software community will quickly adopt this attribute and we're pleased that a number of blog software makers have already signed on:

Brad Fitzpatrick - LiveJournal
Dave Winer - Scripting News
Anil Dash - Six Apart
Steve Jenson - Blogger
Matt Mullenweg - WordPress
Stewart Butterfield - Flickr
Anthony Batt - Buzznet
David Czarnecki - blojsom
Rael Dornfest - Blosxom
Mike Torres - MSN Spaces"

First, you notice that it was specifically the web software community, and not the wider public, that Google were allowing to control the use of this new link-poisoning system. And second, just look at the raft of Silicon Valley brands that Google had already lobbied and onboarded, behind the scenes in 2004. This is a classic example of the Silicon Valley collective working as an almighty, anti-competitive gang, out of the public gaze.

WHAT WAS THE EXPRESSED PURPOSE OF NOFOLLOW?

Nofollow was originally billed as an anti-spam measure, which would "block" spammers who were dumping links into comment threads. There were two main holes in this narrative.

  • It was just about the worst way to supposedly "block" spam that anyone could invent. In order to "block" spammers, the system relied on them: a) knowing that the nofollow attribute existed, and b) caring. Virtually no spammer would satisfy both of those conditions. So nofollow would not stop spammers dropping links or in any way solve the public-facing problem of comment spam. Indeed, so non-existent an impact did nofollow have on reducing comment spam, that it was urgently followed by a range of measures or tools, such as Akismet, which took a real and serious approach to controlling the problem. Turned out that the best way to block spam was... Well, ya know... Actually BLOCK it.
  • Even if Google really did want to negate the SEO potential of comment spam, a much simpler, more economical and more logical solution would have been to place a "comments" tag around each comments block. This would additionally have enabled the search engine to clearly define comments blocks, and evaluate everything in them accordingly - not just the links.

In relation to the second point, why didn't Silicon Valley take the simple option of tagging comment threads? Because, dear friend, the vast majority of the links in comment threads pointed back to Silicon Valley's own domains. And the tech collective wanted those links to be free from the poison. Oh yes. You see, a link in a comment was "NoT sPaM" if it pointed to a Silicon Valley silo. They wanted the full search ranking juice on that. The link was only deemed to be spam if it pointed to someone else's property. And that is why Big Tech took the apparently illogical step of poisoning individually selectable links, rather than the whole user-generated block.

Here's a perfect example. A Flickr page from 2006. Including avatars, there are nearly a hundred links in the comments thread, but they all lead back to Silicon Valley, so they don't carry the nofollow tag. Result? Across the breadth of the platform, Silicon Valley gets millions of links' worth of search ranking juice (most of those links just as unnecessary as external spammers' links). But outsiders get nothing - even when their links are relevant and categorically NOT spam.

Unlike with Flickr, WordPress and Blogger, on Wikipedia there was only one possible rationale for "nofollow". To deny search visibility to smaller sites.

This charade was exemplified within Wikipedia - a Google-funded, Google-blessed, Silicon Valley content-farmer whose purpose was solely to suck in all of the Web's value, placing it under the control and stewardship of Silicon Valley. In order to do this, Wikipedia needed to plunder the smaller sites which already offered that value directly to the public. It was then compelled to link to those smaller sites as references.

I previously explored the tricks Wikipedia used to cripple the attribution it gave to those sites, but one of the most sickening, and certainly THE most effective of these tricks, was the addition of the nofollow attribute to outbound links.

The dynamics of Wikipedia were totally different from those of regular UGC (user generated content) sites with comment sections. Wikipedia had no parallel to the comment section, and it was not a UGC platform in the accepted sense. It was really a collaboration platform. Strictly moderated in all areas, with no "unattended" walls of low-quality or pointless blather.

What's more, Wikipedia was linking to sources it acknowledged to be valuable. So valuable, in fact, that the platform was actually republishing the information it had found on those sites. Wikipedia's outbound links were definitively NOT spam. So there was no case at all for the use of "nofollow" in those links, other than to cripple smaller publishers' search ranking and stop them from competing.

Notably, Google did not penalise Wikipedia for this dire misuse of the nofollow tag. That failure to punish a clear and obvious manipulation of the search results revealed precisely what the system was designed to do: prevent smaller websites from competing with Silicon Valley silos for search visibility.

This is archetypal Silicon Valley self-preferencing. Sly, elitist, invisible to the general public, and because of the scale advantages Silicon Valley already had, impossible for outsiders to compete with. Scale-ist, if you like. Prejudiced towards large scale content-farmers.

The whole world now saw "nofollow" as Silicon Valley had seen it from the start. Not as an "aNtI-sPaM mEaSuRe", but as an anti-competitive weapon.

THE DEATH OF NOFOLLOW

It was Wikipedia, above all other platforms, that gave the game away. Wikipedia was the silo that revealed Silicon Valley's anti-competitive skulduggery to owners of smaller sites. The ploy had worked on Flickr and other genuine UGC platforms, where there was an argument that "nofollow" existed to control spam. Even there, it was sleight of hand, because those other platforms were still spamming their own links unpoisioned. But unlike with Flickr, WordPress, Blogger, etc, on Wikipedia there was only one possible rationale for "nofollow". To deny search visibility to smaller sites.

Once the admins of those smaller sites began to see Wikipedia calculatedly using Google's link-poisoning scheme to suck in their value and then bury them, they began to poison back.

It would take years for the domino effect of nofollow to reach its crescendo, but eventually, link-crippling hit critical mass, and the domino line lurched into rapid fall. By the late 2010s, almost every outbound link, on every serious site, was now attributed with "nofollow" unless there was some kind of reciprocal deal in play. The whole world now saw "nofollow" as Silicon Valley had seen it from the start. Not as an "aNtI-sPaM mEaSuRe", but as an anti-competitive weapon.

"Never give anyone a dofollow link without a transaction."

As usual, Silicon Valley's capitalism-on-steroids had managed to result in a more bitter, more mean, more transactional society. But there was also a backlash for Google itself.

With the nofollow attribute having become a widespread default, used regardless of content quality or origin, there was no longer any way for a search algorithm to distinguish spam links from valid links. We saw the effect of this played out in the actual search results. By 2019, with "nofollow" almost part of the fabric of hyperlinks, Google had totally lost its ability to recognise good content. Its search results were embarrassing, and it only had itself to blame.

Nofollow was dead.

That year, Google very quietly readjusted its algorithms to discriminately negate the poison of "nofollow", and introduced a brand new link attribute to replace it. The new attribute - "ugc" - can now be found in use on some modern equivalents of those old Silicon Valley platforms where link-poisoning began. But all kinds of damage has been done, both to Google's reputation, and to the accessibility of the Web's true treasure trove.

The "ugc" attribute has so far been met with much greater scepticism than "nofollow". And why wouldn't it be? We're just going round in circles, and the thrashing monster of Google is now in any case losing its command. People are realising that jumping through Google's hoops now brings such a short-term gain that it's really not worth bothering to jump through them anymore.

I used to receive Google's "Webmaster" alerts and immediately dash off to comply with whatever near-incomprehensible technical demand the Big G was making. I now ignore them. And here's a little secret: it hasn't made any difference. Search engine optimisation is now, to coin a very offscreen catchphrase of the late Jim Bowen, "one big fuck-up".

If there was no existing mechanism through which Silicon Valley could justify unfair and anti-competitive self-preferencing, Google could be relied upon to invent one. It will be interesting to see, should the Google empire start to crumble, whether the rest of the Silicon Valley BigBrotherhood cares to return the favour.