Last week Google announced a significant change to their crawler’s behavior. The “Googlebot” finally has the ability to interpret JavaScript, the last remaining core construct used to create and manipulate content on web pages (HTML and CSS being the other two).
In an official post on their Webmaster Central Blog, Google shared that when it first began crawling sites, the Internet was a very different place. Sophisticated JavaScript wasn’t really commonplace; instead, it was used to “make text blink”.
The Internet and our usage of it have changed over the years, and JavaScript’s special ability to reach into web pages and change content programmatically has gained more recognition. For the last ten years or so, web pages have relied on JavaScript, and while it’s recommended to have them degrade gracefully so users who turn off JavaScript can still access the page, it’s fair to say that this doesn’t always happen.
Given our dependence on JavaScript and other technologies that have been developed with JavaScript as a foundation, It’s a little surprising to me that Google, the dominant search engine worldwide, waited this long to index JavaScript. The reason they hadn’t done this earlier, presumably, is because JavaScript has the potential to confuse search engine spiders: it can manipulate content, lead to other pages, cloak links and content, etc.
Tests conducted a couple of years ago have demonstrated that Google has had some (extremely limited) capacity to process JavaScript. Given this partial support, it stands to reason that the Googlebot still isn’t able to process all JavaScript, just more of it. Indeed, the Webmaster Central Blog post does admit “sometimes the JavaScript may be too complex or arcane for us to execute”.
The Implications
So what are the implications crawlers of being able to execute JavaScript for the state of SEO?
Let’s take a look at some of the changes you’re likely to see on Google and which sites they’ll have the biggest effect on.
1. Better Flow of Link Juice
It may be a little obscure, but some websites use JavaScript links frequently within their design. Entire navigation menus are sometimes fully reliant on JavaScript. The ability to parse these links will result in better “link juice” distribution. This will be beneficial for most pages, but it may dilute the value of some (imagine a page that has an overwhelming excess of outbound JavaScript links).
2. Poor Load Times
The use of excessive JavaScript is rampant and, often times, a browser has to make a significant quantity of additional requests and spend time downloading this JavaScript. Now that the Googlebot has to do this too, many sites’ load times in the eyes of Google are likely to increase. To see if you’re affected log in to your Google Webmaster tools and check your “Crawl Stats” graph over the past few months.
3. Higher Crawler Abandonment Rates
Since the Googlebot won’t be capable of processing extremely complex JavaScript, it might give up on indexing pages that use a lot of JavaScript to produce an unconventional user experience—you know, the sort of sites you see featured on Awwwards every now and again.
4. Innovation in Black Hat SEO
This change on Google’s behalf opens the doors to new ways to trick the Googlebot into indexing content it doesn’t want to. For instance, after analyzing the limitations of the Googlebot’s ability to read JavaScript, it’s likely that sooner or later, someone will develop new ways to cloak content (show real visitors one thing while showing the Googlebot something else).
Of course, the above-mentioned are predictions. We can’t forget that SEO is, at its core, reverse engineering. A more accurate understanding over what Google’s really doing will come from testing and checking your data.
The engineers over at Google mentioned that they’ve been rolling out the ability to read JavaScript over the past “few months”, so if you’ve seen changes in your Webmaster Tools in the February—May 2014 window let us know in the comments!