SEO: Why JavaScript Suck (Most of the Time)

Inspired by Jacob Nielsen renowned 1996 blog post “Why Frames Suck (Most of the Time), I can’t help it – JavaScript suck too. At least when it comes to SEO.

Despite what many developers believe, using JavaScript Frameworks and client side JavaScript rendering, is still not a reliable method for publishing content and links intended for the search engines to crawl and index.

But I understand the confusion. Google is backing one of the most popular JavaScript frameworks Angular and have publicly said that Google can now crawl, render and index content and links published with client side JavaScript.

The problem is, that it is simply not true. At least not all the time. Google may sometimes cache and index JavaScript and they may sometimes not. And when they don’t you pay the price. Google give no guarantee client side JavaScript will always work and there is no way to appeal your case.

In this post I will take a close look at why JavaScript Suck for SEO and a large recent JavaScript SEO study.

Good reasons not to index JavaScript

There are many good reasons for the search engines not to execute client side JavaScript and index the output.

Historically one of the reasons is that search engines can’t be sure how and to what degree search users will be able to replicate the same result.

So in worst case search engines would cloak users – read and index content on a site, that the search users following a link from the search engine to a site cannot see. Naturally they don’t want to give this kind of bad experience to search users.

Today, off course, most users have JavaScript enabled browsers and most of them will execute most JavaScript fine. Most! Most, but still not all. So it is not fully reliable. Google need to keep all users happy.

Another good reason is that it takes far more computing power to execute a JavaScript and filter and index the output compared to just reading the content of a simple HTML file and index it. Far more!

If you multiply this with the thousands of billions of pages Google crawl you get the picture. Even Google have limited resources and spending it all on parsing JavaScript may just not be the the most wise use of them.

In addition to this JavaScripts can be very complex and extremely resource intensive to execute. Unlike HTML, where search engines also have a limit of how much they will download and index, with JavaScript you get nothing if you abandon the execution half way through.

So even if search engines decided to always execute JavaScript in all forms on all pages, they would – due to the fact that they will always need to have some sort of resource threshold, often end up with zero output to index. So its not reliable.

What Google say about Javascript

As I have pointed out before you always need to read between the lines when you listen to Google. I would not go as far as to call Google liars but it is a fact that …

• Google is often too optimistic about what they can and will do. When they claim they can now execute JavaScript it is true that they sometimes do that. But it is also true that they often do not. Its not reliable.
• Very few of the thousands of people working on the many Google products know anything about search – and even less about SEO.
• Google is not in the SEO business – they are in the search engine business. Its not the same! Their job is to build a great search engine and search user experience. Our job as SEOs is to market our sites. Many of Googles recommendations are therefore naturally more focused on what we as webmasters can do to help them build a better search engine. Fair enough. But just don’t believe that is the same as the truth about how you get the best possible results in SEO.

So the fact that Google is behind one of the biggest JavaScript frameworks Angular is in no way a guarantee that it will work well for SEO.  The best Google engineers knows and acknowledge this.

Jeff Whelpley is one of Googles leading engineers working on the Angular team. He said on the Angular U conference, in June 2015:

If you search for any competitive keyword terms, it’s always going to be server rendered sites. And the reason is because although Google does index client-side rendered HTML, it’s not perfect yet and other search engines don’t do it as well. So if you care about SEO, you still need to have server-rendered content.

The big JavaScript SEO test

Bartosz Góralewicz, the CEO at Elephate, have made one of the best and most extensive studies of how using JavaScript frameworks and client side JavaScript influence important basic SEO factors.

The goal of the experiment was to find out if, and to what degree, Google will execute JavaScripts, cache the text content and links and index it.

A good number of the most popular JavaScript frameworks was tested, such as Angular (1 and 2), React (by Facebook), JQuery, Ember and plain JS – most of them both inline and in external files.

The following was checked across the included JavaScript frameworks:

• Fetch and render via Google Search Console – does it render properly?
• Is URL indexed by Google?
• Is URL’s content visible in Google’s cache?
• Are links displayed properly in Google’s cache?
• Search for unique content from framework’s page.
• Check if ”*framework*/test/” URL was crawled.

Among the key findings was:

• Google appear to more often execute inline JavaScript than external JavaScripts
• The fact that Google Cache content doesn’t mean it is also indexed!
• Google Search Console’s Fetch and Render doesn’t work the same as Googlebot

You can read the entire experiment here but I recommend that you start by reading my comments below.

My comments on the JavaScript SEO test

I think they did a great job with this experiment. However, there are a few minor flaws and recommendations that I would like to comment:

1. They did not mention in the notes to the experiment what browser they used to access the “secret” URL’s. If they used Google, or any other Google tools, there is a risk that Google found out about the “/test/” URL, used to check if they read and follow links in JavaScripts or not.
2. Also, in addition to the above you cannot base a test on “security by obscurity” – that because they told noone about test URL’s noone found them,accessed them or in worst case linked to them or submitted them to Google. Obscurity is not security.
3. In the summary of the experiment they write “If you want to know which frameworks work well with SEO …” which I think do not make sense. As I will describe in more details below, it is simple a much to unreliable strategy to expect Google – and all other search engines, to perfectly execute client side JavaScript. So there is no JavaScript framework that work well in SEO – only some that work less bad than others, which is just not good enough to base your SEO strategies on.

Regarding my first two comments it is actually not so important. The least interesting part of this experiment is if Google read and follow the link. Not that it is not important that they do, but if they read the content, they will will in all likelihood also see the link. I am not so nervous about this.

How about Bing, Yandex etc?

The experiment was entirely focused on Google. Being the most dominant search engine in most regions that is understandable but you should not forget about Bing, Yandex if you are in Russia or other local search engines that may be important in your target region.

You cannot, based on this experiment, know what other search engines may do with JavaScript. And in some regions other search engines can be very important too.

Repeating the test reveal some very surprising results!

When I first looked at the test I must say I was surprised by some of the results. Personally I have found Google to be much less competent of executing JavaScript, caching it and making it indexable.

I do trust that Bartosz Góralewicz did all he could to run the best possible experiment but I suspected that the differences between his findings and my genereal observations could be due to inconsistencies in Googles dealings with JavaScript.

So I decided to check some of Bartosz Góralewicz results …

What I found is different results than what he reported. Lets look at the details of the Angular test – being one of the biggest of the tested JavaScript frameworks. And the only one backed by Google (so also the one that most developers expect would work perfectly in Google).

Below you see the original chart with the results from the study.

 

I am mostly interested to find out if the URL get indexed, cached and the content is indexable. So I tested this again. Here is my results.

As you can see my results do not entirely match the results Bartosz Góralewicz got. Actually, it is worse than what he got. I am not saying there is anything wrong with his data. I am sure this is what he got but I can also assure you the above is what I got – when I tested.

My conclusion is – based on both this and similar data I have seen in the past, that indexing of client side JavaScript is just not very consistent. Sometimes Google will execute, cache and index some JavaScript content – and sometimes not. Even the same scripts may work some days and not on others.

That is just not good enough for you to base your SEO strategies on!

One of the most surprising results I found after collecting data for the test again is the fact that Google apparently some times do not cache the JavaScript rendered text but do index it! In other words – the text is not in Googles cache but you can search for it.

I have seen the opposite before – that Google have cached a text that is not indexed and searchable but not this other way around.

But as with almost all SEO there is a good explanation for this . I will explain, but lets first look at two different examples of this …

First an example is using React JS library from Facebook. As you can see the text only version of Googles cache of the page do not show any of the JavaScript rendered text.

But if you search for a unique text string from the JavaScript rendered text on the page it is found in Google search results.

If you look at the code behind the page it is easy to see why the above is happening. The text is with React loaded directly in the code and stored in a variable (Hello) and then “inserted” into the DIV ID “app-root”.

My guess is that because Google did not execute the JavaScript, the text is not visible to them in the DIV and therefore not displayed in the cache. However, because the full text is easily accessible to Google in the code they do read it and index it and make it searchable.

With Angular 1 I noticed the same thing – content not being in the Google cache but indexed and searchable.

 

However, with Angular 1 the text is not embedded directly in the code as it is with React 1 – Google can’t just read it by looking in the code. Instead, the text is found in the external JS file /script.js that is linked to from the embedded code.

So, apparently, Google is – in my tests, reading the texts they can see directly in embedded code or external files, but is not executing the script and inserting the texts in the right placeholders. So it do not appear in the cache but is searchable.

If this type of “code indexing” is good enough to rank for anything competitive I did not test but I suspect not.

Externalize JavaScript or not?

The Javascript SEO experiment show that there is apparently more JavaScript being executed when inlined instead of included in an external file.

Bartosz Góralewicz suggest that this may be due to the extra resources it could take to download and execute the external files. I am not sure that is true. I think it is more likely just another proof of the inconsistency there is with how Google execute JavaScript.

For SEO reasons we usually recommend that all JavaScript is put in an external JS file. With this experiment in mind Bartosz Góralewicz suggest that you may reconsider this and instead inline your scripts.

However, in my opinion you should not. Because you should not, in any case, render your important content and links with JavaScript. You should focus your SEO content strategy entirely on server rendering.

JavaScript indexing is still not reliable

After carefully reading Bartosz Góralewicz JavaScript SEO experiment and and doing my own testings of his data my conclusion remain the same as it has always been: Client side JavaScript rendering of content and links is just not a reliable SEO strategy.

It may work some days. Others days it may not. If your JavaScripts are too complex it may not work at all. Its unreliable and as such a really bad SEO strategy.

So I still recommend that you only use client side JavaScript rendering of content that is not important for you SEO – so none of the unique page content and none of the important navigational links.

On the other hand, you cannot use JavaScript to hide content or links, as some SEOs do. That is not reliable either. Google may execute your scripts. You never know.

JavaScript is cool – just not for SEO

Just to be sure you understand – I don’t hate JavaScript. In fact, I love it. There are so many great things you can use it for. SEO is just not one of them.

Its actually similar to Flash or frames. There are applications where it may be useful – SEO is just not one of them.

So, whatever way you use client side rendered JavaScript just be sure it do not include the important unique text elements and navigational links on your public pages intended for search engine indexing.