“Single-page” JS websites and SEO

Question

There are a lot of cool tools for making powerful \"single-page\" JavaScript websites nowadays. In my opinion, this is done right by letting the server act as an API (and no

Answer 1

While #2 might be "easier" for you as a developer, it only provides search engine crawling. And yes, if Google finds out your serving different content, you might be penalized (I'm not an expert on that, but I have heard of it happening).

Both SEO and accessibility (not just for disabled person, but accessibility via mobile devices, touch screen devices, and other non-standard computing / internet enabled platforms) both have a similar underlying philosophy: semantically rich markup that is "accessible" (i.e. can be accessed, viewed, read, processed, or otherwise used) to all these different browsers. A screen reader, a search engine crawler or a user with JavaScript enabled, should all be able to use/index/understand your site's core functionality without issue.

pushState does not add to this burden, in my experience. It only brings what used to be an afterthought and "if we have time" to the forefront of web development.

What your describe in option #1 is usually the best way to go - but, like other accessibility and SEO issues, doing this with pushState in a JavaScript-heavy app requires up-front planning or it will become a significant burden. It should be baked in to the page and application architecture from the start - retrofitting is painful and will cause more duplication than is necessary.

I've been working with pushState and SEO recently for a couple of different application, and I found what I think is a good approach. It basically follows your item #1, but accounts for not duplicating html / templates.

Most of the info can be found in these two blog posts:

http://lostechies.com/derickbailey/2011/09/06/test-driving-backbone-views-with-jquery-templates-the-jasmine-gem-and-jasmine-jquery/

and

http://lostechies.com/derickbailey/2011/06/22/rendering-a-rails-partial-as-a-jquery-template/

The gist of it is that I use ERB or HAML templates (running Ruby on Rails, Sinatra, etc) for my server side render and to create the client side templates that Backbone can use, as well as for my Jasmine JavaScript specs. This cuts out the duplication of markup between the server side and the client side.

From there, you need to take a few additional steps to have your JavaScript work with the HTML that is rendered by the server - true progressive enhancement; taking the semantic markup that got delivered and enhancing it with JavaScript.

For example, i'm building an image gallery application with pushState. If you request /images/1 from the server, it will render the entire image gallery on the server and send all of the HTML, CSS and JavaScript down to your browser. If you have JavaScript disabled, it will work perfectly fine. Every action you take will request a different URL from the server and the server will render all of the markup for your browser. If you have JavaScript enabled, though, the JavaScript will pick up the already rendered HTML along with a few variables generated by the server and take over from there.

Here's an example:

<form id="foo">
  Name: <input id="name"><button id="say">Say My Name!</button>
</form>

After the server renders this, the JavaScript would pick it up (using a Backbone.js view in this example)

FooView = Backbone.View.extend({
  events: {
    "change #name": "setName",
    "click #say": "sayName"
  },

  setName: function(e){
    var name = $(e.currentTarget).val();
    this.model.set({name: name});
  },

  sayName: function(e){
    e.preventDefault();
    var name = this.model.get("name");
    alert("Hello " + name);
  },

  render: function(){
    // do some rendering here, for when this is just running JavaScript
  }
});

$(function(){
  var model = new MyModel();
  var view = new FooView({
    model: model,
    el: $("#foo")
  });
});

This is a very simple example, but I think it gets the point across.

When I instante the view after the page loads, I'm providing the existing content of the form that was rendered by the server, to the view instance as the el for the view. I am not calling render or having the view generate an el for me, when the first view is loaded. I have a render method available for after the view is up and running and the page is all JavaScript. This lets me re-render the view later if I need to.

Clicking the "Say My Name" button with JavaScript enabled will cause an alert box. Without JavaScript, it would post back to the server and the server could render the name to an html element somewhere.

Edit

Consider a more complex example, where you have a list that needs to be attached (from the comments below this)

Say you have a list of users in a <ul> tag. This list was rendered by the server when the browser made a request, and the result looks something like:

<ul id="user-list">
  <li data-id="1">Bob
  <li data-id="2">Mary
  <li data-id="3">Frank
  <li data-id="4">Jane
</ul>

Now you need to loop through this list and attach a Backbone view and model to each of the <li> items. With the use of the data-id attribute, you can find the model that each tag comes from easily. You'll then need a collection view and item view that is smart enough to attach itself to this html.

UserListView = Backbone.View.extend({
  attach: function(){
    this.el = $("#user-list");
    this.$("li").each(function(index){
      var userEl = $(this);
      var id = userEl.attr("data-id");
      var user = this.collection.get(id);
      new UserView({
        model: user,
        el: userEl
      });
    });
  }
});

UserView = Backbone.View.extend({
  initialize: function(){
    this.model.bind("change:name", this.updateName, this);
  },

  updateName: function(model, val){
    this.el.text(val);
  }
});

var userData = {...};
var userList = new UserCollection(userData);
var userListView = new UserListView({collection: userList});
userListView.attach();

In this example, the UserListView will loop through all of the <li> tags and attach a view object with the correct model for each one. it sets up an event handler for the model's name change event and updates the displayed text of the element when a change occurs.

---

This kind of process, to take the html that the server rendered and have my JavaScript take over and run it, is a great way to get things rolling for SEO, Accessibility, and pushState support.

Hope that helps.

Answer 2

Interesting. I have been searching around for viable solutions but it seems to be quite problematic.

I was actually leaning more towards your 2nd approach:

Let the server provide a special website only for the search engine bots. If a normal user visits http://example.com/my_path the server should give him a JavaScript heavy version of the website. But if the Google bot visits, the server should give it some minimal HTML with the content I want Google to index.

Here's my take on solving the problem. Although it is not confirmed to work, it might provide some insight or idea's for other developers.

Assume you're using a JS framework that supports "push state" functionality, and your backend framework is Ruby on Rails. You have a simple blog site and you would like search engines to index all your article index and show pages.

Let's say you have your routes set up like this:

resources :articles
match "*path", "main#index"

Ensure that every server-side controller renders the same template that your client-side framework requires to run (html/css/javascript/etc). If none of the controllers are matched in the request (in this example we only have a RESTful set of actions for the ArticlesController), then just match anything else and just render the template and let the client-side framework handle the routing. The only difference between hitting a controller and hitting the wildcard matcher would be the ability to render content based on the URL that was requested to JavaScript-disabled devices.

From what I understand it is a bad idea to render content that isn't visible to browsers. So when Google indexes it, people go through Google to visit a given page and there isn't any content, then you're probably going to be penalised. What comes to mind is that you render content in a div node that you display: none in CSS.

However, I'm pretty sure it doesn't matter if you simply do this:

<div id="no-js">
  <h1><%= @article.title %></h1>
  <p><%= @article.description %></p>
  <p><%= @article.content %></p>
</div>

And then using JavaScript, which doesn't get run when a JavaScript-disabled device opens the page:

$("#no-js").remove() # jQuery

This way, for Google, and for anyone with JavaScript-disabled devices, they would see the raw/static content. So the content is physically there and is visible to anyone with JavaScript-disabled devices.

But, when a user visits the same page and actually has JavaScript enabled, the #no-js node will be removed so it doesn't clutter up your application. Then your client-side framework will handle the request through it's router and display what a user should see when JavaScript is enabled.

I think this might be a valid and fairly easy technique to use. Although that might depend on the complexity of your website/application.

Though, please correct me if it isn't. Just thought I'd share my thoughts.

Answer 3

Use Google Closure Template to render pages. It compiles to javascript or java, so it is easy to render the page either on the client or server side. On the first encounter with every client, render the html and add javascript as link in header. Crawler will read the html only but the browser will execute your script. All subsequent requests from the browser could be done in against the api to minimize the traffic.