Improve search results on Google

[acorncom]

Currently, all Google links point to /v1.10.0 guides instead of more recent guides. In order to fix that, we need to work out how to have a guides page that Google ends up looking at as more "authoritative" (probably by keeping the link the same). Additional context and discussion is available here: #604

Once we've worked through that, we'll also need to inform Google that the v1.10.0 guides are not the most current guides (probably with a <canonical> tag put in the header of the old files).

We need to decide what our <canonical> url for guides should be. We seem to have two options:

1. Have a /current link where the master version of guides always lives.
2. Have current guides live under guides.emberjs.com with no version number specified. This is the approach taken by the Ruby on Rails folks.

In either of these cases, we will also want to publish the guides at a static version number so that blog post authors can link to versioned guide pages without having to worry about those pages changing later.

Feedback about pros / cons to these approaches would be quite welcome. What issues / problems might we run into from an SEO point of view?

[robbyrussell]

Perhaps we can lean on what the Read the Docs folks are doing. It appears that they're doing something to option 1 on previous versions always referencing the latest as the canonical.

For example, looking at Sphinx 1.3.5 docs, I see the following:

<!-- 
Always link to the latest version, as canonical.
http://docs.readthedocs.org/en/latest/canonical.html
-->
<link rel="canonical" href="http://www.sphinx-doc.org/en/stable/" />

Found this after looking through http://docs.readthedocs.org/en/latest/versions.html

[tonycoco]

Agree that option #1 seems the most legit. Plus, in reading http://read-the-docs.readthedocs.org/en/latest/versions.html#redirects-on-root-urls they recommend that a redirect is placed in the root to the "latest" doc branch.

[tomdale]

@robbyrussell One potential problem with the <link rel="canonical"> approach is that it makes it difficult for users of an older version.

I'll give you a concrete example from my own experience. I've been writing Node.js apps that target AWS Lambda, which only supports Node 0.10. The latest release of Node is v5, so if I search for e.g., node child_process spawn on Google, it gives me the Node v5 docs.

Giving me the latest version for this query is great. But I see that it's for v5, so I add my version to the end of my query: node child_process spawn 0.10. However, because all of the 0.10 docs have a <link rel="canonical"> pointing to the current version, they're essentially invisible to Google. For this query, I get irrelevant results. Even worse, doing a search for node 0.10 api docs doesn't return the right results. You have to manually navigate to https://nodejs.org/en/docs/ and click the 0.10 link; as far as I can tell, there's no way to find them in Google.

(I assume this problem is related to <link rel="canonical">. It may be that they put it in a robots.txt or something. But I do see that 0.10 docs point to the latest release as canonical.)

[acorncom]

Although yes, doing a redirect is far easiest (and we could have done it already), we're trying to avoid stranding folks on older versions with a poor search experience (as @tomdale described). Any suggestions on how to do that well? :-)

[danDanV1]

What is the user-experience design goals for SEO + the docs?

It seems to me that the use-case for the Ember website should most likely be disseminating the current version information. It is much more suitable to make someone who is searching for outdated information have to navigate the site to their outdated version.

How about no-index on the some of the old docs? As a new ember developer, I am constantly googling. It's very frustrating to keep getting outdated information. It makes my experience of adopting Ember about 10x worse then past frameworks I've learned. I've wasted many hours Googling for answers then only to switch the website to the latest version of that page and find that what I was searching for is not mentioned in the new docs! Hello adoption churn.

It's not very effective from a user-design experience to have an old page index and found at the top of search results, only to have it redirect to another page which may not contain the information. For example, some old guide pages have very detailed examples of explaining how something works, where as the new 2.3.0 doc page doesn't have it.

Look at this example of a search I performed today:
"ember component trigger action"
-I wanted to find an example of how I can I trigger an action in my component.
-I tried this.action.actionName() but it didn't work, "this" reference in my action was in wrong scope, so it didn't work.
-Should be a quick search, right?

Onward to Google for my answer. The search returns this page as #1 in Google:
https://guides.emberjs.com/v1.10.0/components/sending-actions-from-components-to-your-application/

I didn't know that it is a 1.10 version page, from I saw in Google:

Ember.js - Components: Sending Actions from Components ...
https://guides.emberjs.com/.../components/sending-actions-from-compon...
So how do you trigger sending a component's primary action? After the relevant event occurs, you can call the sendAction() method without arguments: ...

It looks pretty relevant to me. I'm excited to be soon solving my problem.

After a minute of trying to find my answer on the page, I can't find it. It's all about sendAction() to the application. I just want to trigger an action in my component. I realize that is old docs, maybe the new version has it? My app is using 2.3.0. There's no button to switch to that version of this particular page, nor a warning.

So I have to switch the guides to 2.3.0 and it takes me to the index page. Great. Then I have to guess what page is appropriate. I don't want to send the action to the application, I just want to trigger an action. But there is no guarantee that the browsing through the components section I will find the page with the correct keywords I was searching for. It defeats the purpose of using a search engine.

What I was looking for was:
this.send('actionName')
"Triggers a named action on the ActionHandler"
http://emberjs.com/api/classes/Ember.Component.html#method_send

There's not even any mention of this in the any of the guides, just in the api. I think the only way I was able to find the answer was doing an on-page search on the api page for "action" and looking at every reference.

I took me about 30 minutes to figure this out, when it should have only taken 2 minutes max. I got sent on a goose-chase because the outdated docs/mismatch renders google searches useless. This happens on a daily basis.

In every programming site/forum, eg. stack overflow, slack, etc., most experienced programmers always complain "read the docs" or "do a google search". What's a new adopter suppose to do when you can effective use google to search the docs?

[tomdale]

@edeis53 Thank you so much for taking the time to write that up. I'm sorry it was such a frustrating experience for you. Concrete examples like this make it much easier to improve.

I actually think there was a double-fail on our part in your example. First, yes, the old documentation showed up before the new documentation, which sucks. The second thing that happened is we have two things called "actions" that are very different conceptually. A big reason that documentation was removed was because it didn't apply to the new actions; it wasn't removed "just because," it was removed because it is no longer about a system that we consider a best practice. But of course this isn't a satisfying answer, because the old documentation discussed how to do something you wanted to do and the new ones don't!

I think the lessons that we can learn from this are:

1. If you have two systems that are sufficiently different that most of the old documentation doesn't apply, call it something else. (AKA the SproutCore 2.0 problem)
2. As much as possible, make sure the new documentation describes the same set of use cases as the old documentation.

How about no-index on the some of the old docs? As a new ember developer, I am constantly googling. It's very frustrating to keep getting outdated information. It makes my experience of adopting Ember about 10x worse then past frameworks I've learned.

I can sympathize with this, but you're a new user. Imagine how you'd feel if you were hired to work on an application on an older version of the framework. These people want to use Google too—and not being able to would make them just as frustrated as you (rightly) are now.

I think the answer, if possible, is to always prefer newer content, but still make older content indexable so that more specific queries that contain a version number still appear. If that's not possible, though, then yes we should probably have Google always index the newest content and perhaps provide our own version-specific search for the guides.

[acorncom]

Spent some time discussing options with @trek and we've basically come down to two options:

1. Use current or some equivalent and setup canonical urls that then redirect to the current version (a sample setup is discussed here)
2. Leave things as they are

If we go the first route, we will make people quite happy who are using the latest version of Ember. However, for anyone using anything other than the current version we will effectively (and efficiently!) eliminate any chance that they can find what they are looking for on older versions. If I'm searching for Django views 1.7 due to being on an older version, nothing comes up.

So we come to a fork in the road. Do we care more about finding current content only? Or do we want to continue some chance at finding what people are after if they're not on the most current release? We're not seeing many good middle ground solutions ...

/cc @tomdale @wycats

For background context, Ruby on Rails does guides versioning (much like we do) without canonical urls. But they don't do a 6-week release cycle, which seems to allow more "link juice" to accrete for a given version of their guides.

[danDanV1]

We could also take a Redhat or Centos approach, where versions have a lifespans, and focus to improve search for active (stable) versions.

Also, perhaps use data instead of guessing. Ember-cli send reports to Google Analytics every time "ember server" is run, or analytics from the ember guides/api. That would give a sense of what versions developers are actually using and optimize for those versions.

Considering how fast development on Ember moves, it seems like option #1 might be a good idea. Because the current setup isn't working well. It's always going to take a while for link juice to accumulate. We may be releasing new versions before links are even made to the previous version. Especially considering most of the Q&A is via Slack, so there isn't any hyperlink trail to help them out. The fact that slack isn't indexed may be part of the problem. Everything on Stackoverflow is usually linking to old versions, all the new chat is in Slack.

I agree it doesn't seem to be such an issue with slower moving frameworks. My last few project were with Yii framework, and they just have two branches 1.0 and 2.0 branch, with incremental in each one. Very easy to find the correct information, never was mislead once. But again, all the chat was either in a forum or Stackoverflow, so plenty of link juice and relevant text for good search.

I find this discussion quite ironic in a way because it seems to mirror the old days of IE6 ... where web developers hands were tied and couldn't use the latest technologies because they were stuck supporting old platforms. However in this case, were talking about documentation.... now were hindering developers at the knowledge level, and development of Ember is moving way faster than past technologies. I hope you can see my point there. Even harder to implement new stuff if you can't learn about it.

[acorncom]

Per a discussion in the docs meeting today, we're going to explore whether schema.org markup might aid with this.

Background on Google's support for it is available here: https://developers.google.com/structured-data/

Potential schemas to use: https://schema.org/Article or https://schema.org/CreativeWork

cc @wycats

[tomdale]

@acorncom Sorry I missed the meeting. For those of us that weren't there, can you provide some more context about how schemas will help address the issue?

[acorncom]

@tomdale Sure, sorry those notes weren't more detailed.

Re: how schemas will help address the issue. We're not actually sure they will, but Yehuda was saying there had to be a way to help search engines understand versioned documents and he suggested that schemas might be the answer. He was going to ping Paul Irish to ask about that (I would assume after EmberConf). My job was to do the initial leg work on what schemas might fit what we're looking for.

But to clarify, I haven't seen any documentation sites using schemas yet (and I've checked a number at this point) 😄

[acorncom]

Based on further discussions at our learning team meeting yesterday, due to the headaches we're having, we've decided to use <link rel="canonical"> despite the fact it eliminates the ability to do Google searches on old versions of the guides. We'll wait to make these changes until we've replaced the search system on guides (see #1184 for details) so that people on old versions can at least search within old versions of guides and get good results.

[joshhornby]

We are looking to adopt Ember at the company I work for, but find Googling for docs quite frustrating - I echo the thoughts of @edeis53.

So if there is anything I can do to help this along let me know, it will save my self and countless others many hours in the future.

[acorncom]

@joshhornby Would love to get your help on that. This issue is blocked by #1561 If you'd be interested in helping me push that to completion, it'd be great. I'm also happy to discuss things further on the Ember Community Slack (ping me directly or we can talk on the #-team-learning channel. Thanks!

[williamhaley]

@acorncom Although I'm very new to Ember, I'd love to help move this issue along if I can. Is the issue you referenced above still the blocker to getting this done?

In the short term, I've tried using some Google syntax to help out, but even with that it's a hodgepodge of old and new docs in the search results.

model site:"https://*.emberjs.com" "v2.11.0"

Unfortunately, I have to agree with the sentiments above. Navigating the docs has been a bit frustrating, and one of the more difficult part of adoption so far. I'm the only dev at our company using Ember at the moment, but I'm nervous about bringing on new devs in the near future and the ramp-up it takes to get used to the docs and reliably find the latest info.

[locks]

@williamhaley are you on the Slack group? Our guides/website/api docs in-page search has been more useful since we switched providers, have you found that lacking as well?

[williamhaley]

Thanks @locks. I had tried the in-page search tool when I was starting out with Ember about a month ago, but I gave up on it after a couple tries. The reason was that, at the time, I didn't really know the Ember API or lexicon, and the sorts of searches I was doing didn't bring back results that I found especially helpful as a novice.

For instance, if I search "nest route" using the in-page search tool, the first result it brings me is for the Ember.Route documentation. Accurate, but not really helpful based on my search.

Alternatively, the first Google result for "ember nest route" is the guide titled "Adding Nested Routes". That is exactly what I wanted.

Even though Google's result is for v2.7.0, I still find that search result much more useful. My daily workflow typically involves Google searches like this, and then changing the docs to show me the latest version.

Now that I'm slightly more familiar with Ember, I do see that the in-page search is pretty solid if you already know the name of the API/Class/whatever you may want to search for. When I search for something like modelFor or link-to, the search returns exactly what I'd expect. In that case, I feel like it's just as reliable as using Google.

Though, I feel like Google is still better for natural language searches. Google always seems to bring back exactly what I want at the top of the results for both natural language searches and API specific searches. The only catch being that the versions are all over the place in the results.

To me, a big motivator behind this issue is that for someone coming to Ember as an outsider, Google is usually the first step. I know this is a superficial argument, but seeing a mix of Ember doc versions in the search results looks weird. It makes it seem like something is wrong/broken. I feel like most users who are starting out expect the latest version of the docs show up at the top of the results.

My experience and opinions related to the docs is pretty much identical to what was reported above by @edeis53

Sorry, I don't mean to knock the in-page search or the guides/docs. Just trying to share my experience as someone starting out and having a rough time using my normal methodologies. My usual method for searching APIs and documentation is Google, and I do not find that as effective for Ember as I'm used to in most other languages/frameworks.

I think the in-page search will be helpful in the mean time, but I do really think this needs to be rectified to make the docs more approachable to newcomers, and I'm happy to help in that process at all if I can.

I just joined the Slack group too, which seems like a helpful resource. Thanks.

[acorncom]

@williamhaley

Sorry, I don't mean to knock the in-page search or the guides/docs. Just trying to share my experience as someone starting out and having a rough time using my normal methodologies. My usual method for searching APIs and documentation is Google, and I do not find that as effective for Ember as I'm used to in most other languages/frameworks.

It's not taken as a knock on the in-page search, appreciate your thoughtful response and clear articulation of the issues you're hitting. I'd also agree with you, it can definitely improve.

I think the in-page search will be helpful in the mean time, but I do really think this needs to be rectified to make the docs more approachable to newcomers, and I'm happy to help in that process at all if I can.

👍 That would be hugely appreciated. We're a small volunteer team with not a lot of time to devote to this, so the Google issue has been one hanging over our heads for the last year as I've slowly labored away at it. If you've got some time/energy to help me push on it, we're getting close to the point that we can start flipping the switch. I'll look you up on Slack to see if we can coordinate things a bit more.

[acorncom]

Thanks for all the feedback here folks, we're starting in on actually making these changes. Tracking the work here: #1816 If you have time, would love to get some help here!

[acorncom]

Closing this issue ...