Fix googleability of docs

[brson]

Googling for Rust documentation gives very inconsistent results, often to very old docs. Newbies often are led astray.

Old description:

This is a topic I haven't paid much attention to, but it's important, and I know our googlability has traditionally been very bad. I don't know at all what needs to be done off hand but it's something we should dedicate some effort to.

Nominating.

Tasks:

• description and keyword meta for home page
• description and keyword meta for standalone docs (needs rustdoc needs a way to specify HTML metadata for standalone docs #16178)
• description and keyword meta for crate docs
• descriptive titles for standalone docs
• teach rustdoc to meta (rustdoc needs a way to specify HTML metadata for standalone docs #16178)
• add <link rel="canonical"> to docs (needs rustdoc needs a way to specify HTML metadata for standalone docs #16178)

[alexcrichton]

cc #9461

[pnkfelix]

To clarify, the particular problem that has plagued us is that it is too easy to end up with outdated documentation when googling for help with problems.

[pnkfelix]

Assigning 1.0, P-high.

[emberian]

I imagine this could be partially alleviated with some well-placed meta tags.

[Valloric]

The very first step should be reading Google's Webmaster Guidelines and related documents.

Also, here's a good doc Google has prepared on sensible SEO things webmasters can do.

I imagine this could be partially alleviated with some well-placed meta tags.

Here's a really nice video on the topic of meta tags and their usefulness by Google's Matt Cutts: https://www.youtube.com/watch?v=RBTBEfd7z_Y

[adrientetar]

The DNS issue is #4394.

[brson]

@Valloric thank you for those helpful links.

[brson]

I've started a task list in the OP.

I did some basic searches for Rust datatypes in a private window today and the results looked good to me. I'm not sure if there's a whole lot we need to do.

Google is preferring the 'shorthand' doc urls, not the ones with version numbers in them.

I intend to run www.rust-lang.org through Google's various webmaster tools to see if it has any suggestions.

[Valloric]

Google is preferring the 'shorthand' doc urls, not the ones with version numbers in them.

Well, you are using robots.txt to prevent Google from indexing the old API docs.

[brson]

@Valloric we are not preventing indexing of the master docs, which are visible in two locations: doc.rust-lang.org and doc.rust-lang.org/master.

[brson]

I've added a meta description to the home page, which is only slightly better than the one goog already pulls out of the page.

[Valloric]

If you're showing the same content from two different URL structures, I recommend reading this Google support doc on canonical URLs.

[brson]

Nominating for removal from 1.0. Not critical. Worst problems are solved.

[Valloric]

@brson Have you made sure you aren't serving the same page content from two different URL structures without canonical annotations? http://doc.rust-lang.org/master/std/index.html and http://doc.rust-lang.org/std/index.html show the same page for me. As I mentioned previously, I recommend looking at this Google support doc on canonical URLs.

In your case, making sure one of those pages uses <link rel="canonical" href="[canonical page url]" /> would be a good idea.

Even better, make one of those URLs 301 to the other or better still, have just one URL structure for getting to the content.

[brson]

@Valloric No, I have not.

[Valloric]

@brson Consider fixing that before you get tons of people searching for Rust docs. It's always a good idea to have your canonical urls sorted out, for any site.

[pnkfelix]

Removing from 1.0 milestone but leaving as P-high. No one wants to block the release on just the issue pointed out by @Valloric 's comment here: #12466 (comment)

[brson]

Closing in favor of specific linked bugs.