Add previous API versions to website

[ghost]

I previously discussed with @travisbrown if previous API versions should be published to the website. The downside was that search engines tag on old versions.

I'm resurrecting this topic based on a chat with @ochrons, as the solution appears to be "just a question of couple of soft-links and a robots.txt"

Given this, I'm +1 for having the previous versions. Any other opinions/votes?

[ochrons]

So in practice you would have a directory structure like

docs/latest
docs/0.1.0
docs/0.1.1

where docs/latest is a symbolic link to the latest doc directory

and then have a robots.txt at root preventing crawlers from indexing anything but docs/latest

[travisbrown]

One drawback of that approach for GitHub-hosted project sites specifically is that it will require @non to create and maintain a separate gh-pages project in order to place the robots.txt file at the root of non.github.io.

We take a slightly different approach (in a one-off way) with one particular set of old docs we didn't want to break for Twitter Util, where we use canonical links to the new docs in the old docs. At least in theory this means we continue to get SEO credit for old links without having them indexed. It would be more painful to do this in a non-one-off way, though.

The fact that the standard library, Play, etc. still haven't gotten this right after all these years suggests that there is a maintenance cost, even if it's small, but I'm definitely not opposed if @non and others want to pay it.

[ghost]

This is the standard libraries robots.txt

User-agent: *
Allow: /

I reckon - between all of us - we could better that 😉

[travisbrown]

@inthenow But that's the annoying thing about situations like this: half the time there's some incredibly elaborate story about why some painful issue hasn't been fixed, and the other half it's just that nobody's ever bothered (let's hope this is a case of the latter).

[ghost]

@travisbrown LOL 👍

[armanbilge]

This will be fixed as of #4160. We are now publishing unidoc artifacts to maven which for every version can be browsed at javadoc.io.