Add linkcheck test for the docs

[s-weigand]

Since it is hard for big documentations to keep all links up with changes in a project, it's a good idea to check the links are working, to prevent broken links as in #6525 and #6541.

Luckily sphinx has this feature built in, so I added it to the test suite.

To prevent the test suite from false positive breaking, i.e. if a website is temporarily down, I made the test allow_failures in the travis build matrix.

I also added some url patterns to be ignored by the test, since they are only an example case, are local development only or test false positive since the ref is generated by javascript.

[stsewd]

I'm -1 on putting this on travis, but +1 on having this in our tox env. The reason is that linkcheck is really slow. And we should be using the ref role for our internal links (the docs fail when there is a broken ref). Also, the problem from #6541 is for another reason.

[s-weigand]

True that linkcheck takes very long, but the codecov test takes even longer, so it wouldn't add an additional delay for the whole test suite to finish.

[stsewd]

True that linkcheck takes very long, but the codecov test takes even longer, so it wouldn't add an additional delay for the whole test suite to finish.

Fair enough. Tests take ~20 min, and linkcheck takes ~11 min

[stsewd]

Thanks!

[mgeier]

You can probably speed it up if you add a few more ignores:

linkcheck_ignore = [
    ...,
    r'https://github\.com/readthedocs/readthedocs\.org/issues',
    r'https://github\.com/readthedocs/readthedocs\.org/pull',
    r'https://docs\.readthedocs\.io/\?rtd_search',
    r'https://readthedocs\.org/search',
]

[s-weigand]

@mgeier Confirmed the speedup locally, 3min instead of 10min.

[davidfischer]

This seems fine. While I feel our tests already take forever, this seems like a worthwhile check.

There are definitely some broken links in our docs (mostly example code) that could be fixed.