Remove robots.txt disallow rules on these public services

[unixfox]

Completely remove any disallow rules on these websites in order to have the public services indexed and thus easily accessible from normal people:

• https://invidious.io
• https://redirect.invidious.io - https://invidio.us
• https://api.invidious.io - https://instances.invidio.us
• https://docs.invidious.io
• https://uptime.invidio.us

When I mean completely removing them: https://stackoverflow.com/questions/36405998/is-this-robots-txt-syntax-with-an-empty-disallow-correct#comment77953337_36406046

[TheFrenchGhosty]

https://redirect.invidious.io - https://invidio.us
https://api.invidious.io - https://instances.invidio.us

Both of those shouldn't be indexed.

Otherwise #2

[unixfox]

redirect.invidious.io - invidio.us
api.invidious.io - instances.invidio.us

Both of those shouldn't be indexed.

Otherwise #2

They should, so that when normal people type "instances invidious" they find the correct link.

[TheFrenchGhosty]

@unixfox the instance list is on the documentation / on instances.invidious.io

[unixfox]

Let's not make normal people confused, normal people want a quick answer, not dig into multiple websites for finding one link.

We want more people to discover invidious, not fewer people. Every step that make normal people life easier is a good step towards adoption.

[TheFrenchGhosty]

This is why "normal people" are sent to instances.invidious.io

The others aren't made for normal people.

[unixfox]

Ok then we remove the disallow rule on instances.invidious.io?

[Perflyst]

I don't think we should index all those pages. What about indexing the following
invidious.io - main page about the project
instances.invidious.io - instance list for end users
docs.invidious.io - docs for end users and admins

I don't think we should index the following:
api.invidious.io - not meant for end users
todo.invidious.io - just a redir to github
git.invidious.io - just a redir to github

I am not sure about the following:
redirect.invidious.io - redirect service for end users but rather used for sharing links than clicking on google (?)
uptime.invidious.io - relevant for end users but linked on instance page, also external uptimerobot

[unixfox]

Let's index first all the sites we all agree on.

[TheFrenchGhosty]

Agree with with perflyst on having the 3 indexed, the rest is either "pointless" to index (and are linked to on the website), or just not meant to be used by end user.

[unixfox]

Reopening as I don't agree with the other websites.