Expand robots.txt for Kedro-Viz and Kedro-Datasets docs

[DimedS]

This PR updates the robots.txt file to include explicit allowances for the indexing of the Kedro-Viz and Kedro-Datasets documentation sections by search engines.

Changes Made:

• Added Allow: /projects/kedro-viz/en/stable/ to permit the indexing of the stable version of Kedro-Viz documentation.
• Added Allow: /projects/kedro-datasets/en/kedro-datasets-2.1.0/ to permit the indexing of the Kedro-Datasets documentation for version 2.1.0.

Previously, only the kedro project documentation was allowed for indexing, as specified in our robots.txt. By updating this file, we making it easier for users to find information about Kedro-Viz and Kedro-Datasets directly via search engines.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

Is this ready to be reviewed? How should this be tested.

https://www.google.com/search?q=kedro+docs+configloader&oq=kedro+docs+configloader&gs_lcrp=EgZjaHJvbWUyCQgAEEUYORifBTIGCAEQRRg8MgkIAhAhGAoYoAEyCQgDECEYChigATIJCAQQIRgKGKAB0gEIMzk0NGowajmoAgCwAgA&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8

I see 0.18.1 docs on Google search so I suspect something isn't working properly.

[DimedS]

Is this ready to be reviewed? How should this be tested.

https://www.google.com/search?q=kedro+docs+configloader&oq=kedro+docs+configloader&gs_lcrp=EgZjaHJvbWUyCQgAEEUYORifBTIGCAEQRRg8MgkIAhAhGAoYoAEyCQgDECEYChigATIJCAQQIRgKGKAB0gEIMzk0NGowajmoAgCwAgA&client=ms-android-google&sourceid=chrome-mobile&ie=UTF-8

I see 0.18.1 docs on Google search so I suspect something isn't working properly.

Yes, it's ready for review. Overall, it seems to function properly, though we are experiencing some issues specifically with Google—I'm not sure of the exact cause. However, it's mostly fine with other search engines; please see the screenshots from Yahoo, Yandex, and DuckDuckGo for reference. I believe it's impossible to test it before merging, as search engines will need some time to recognize the updated robots.txt and reflect those changes in their search results.
Maybe we need to do something special with google.

[astrojuanlu]

we are experiencing some issues specifically with Google—I'm not sure of the exact cause

Interesting, might be related to #3708 somehow?

[astrojuanlu]

I suggested 1 modification but overall I am okay with this quick and dirty approach 👍🏽

Longer term we should re-evaluate our policy in robots.txt. Originally it was done so that very old versions are indexed by search engines, but not shown on the Read the Docs flyout menu. But nowadays I'm of the opinion that maybe only the most recent stable version should be indexed. We don't have to make that change right now though.

[tynandebold]

This looks good to me.

[noklam]

User-agent: *
Disallow: /
Allow: /en/stable/
Allow: /en/0.19.3/
Allow: /en/0.19.2/
Allow: /en/0.19.1/
Allow: /en/0.19.0/
Allow: /en/0.18.14/
Allow: /en/0.17.7/

For the reference, the robots.txt is still not updated. https://docs.kedro.org/robots.txt

[astrojuanlu]

@noklam it will only get updated in the next release, I believe