-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(docs): allow to configure noIndex per doc version #7963
Conversation
✅ [V2]
To edit notification comments on pull requests, go to your Netlify site settings. |
⚡️ Lighthouse report for the deploy preview of this PR
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question: how does this play with the sitemap? Over there, we only test tag.props.name === 'robots' && tag.props.content === 'noindex'
. Maybe it should be tag.props.content.includes("noindex")
instead?
✅ [V2]
To edit notification comments on pull requests, go to your Netlify site settings. |
Size Change: +45 B (0%) Total Size: 813 kB
ℹ️ View Unchanged
|
Yes definitively! I thought it was already the case actually 😅 will send another PR and edited our sitemap plugin so that we can see the problem. Will merge a sitemap plugin fix before this PR, ensuring we can see the problem fixed here. |
BTW I guess having a noIndex: true flag for plugin instances in a generic way could be useful 🤷♂️ we also want to have a label, wonder what else makes sense as a general option for a plugin instance. Not going to implement this here, but let's keep this in mind |
working fine now those tag urls are still there in the sitemap and indexed, will do another refactor to fix that <url>
<loc>https://docusaurus.io/tests/docs/tags/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/a/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/b/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/c/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/d-custom-permalink/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/e/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url>
<url>
<loc>https://docusaurus.io/tests/docs/tags/some-tag/</loc>
<changefreq>weekly</changefreq>
<priority>0.5</priority>
</url> |
Motivation
You might want to prevent crawlers and google to index older versions of your docs
See also #7960
Test Plan
unit tests + local tests + dogfood on test docs instance (as it's not really useful to index it)
Test links
Deploy preview:
Limitation: the versioned docs tag pages won't be noIndexed. But a further refactor will fix it soon. (fixed in #7966)