Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disallow indexing few pages for crawling robots #6283

Closed
yangwao opened this issue Jun 19, 2023 · 5 comments · Fixed by #6286
Closed

Disallow indexing few pages for crawling robots #6283

yangwao opened this issue Jun 19, 2023 · 5 comments · Fixed by #6286
Assignees
@floyd-li
Labels
$ ~<50usd A-seo-search-socials p1 preventing everyone from using app

Comments

@yangwao
Copy link
Member

yangwao commented Jun 19, 2023

Currently, we are fighting to get good content on Google SEO and be in bit control; we got an advisory to drop a lot of "bad" pages which are not interesting for bots and raise our quality for them and get speedy-turn-around-reindex times.

image

Pages we should disallow for robots

  • items - like URLs with "/details/" & "/gallery/"
  • profiles - "/u/"
  • all prefixes with "beta.", "canary", "nft.", "form.", "*.w." (like polysearch, image, image-beta, )
  • transfer - "/transfer/"

beta is currently 60% URLs
nft. has fair portion

we will follow up with pre-render integration as I sense it's culprit CF doesn't do it.

Ref

https://support.google.com/webmasters/answer/7440203#crawled
@yangwao yangwao added $ ~<50usd p1 preventing everyone from using app A-seo-search-socials labels Jun 19, 2023
@yangwao yangwao mentioned this issue Jun 19, 2023
Closed
@floyd-li
Copy link
Member

imo an update on robots.txt file can fix this. i'll take it.

@floyd-li
Copy link
Member

👋

@kodabot
Copy link
Collaborator

kodabot commented Jun 20, 2023

ASSIGNED - @floyd-li 🔒 LOCKED -> Wednesday, June 21st 2023, 14:07:10 UTC -> 36 hours

@floyd-li floyd-li mentioned this issue Jun 20, 2023
Merged
16 tasks
@yangwao
Copy link
Member Author

yangwao commented Jun 22, 2023

maybe?

image

@floyd-li
Copy link
Member

maybe?

image

maybe it need some time to take effort?
just made some tests on https://en.ryte.com/free-tools/robots-txt/ seems the rules in robots.txt are correct
图片

@yangwao yangwao mentioned this issue Jun 26, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@floyd-li floyd-li

Labels
$ ~<50usd A-seo-search-socials p1 preventing everyone from using app
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: update robots.txt floyd-li/nft-gallery
3 participants
@yangwao @floyd-li @kodabot