Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does user agent matching match Google's spec? #5

Open
benjaminestes opened this issue Nov 18, 2019 · 1 comment
Open

Does user agent matching match Google's spec? #5

benjaminestes opened this issue Nov 18, 2019 · 1 comment
Assignees
Labels
question Further information is requested

Comments

@benjaminestes
Copy link
Owner

https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents

Only one group is valid for a particular crawler. The crawler must determine the correct group of lines by finding the group with the most specific user-agent that still matches. All other groups are ignored by the crawler. The user-agent is case-sensitive. All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot). The order of the groups within the robots.txt file is irrelevant.

@benjaminestes benjaminestes added the question Further information is requested label Nov 18, 2019
@benjaminestes benjaminestes self-assigned this Nov 18, 2019
@benjaminestes
Copy link
Owner Author

This bit here:

All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot).

What this means is that user agents must essentially match the regular expression [a-zA-Z_-]+.
i.e googlebot/1.2 and googlebot* are nonconforming user agents and Google is choosing to handle them by discarding everything after the first invalid character.

I don't know off the top of my head whether the library is doing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant