You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Only one group is valid for a particular crawler. The crawler must determine the correct group of lines by finding the group with the most specific user-agent that still matches. All other groups are ignored by the crawler. The user-agent is case-sensitive. All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot). The order of the groups within the robots.txt file is irrelevant.
The text was updated successfully, but these errors were encountered:
All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot).
What this means is that user agents must essentially match the regular expression [a-zA-Z_-]+.
i.e googlebot/1.2 and googlebot* are nonconforming user agents and Google is choosing to handle them by discarding everything after the first invalid character.
I don't know off the top of my head whether the library is doing this.
https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents
The text was updated successfully, but these errors were encountered: