You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bug Steps:
Attempt to extract the delay for a user agent from a robots.txt file where the crawl-delay for a specific user agent appears after the rule for *. Example: https://www.archives.gov/robots.txt
Bug Steps:
Attempt to extract the
delay
for a user agent from arobots.txt
file where thecrawl-delay
for a specific user agent appears after the rule for*
. Example:https://www.archives.gov/robots.txt
Expected results:
Actual results:
See https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents for precedence rules.
The text was updated successfully, but these errors were encountered: