New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

You should be able to provides the robots.txt #192

Open

panthony opened this issue Apr 2, 2018 · 0 comments

Labels

feature

panthony commented Apr 2, 2018

What is the current behavior?

Today the project automatically resolves the robots.txt.

What is the expected behavior?

It would be useful to be able to provides the robots.txt instead to bypass the default behavior of resolving it automatically.

What is the motivation / use case for changing the behavior?

You may want to provides a different set of rules (let's say I'm the owner of the site and I want to check of the crawler would behave with a different robot.txt)
In a big distributed environment, maybe you want to resolve the robots.txt once and share it with all the workers

yujiosaka added the feature label

yujiosaka mentioned this issue

Crawler should honor the Crawl-Delay if obeyRobotsTxt:true #194

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment