Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

You should be able to provides the robots.txt #192

Open
panthony opened this issue Apr 2, 2018 · 0 comments
Open

You should be able to provides the robots.txt #192

panthony opened this issue Apr 2, 2018 · 0 comments
Labels

Comments

@panthony
Copy link

panthony commented Apr 2, 2018

What is the current behavior?

Today the project automatically resolves the robots.txt.

What is the expected behavior?

It would be useful to be able to provides the robots.txt instead to bypass the default behavior of resolving it automatically.

What is the motivation / use case for changing the behavior?

  • You may want to provides a different set of rules (let's say I'm the owner of the site and I want to check of the crawler would behave with a different robot.txt)

  • In a big distributed environment, maybe you want to resolve the robots.txt once and share it with all the workers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants