Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Context/Developer docs] Exclude some URL query parameters #3161

Open
comxd opened this issue Sep 18, 2024 · 0 comments
Open

[Context/Developer docs] Exclude some URL query parameters #3161

comxd opened this issue Sep 18, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@comxd
Copy link

comxd commented Sep 18, 2024

Being able to define a list of query parameters for the crawled URL would make it possible to avoid, for example, crawling documentation in several languages.

In the following logs for a crawl job, I would have liked to exclude the hl parameter:

Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=it
[INFO]: Fetching https://cloud.google.com/tasks/docs
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=ja
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=pt-br
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=es-419
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=de
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=fr
[INFO]: Fetching https://cloud.google.com/tasks/docs?hl=zh-cn
...

Please reply with a 👍 if you want this feature.

@comxd comxd added the enhancement New feature or request label Sep 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant