Use a random user agent string with each http request.

[ehsandeep]

Sometimes you might receive http 429 error or other http errors because of the user agent string. The websites (not only archiveis) are blacklisting certain user agent strings and/or identical get/post requests that are being repeated in a short period of time. In that case, to solve this issue you have to change the user agent string in the agent.go file or preferably use a random user agent string with each http request.

Originally posted by @maldevel in #445 (comment)

[ehsandeep]

We already have this support in nuclei and httpx project.

[taythebot]

I'm working on implementing this feature and was wondering if we should keep the default user agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 or change it to subfinder - Open-source Project (github.com/projectdiscovery/subfinder) to match httpx?

[ehsandeep]

@taythebot httpx has now randon-agent enabled as default, we can follow the same here as the primary reason for this support to avoid blocking from WAP with known agents.

[taythebot]

@taythebot httpx has now randon-agent enabled as default, we can follow the same here as the primary reason for this support to avoid blocking from WAP with known agents.

If random-agent is enabled by default, should there still be a flag for it like httpx does? To my understanding having a boolean flag whose default value is true and passing the flag itself doesn't do anything.

I see that httpx has an unsafe flag which when passed uses the default user agent. Should we follow the same behavior?

[taythebot]

I've gone ahead and added the unsafe flag which uses the default user agent. If the random-agent or another flag is required please let me know!

Edit: unsafe flag is removed