Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement IP Rotation. #17

Open
the-siegfried opened this issue Mar 30, 2022 · 0 comments
Open

Implement IP Rotation. #17

the-siegfried opened this issue Mar 30, 2022 · 0 comments

Comments

@the-siegfried
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Currently the application which serves as a Crawler/Extractor only supports connectivity to either the Clearnet or Tor network using the localhost address over a sock5 proxy, which obtains a fixed address to communicate over. In order to improve anonymity and the service the project provides the capability to rotate IP addresses ought to be supported.

Describe the solution you'd like

  • Refactor the implementation of the connect_tor() method in order to support privoxy and proxy rotation.
  • Implementation for proxy rotation support of clearnet crawling.

Additional context
Modern web applications also tend to be supported be Web Application Firewalls (WAFs) and other technologies which can detect crawlers and bots and defer or block assess to the site. By rotating IPs we are consciously evading these detection and mitigating controls as to not disrupt the applications core service.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants