Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
Themis3000 authored Dec 24, 2021
1 parent ea5006c commit 85cf5e1
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,11 @@ There are 3 anonymity levels, indicated as integers between 0-2.
- https://geonode.com/free-proxy-list

## FAQ

- Help! It isn't working!

Proxies are checked to see if they are truely transparent or not automatically by making a request to a remote http server, which returns information on what the recived request looked like. This package uses http://themiserver.duckdns.org:5001/ in order to achive that. The http server is ran from my home, so it may have outages here and there if I ever need to restart my server. If you need to know that this package will behave in a stable way, use test=False when invoking the find_filter method. e.g. `find_filter(limit=20, min_anon_level=1, test=False)`

- Why implement so many websites for scraping?

Websites are always changing, or going down, or banning ip's very quickly. In order to make sure this package stays reliable it is essential that it implements many websites
Expand All @@ -53,4 +58,4 @@ to install FreeProxyScraper, along with the tools you need to develop, run the f
pip install -e .[dev]
```

If you'd like to contribute to development, right now the most needed thing is writing more plugins. In order to help, you need basic knowledge of BeautifulSoup4 and a little of patience with websites purposely making it hard for you to scrape information. Check out `src/plugins/examplePlugin.py` to see an example layout of a plugin file.
If you'd like to contribute to development, right now the most needed thing is writing more plugins. In order to help, you need basic knowledge of BeautifulSoup4 and a little of patience with websites purposely making it hard for you to scrape information. Check out `src/plugins/examplePlugin.py` to see an example layout of a plugin file.

0 comments on commit 85cf5e1

Please sign in to comment.