- One master server.
- Multiple "workers" which supply scraping network bandwidth + processing power.
- How to update worker code?
- Server sends the before/after parts of the job processing pipeline to the workers.
- This code is maintained in normal python code.
- Insert obtained data into database + create more jobs.
This repository has been archived by the owner on Jul 24, 2020. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
PyJaipur/web-scrape
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
Web scraping, march 2020
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published