Filter links before downloading / adding to the queue #175
Conversation
This commit speeds up scraping for scenarios where pages have a high branch factor, that is many links and a majority of these links is excluded by the --exclude / --include rules. This also improves memory usage in these scenarios since the links are not stored. Also the network traffic is reduced by not downloading these links in the first place.
Codecov Report
@@ Coverage Diff @@
## master #175 +/- ##
==========================================
- Coverage 64.73% 64.45% -0.28%
==========================================
Files 17 17
Lines 621 633 +12
==========================================
+ Hits 402 408 +6
- Misses 219 225 +6
|
This allows fine-grained control whether a page is visited, that means its links analyzed, and saved to disk. The decoupling of download and visit filter means the complete website may still be explored while only downloading some files. To speed up scraping, irrelevant links can be easily excluded from visiting.
|
I thought a bit more, and what we want is separate regex filters for downloading and visiting:
To put in other words:
If this is merged, I advise to use a squash commit to hide the intermediate development. |
|
I always fail to think of useful tests. I could just duplicate the existing tests in |
You could put valid link (that should be downloaded) to visit in a page that should not be visited and check if the valid link has been downloaded or not. |
Otherwise a failed test leaves the directory populated, which fails future test runs.
|
Basically just duplicated the existing tests and adjusted for visit filter regex. |
|
Any update on this? It's almost a bug without the feature at this point, --exclude and --include are not doing what they are supposed to 😢 |
|
Sorry for the delay, I will review this tomorrow |
|
Humm the CI is not running |
|
We are in this case... https://github.saobby.my.eu.orgmunity/t/missing-approve-and-run-button/200572 |
|
The CI works perfectly on master, all good 🚀 🔥 |
This commit speeds up scraping for scenarios where pages have a high branch factor, that is many links and a majority of these links is excluded by the --exclude / --include rules.
This also improves memory usage in these scenarios since the links are not stored.
Also the network traffic is reduced by not downloading these links in the first place.