New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement a browser-based spider #3

Open

turicas opened this issue Oct 26, 2019 · 0 comments

Owner

turicas commented Oct 26, 2019

The current spider is pretty fast, but it extracts static dependencies from HTML, JS and CSS code - if any dependency is loaded dynamically it won't archive it. We may use @ArturGaspar's scrapy-qtwebkit or https://github.com/scrapy-plugins/scrapy-splash to do the job (read the HAR for each response and fetch all resources loaded).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment