planet_n_spider v0.0.4 / dependency updates and CI/CD migration to Poetry v2.0 #125
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- feat: hide cookiebanner in thumbnails (by forging the cookiebanner response) - Planet-N uses the "Simple Cookie Control" WordPress Plugin (see: https://wordpress.com/plugins/simple-cookie-control), which sets a cookie after accepting/declining the dialog (for 6 months)
- see: https://github.com/browserless/browserless/pkgs/container/chrome/327417313?tag=v2.24.1 - changelog: https://github.com/browserless/browserless/blob/main/CHANGELOG.md#v2241 - change: to keep the docker-compose.yml in sync with the Kubernetes Helm Chart for Staging and Prod, switching back to "headless/chrome" (instead of the recently used "headless/chromium" image)
Criamos
added
enhancement
Criamos
force-pushed
the
planet_n_cookiebanner
branch
from
c6e3e46 to
f0f01bc
Compare
- this fixes the playwright wheel warning that first occurred with v1.49.0 (see: https://github.com/microsoft/playwright-python/releases/tag/v1.49.1)
- the recent browserless update apparently changed the API endpoint from "ws://localhost:3000" to "ws://localhost:3000/chrome/playwright", which can be accessed via playwright's "Browsertype.connect()"-method - the "connect_over_cdp()"-method seems no longer necessary for future reference: - the docs at https://docs.browserless.io/#connect-using-puppeteer-or-playwright appear to be outdated and don't reflect these recent changes - use "http://localhost:3000/docs" to look up the most-current documentation for the browserless playwright integration instead
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes the following changes:
planet_n_spiderv0.0.4: hide cookie-banners of the "Simple Cookie Control" WordPress Pluginspider-classes (e.g., utility classes like thees_connector,license_mapperand other modules) with loguru to increase the readability and helpfulness of crawler logs, especially when encountering non-Scrapy-related log messagesscrapy.Spider.logger(see Scrapy Docs: Logging from Spiders))pyproject.tomlfile necessary (see: pyproject.toml specification)poetryinstallation to v2.0 the next time you open the project in your IDE! (restarting your IDE might be necessary afterwards)browserlessAPI Endpoint (see:converter/web_tools.py) for the most recentbrowserless-image in combination withplaywright v1.49.1