fix(parsers): use_proxy decorator to not override default args #7686
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When
WEBPROXY_USERNAME
/WEBPROXY_PASSWORD
is configured and ause_proxy
decorated parser function was called without args/kwargs, like done when usingpython -m parsers.KPX
instead ofpoetry run test-parsers KR
, then the following error showed up:It was because the
use_proxy
logic always passed azone_key
args explicitly, even if it wasn't received, which made aNone
value get passed explicitly and short-circuiting thefetch_...
functions default value.Changes
Makes calling
fetch_production()
without args or kwargs not run into issues stemming fromzone_key
being explicitly passed as None.Adds a comment on what webshare.io subscription is useful for use by this project.
Re-configures session.proxies back to what it was in a way that I think now will work, while previously I think we just stored a reference to the proxies object that we then also modified.
This likely won't impact anyone.
Adjusted logging level from Error to Warning. As it kept running without further complaints it felt more like a warning, and it could reasonably succeed without a proxy depending on where the code is running.
Adjusted the code for readability, which is an opinionated matter of course.
Double check
poetry run test_parser "zone_key"
Things still work, with and without having
WEBSHARE_...
environment configured.pnpx prettier@2 --write .
andpoetry run format
in the top level directory to format my changes.