Logging & parameterization improvements #15
Conversation
…start date and open_in_browser from command line. Pass list of permit types to crawler. Set built-in scrapy module logs to WARN. Improve error handling around mailtrap env vars.
Integrate changes from base
| attrs==24.2.0 | ||
| Automat==22.10.0 | ||
| certifi==2024.7.4 | ||
| cffi==1.17.0 | ||
| charset-normalizer==3.3.2 | ||
| constantly==23.10.4 | ||
| cryptography==43.0.0 | ||
| cssselect==1.2.0 | ||
| defusedxml==0.7.1 | ||
| filelock==3.15.4 | ||
| hyperlink==21.0.0 | ||
| idna==3.7 | ||
| incremental==24.7.2 | ||
| itemadapter==0.9.0 | ||
| itemloaders==1.3.1 | ||
| jmespath==1.0.1 | ||
| lxml==5.2.2 | ||
| packaging==24.1 | ||
| parsel==1.9.1 | ||
| Protego==0.3.1 | ||
| pyasn1==0.6.0 | ||
| pyasn1_modules==0.4.0 | ||
| pycparser==2.22 | ||
| PyDispatcher==2.0.7 | ||
| pyOpenSSL==24.2.1 | ||
| queuelib==1.7.0 | ||
| requests==2.32.3 | ||
| requests-file==2.1.0 | ||
| Jinja2==3.1.4 | ||
| mailtrap==2.0.1 | ||
| Scrapy==2.11.2 | ||
| service-identity==24.1.0 | ||
| six==1.16.0 | ||
| tldextract==5.1.2 | ||
| Twisted==24.3.0 | ||
| typing_extensions==4.12.2 | ||
| urllib3==2.2.2 | ||
| Twisted==24.7.0 | ||
| w3lib==2.2.1 | ||
| zope.interface==7.0.1 | ||
| mailtrap==2.0.1 | ||
| jinja2==3.0.3 |
There was a problem hiding this comment.
If you think this looks funny - I used https://github.com/fpgmaas/deptry to separate our direct dependencies (Scrapy, Twisted) from indirect dependencies (charset-normalizer, certifi) that were pulled in automatically.
One (of many) of the great things about using a dependency manager like Poetry (or Hatch or Rye) is that it separates your direct dependencies in pyproject.toml from indirect dependencies, which are stored in a separate poetry.lock. While converting this project to poetry would be quick, I don't want to single-handedly inflict a change on your personal development process by doing so.
There was a problem hiding this comment.
I'm game for switching to poetry. pip is just muscle memory for me at this point - I rarely even consider that package management has improved beyond the requirements.txt file.
There was a problem hiding this comment.
I'm happy to conduct that switch in a future PR if you want to see what that looks like. I don't know if it's worth sacrificing you & the other contributors' muscle memory for that on a project like this, though. Up to you!
| "--start_date", | ||
| required=False, | ||
| default=(date.today() - timedelta(days=1)).strftime("%m/%d/%Y"), |
There was a problem hiding this comment.
Change this to weeks = 1 in order to cause the recreational spider to hit #11
|
|
The sheer verbosity of scrapy's default logging was cramping my dev style, so I changed it. This PR also includes a few other tweaks.
start_dateandopen_in_browsermay be passed on the command line:python main.py --start_date '08/01/2023'.env.dev. In other words, if you create.env.devin your repo root withand then run
docker-compose up --build, those variables will be loaded in..env.devis listed in.gitignoreso there's no risk of accidentially committing it.MAILTRAP_CC_ADDRESSorMAILTRAP_BCC_ADDRESSno longer causes an errorstart_dateandpermit_typesare passed to Jinja, and the template's been tweaked to use them:Output from a run with default arguments:
Note that this change does NOT fix #11.
If you're tired of seeing tiny formatting changes in reviews like this one:
...will install pre-commit which'll run
ruffand otherwise enforce a consistent standard.