Add checksum validation for dbip #22
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 4 commits
January 1, 2022 18:18
Uses the BeautifulSoup4 library to scrape the sha1sum from the dbip webpage. Unfortunately no checksums are downloadable as files.
Option exists because the dbip checksum is retrieved via screen scrape. If the page changes, checksum validation will fail. In that case, the --no-checksum option can be used to workaround the issue until a fix is available. The flag applies to the MaxMind provider as well for consistency. Minor refactoring of maxmind.py: - use requests instead of urllib for downloads - pass file to be checksum'd all at once instead of 4K chunks
Moved hash calculations to read in files in 8K chunks. This doesn't change much for MaxMind since the hash is calculated over the compressed file (~4MB). For Dbip the hash is calculated over the uncompressed file (~25MB) and reading all that into memory at once is unecessary.
For Python 3.8 compatibility. BooleanOptionalAction was added in Python 3.9.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The valid checksum is pulled directly from the
dbipwebpage via screen scrape as its not available as a direct download. For this reason a new command line flag (--checksum/--no-checksum) has been added. In the event a change in the webpage breaks checksum validation, the--no-checksumflag will allowgeoipsetsto continue working.For consistency the flags apply to
MaxMindas well.This is a command line flag only and is not honoured when present in the configuration file.
Fixes #12