Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MaxMind performance improvement #24

Merged
merged 3 commits into from
Jan 9, 2022
Merged

MaxMind performance improvement #24

merged 3 commits into from
Jan 9, 2022

Conversation

chr0mag
Copy link
Owner

@chr0mag chr0mag commented Jan 9, 2022

Instead of appending to files on disk, sets are now sorted in memory and written out to disk all at once. This reduces the time to build all ipv4 and ipv6 MaxMind sets for both ipsets and nftables on my Ryzen 5 5600X from ~45s to ~3s.

# v2.3.1 -- before
(git)-[tags/v2.3.1] % time python -m geoipsets -c ~/geoipsets.conf --provider maxmind --output-dir ~/tests
Building geoipsets...
python -m geoipsets -c ~/geoipsets.conf --provider maxmind --output-dir   26.43s user 16.63s system 96% cpu **44.838 total**

# maxmind-perf branch --after
(git)-[maxmind-perf] % time python -m geoipsets -c ~/geoipsets.conf --provider maxmind --output-dir ~/tests
Building geoipsets...
python -m geoipsets -c ~/geoipsets.conf --provider maxmind --output-dir   1.80s user 0.07s system 56% cpu **3.315 total**

Fixes #16

Julian Phillips added 3 commits January 8, 2022 19:47
Instead of appending to files on disk, sets are now generated in
memory and written out to disk all at once.
@chr0mag chr0mag merged commit 15d011b into main Jan 9, 2022
@chr0mag chr0mag deleted the maxmind-perf branch January 9, 2022 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[MaxMind] Improve performance of set generation
1 participant