Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replacing in memory data with streams #185

Merged
merged 2 commits into from
Apr 19, 2022
Merged

Replacing in memory data with streams #185

merged 2 commits into from
Apr 19, 2022

Conversation

Mzack9999
Copy link
Member

Description

This PR replaces in memory data store with streaming from file, reducing the overall memory consumption

@Mzack9999 Mzack9999 added the Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. label Apr 13, 2022
@Mzack9999 Mzack9999 self-assigned this Apr 13, 2022
@Mzack9999 Mzack9999 linked an issue Apr 13, 2022 that may be closed by this pull request
big wordlist file not working. #178
Closed
@Mzack9999 Mzack9999 added Type: Maintenance Updating phrasing or wording to make things clearer or removing ambiguity. Status: Review Needed The issue has a PR attached to it which needs to be reviewed labels Apr 13, 2022
ehsandeep
ehsandeep reviewed Apr 18, 2022
View reviewed changes
Copy link
Member

@ehsandeep ehsandeep left a comment •
edited by Mzack9999
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory uses drops from 1.6GB to 1GB compared to the latest release while processing / running the scan, not sure if it's expected to use this amount of memory to run dnsx in wordlist mode.

Wordlist used for the testing: https://wordlists-cdn.assetnote.io/data/manual/best-dns-wordlist.txt, on mac system.

echo google.com | ./dnsx -d - -w best-dns-wordlist.txt

Todo:

  • Investigate freeze on large wordlists
  • Identify and make memory usage more efficient

@Mzack9999
Copy link
Member Author

  • After debugging the issue, it was determined that the memory usage is expected during targets generation (unless stream mode is used) due to the goleveldb compaction operation, then it drops to around 100Mb. The rework is about hmap, and it's tracked at [Feature] Switch from goleveldb to filekv with better in memory deduplication hmap#32.
  • The scan never freezes but takes a long time due to the 10M input target list of most non-existent entries.

@ehsandeep ehsandeep merged commit 9b19847 into dev Apr 19, 2022
@ehsandeep ehsandeep deleted the issue-178-huge-wordlists branch April 19, 2022 15:05
@ehsandeep ehsandeep added Status: Completed Nothing further to be done with this issue. Awaiting to be closed. and removed Status: Review Needed The issue has a PR attached to it which needs to be reviewed labels Apr 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Ice3man543 Ice3man543 Ice3man543 approved these changes

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees

@Mzack9999 Mzack9999

Labels
Status: Completed Nothing further to be done with this issue. Awaiting to be closed. Type: Bug Inconsistencies or issues which will cause an issue or problem for users or implementors. Type: Maintenance Updating phrasing or wording to make things clearer or removing ambiguity.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

big wordlist file not working.
3 participants
@Mzack9999 @ehsandeep @Ice3man543