Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enabled low memory mode by default #1365

Merged
merged 7 commits into from
Jun 17, 2024

Conversation

hitenkoku
Copy link
Collaborator

What Changed

  • enabled low memory mode by default

I would appreciate it if you could check it out when you have time

@hitenkoku hitenkoku added the enhancement New feature or request label Jun 15, 2024
@hitenkoku hitenkoku self-assigned this Jun 15, 2024
@hitenkoku hitenkoku linked an issue Jun 15, 2024 that may be closed by this pull request
Copy link
Collaborator

@fukusuket fukusuket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I confirmed that it works :) LGTM!!🚀

hayabusa-sample-evtx

% ./hayabusa-2.16.0-mac-aarch64 csv-timeline -d ../hayabusa-sample-evtx -D -n -u -q -w -o old.csv --debug
% ./hayabusa-low-memory csv-timeline -d ../hayabusa-sample-evtx -D -n -u -q -w -o new.csv --debug -s
% diff old.csv new.csv

all-evtx

2.16.0
% ./hayabusa-2.16.0-mac-aarch64 csv-timeline -d ../all-evtx -D -n -u -q -w -o old-big.csv --debug
Rule Parse Processing Time: 00:00:01.799
Analysis Processing Time: 00:08:03.302
Output Processing Time: 00:01:58.302

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    16.0 GiB    17.1 GiB     1.1 GiB    16.0 GiB
This PR
% ./hayabusa-low-memory csv-timeline -d ../all-evtx -D -n -u -q -w --debug -o new-big.csv
...
Rule Parse Processing Time: 00:00:01.911
Analysis Processing Time: 00:08:57.270
Output Processing Time: 00:00:00.112

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     5.2 GiB     5.3 GiB   384.0 MiB     5.0 GiB

@fukusuket
Copy link
Collaborator

fukusuket commented Jun 15, 2024

Unrelated to this PR, but since memory usage has increased since low-memory was released, I will look into this as a separate issue. (In version 2.13.0, memory usage was 2 GB as shown below🤔)

I created an issue for the investigation.

@YamatoSecurity
Copy link
Collaborator

@hitenkoku Thank you! When low memory mode in enabled, we shouldn't be able to use -R, --remove-duplicate-data and -X, --remove-duplicate-detections, right? I can still add these options without getting an error so could you configure it so that -R and -X require the option -s?

Copy link
Collaborator

@YamatoSecurity YamatoSecurity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hitenkoku LGTM! Sorry, I figured out the bug, requires = "sort-events" needed to be requires = "sort_events" so I just updated it.

@YamatoSecurity YamatoSecurity merged commit 0f11e8b into main Jun 17, 2024
5 checks passed
@YamatoSecurity YamatoSecurity deleted the 1361-enable-low-memory-mode-by-default branch June 17, 2024 23:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable low memory mode by default
3 participants