Investigation of increased memory usage

[fukusuket]

Investigate the possibility of increased memory usage in later releases of the low-memory feature.

[fukusuket]

2.14.0

% ./hayabusa-2.14.0-mac-aarch64 csv-timeline -d ../all-evtx -o big.csv -w -s --debug

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/06/16 14:32

Total event log files: 2,239
Total file size: 8.8 GB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12 (Disabled)

Deprecated rules: 204 (4.98%) (Disabled)
Experimental rules: 1,019 (24.88%)
Stable rules: 240 (5.86%)
Test rules: 2,836 (69.26%)
Unsupported rules: 45 (1.10%) (Disabled)

Hayabusa rules: 162
Sigma rules: 3,933
Total enabled detection rules: 4,095

Output profile: standard

Scanning in progress. Please wait.

[00:08:04] 2,239 / 2,239   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (80)                  Nasreddine Bencherchali (61)   frack113 (52)                      Florian Roth (35)               │
│ oscd.community (28)               Tim Shelton (14)               Roberto Rodriguez @Cyb3r... (10)   Daniil Yugoslavskiy (9)         │
│ Roberto Rodriguez (9)             OTR (8)                        Victor Sergeev (7)                 Timur Zinniatullin (7)          │
│ Gleb Sukhodolskiy (5)             Bhabesh Raj (4)                Sander Wiebing (3)                 Wietze Beukema (3)              │
│ Markus Neis (3)                   Jonhnathan Ribeiro (3)         Michael Haag (3)                   Patrick Bareiss (2)             │
│ Christopher Peacock @sec... (2)   Ján Trenčanský (2)             SOC Prime (2)                      Thomas Patzke (2)               │
│ Sreeman (2)                       Oddvar Moe (2)                 James Pemberton@4A616D65... (2)    KarneadesMarkus Neis (2)        │
│ Anton Kutepov (2)                 Mark Woan (2)                  Center for Threat Inform... (2)    Endgame (2)                     │
│ Jakob Weinzettl (2)               Teymur Kheirkhabarov (2)       @gott_cyber (2)                    JHasenbusch (2)                 │
│ Fukusuke Takahashi (2)            Alexandr Yampolskyi (2)        Swachchhanda Shrawan Poudel (2)    SCYTHE @scythe_io (2)           │
│ Samir Bousseaden (1)              Matthew Green @mgreen27 (1)    Luc Génaux (1)                     Ecco (1)                        │
│ Andreas Hunkeler (1)              D3F7A5105 (1)                  Connor Martin (1)                  Stephen Lincoln @slincol... (1) │
│ Harish Segar (1)                  Eric Conrad (1)                xorxes (1)                         Zach Stanford @svch0st (1)      │
│ pH-T (1)                          Thurein Oo (1)                 Tim Rauch (1)                      Dimitrios Slamaris (1)          │
│ FPT.EagleEye (1)                  @neu5ron (1)                   Open Threat Research (1)           Cybex (1)                       │
│ Tom Kern (1)                      Aleksey Potapov (1)            AlertIQ (1)                        X__Junior (1)                   │
│ Elastic (1)                       Anish Bogati (1)               Cédric Hien (1)                    James Pemberton @4A616D6573 (1) │
│ Mark Russinovich (1)              @redcanary (1)                 Timur Zinniatullin oscd.... (1)    Yusuke Matsui (1)               │
│ Joshua Wright (1)                 Maxime Thiebaut (1)            xknow (1)                          @0xrawsec (1)                   │
│ Dmitry Uchakin (1)                Natalia Shornikova (1)         Max Altgelt (1)                    mdecrevoisier (1)               │
╰─────────────────────────────────╌──────────────────────────────╌──────────────────────────────────╌─────────────────────────────────╯

Results Summary:

Events with hits / Total events: 2,451,184 / 6,611,184 (Data reduction: 4,160,000 events (62.92%))

Total | Unique detections: 2,504,040 | 269
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 18,160 (0.73%) | 34 (21.19%)
Total | Unique medium detections: 33,724 (1.35%) | 109 (25.65%)
Total | Unique low detections: 1,701,040 (67.93%) | 69 (40.52%)
Total | Unique informational detections: 751,116 (30.00%) | 57 (12.64%)

Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,517), medium: 2023-11-06 (18,239), low: 2022-09-18 (912,894), informational: 2022-03-02 (206,023)

Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (21), DESKTOP-A8CALR3 (8), DESKTOP-6D0DBMB (8), evtx-PC (7), Agamemnon (7)
medium: WinDev2310Eval (79), Agamemnon (36), DESKTOP-A8CALR3 (22), DESKTOP-6D0DBMB (21), evtx-PC (14)
low: WinDev2310Eval (43), DESKTOP-6D0DBMB (36), DESKTOP-A8CALR3 (30), Agamemnon (30), evtx-PC (19)
informational: WinDev2310Eval (41), DESKTOP-6D0DBMB (39), DESKTOP-A8CALR3 (37), WIN-TKC15D7KHUR (35), WIN-FPV0DSIC9O6.sigma.fr (34)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                               Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                                File Creation Date Changed to Another Year (15,884)            │
│ n/a                                                                Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a                                                                EVTX Created In Uncommon Location (986)                        │
│ n/a                                                                Proc Exec (Non-Exe Filetype) (60)                              │
│ n/a                                                                File Download with Headless Browser (60)                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                 Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311)                                           Proc Access (1,613,391)                                        │
│ Process Ran With High Privilege (7,673)                            Possible Timestomping (71,065)                                 │
│ Potential Credential Dumping Activity Via LSASS (6,135)            Scheduled Task Created - Registry (8,185)                      │
│ LSASS Access From Program In Potentially Suspicious F... (2,396)   Shell Context Menu Command Tampering (4,283)                   │
│ Uncommon New Firewall Rule Added In Windows Firewall ... (918)     System Drawing DLL Load (1,038)                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                                         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (542,438)                                             Net Conn (14,234)                                              │
│ File Deleted (94,703)                                              Pipe Created (10,602)                                          │
│ Pipe Conn (39,460)                                                 DNS Query (7,251)                                              │
│ Proc Exec (23,695)                                                 WMI Provider Started (706)                                     │
│ Proc Terminated (14,674)                                           Suspicious High IntegrityLevel Conhost Legacy Option (322)     │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: big.csv (2.3 GB)

Elapsed time: 00:08:05.1846

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_144044.log for details.

Rule Parse Processing Time: 00:00:01.752
Analysis Processing Time: 00:08:04.970
Output Processing Time: 00:00:00.099

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     1.0 GiB     2.0 GiB   757.7 GiB  -755.7 GiB                          ok
     reset:     0
    purged:    48.9 GiB
   touched:   128.5 KiB    20.0 MiB    99.3 GiB   -99.3 GiB                          ok
  segments:    19         320         308          12                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0           1.1 Mi     -1.1 Mi                           ok
-abandoned:     5           5           0           5                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    21.0 Ki
   threads:    17          17           1          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   486.869 s
   process: user: 3184.333 s, system: 34.669 s, faults: 305, rss: 1.1 GiB, commit: 1.0 GiB

2.15.0

Elapsed time: 00:08:19.1011

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_145404.log for details.

Rule Parse Processing Time: 00:00:01.621
Analysis Processing Time: 00:08:18.267
Output Processing Time: 00:00:00.099

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     1.0 GiB     2.0 GiB   735.5 GiB  -733.5 GiB                          ok
     reset:     0
    purged:    48.1 GiB
   touched:   128.5 KiB    19.1 MiB    99.4 GiB   -99.3 GiB                          ok
  segments:    20         306         294          12                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0           1.1 Mi     -1.1 Mi                           ok
-abandoned:     2           2           0           2                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    23.6 Ki
   threads:    17          17           1          16                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:   499.984 s
   process: user: 3262.851 s, system: 35.713 s, faults: 292, rss: 1.1 GiB, commit: 1.0 GiB

[fukusuket]

2.16.0

% ./hayabusa-2.16.0-mac-aarch64 csv-timeline -d ../all-evtx -o big.csv -w -s --debug

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Start time: 2024/06/16 15:03

Total event log files: 2,239
Total file size: 8.8 GB

Loading detection rules. Please wait.

Excluded rules: 20
Noisy rules: 12 (Disabled)

Deprecated rules: 204 (4.98%) (Disabled)
Experimental rules: 1,019 (24.88%)
Stable rules: 240 (5.86%)
Test rules: 2,836 (69.26%)
Unsupported rules: 45 (1.10%) (Disabled)

Hayabusa rules: 162
Sigma rules: 3,933
Total detection rules: 4,095

Creating the channel filter. Please wait.

Evtx files loaded after channel filter: 145
Detection rules enabled after channel filter: 4,061

Output profile: standard

Scanning in progress. Please wait.

[00:07:45] 145 / 145   [========================================] 100%

Scanning finished. Please wait while the results are being saved.

Rule Authors:

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Zach Mathis (76)                  Nasreddine Bencherchali (61)   frack113 (52)                      Florian Roth (35)               │
│ oscd.community (28)               Tim Shelton (14)               Roberto Rodriguez @Cyb3r... (10)   Daniil Yugoslavskiy (9)         │
│ Roberto Rodriguez (9)             OTR (8)                        Victor Sergeev (7)                 Timur Zinniatullin (7)          │
│ Gleb Sukhodolskiy (5)             Bhabesh Raj (4)                Sander Wiebing (3)                 Wietze Beukema (3)              │
│ Markus Neis (3)                   Jonhnathan Ribeiro (3)         Michael Haag (3)                   Patrick Bareiss (2)             │
│ Christopher Peacock @sec... (2)   Ján Trenčanský (2)             SOC Prime (2)                      Thomas Patzke (2)               │
│ Sreeman (2)                       Oddvar Moe (2)                 James Pemberton@4A616D65... (2)    KarneadesMarkus Neis (2)        │
│ Anton Kutepov (2)                 Mark Woan (2)                  Center for Threat Inform... (2)    Endgame (2)                     │
│ Jakob Weinzettl (2)               Teymur Kheirkhabarov (2)       @gott_cyber (2)                    JHasenbusch (2)                 │
│ Fukusuke Takahashi (2)            Alexandr Yampolskyi (2)        Swachchhanda Shrawan Poudel (2)    SCYTHE @scythe_io (2)           │
│ Samir Bousseaden (1)              Matthew Green @mgreen27 (1)    Luc Génaux (1)                     Ecco (1)                        │
│ Andreas Hunkeler (1)              D3F7A5105 (1)                  Connor Martin (1)                  Stephen Lincoln @slincol... (1) │
│ Harish Segar (1)                  Eric Conrad (1)                xorxes (1)                         Zach Stanford @svch0st (1)      │
│ pH-T (1)                          Thurein Oo (1)                 Tim Rauch (1)                      Dimitrios Slamaris (1)          │
│ FPT.EagleEye (1)                  @neu5ron (1)                   Open Threat Research (1)           Cybex (1)                       │
│ Tom Kern (1)                      Aleksey Potapov (1)            AlertIQ (1)                        X__Junior (1)                   │
│ Elastic (1)                       Anish Bogati (1)               Cédric Hien (1)                    James Pemberton @4A616D6573 (1) │
│ Mark Russinovich (1)              @redcanary (1)                 Timur Zinniatullin oscd.... (1)    Yusuke Matsui (1)               │
│ Joshua Wright (1)                 Maxime Thiebaut (1)            xknow (1)                          @0xrawsec (1)                   │
│ Dmitry Uchakin (1)                Natalia Shornikova (1)         Max Altgelt (1)                    mdecrevoisier (1)               │
╰─────────────────────────────────╌──────────────────────────────╌──────────────────────────────────╌─────────────────────────────────╯

Results Summary:

Events with hits / Total events: 2,450,614 / 6,463,018 (Data reduction: 4,012,404 events (62.08%))

Total | Unique detections: 2,503,458 | 265
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 18,160 (0.73%) | 34 (20.75%)
Total | Unique medium detections: 33,389 (1.33%) | 107 (26.04%)
Total | Unique low detections: 1,701,040 (67.95%) | 69 (40.38%)
Total | Unique informational detections: 750,869 (29.99%) | 55 (12.83%)

Dates with most total detections:
critical: n/a, high: 2023-11-06 (5,517), medium: 2023-11-06 (18,239), low: 2022-09-18 (912,894), informational: 2022-03-02 (206,005)

Top 5 computers with most unique detections:
critical: n/a
high: WinDev2310Eval (21), DESKTOP-A8CALR3 (8), DESKTOP-6D0DBMB (8), evtx-PC (7), Agamemnon (7)
medium: WinDev2310Eval (79), Agamemnon (35), DESKTOP-A8CALR3 (20), DESKTOP-6D0DBMB (20), evtx-PC (14)
low: WinDev2310Eval (43), DESKTOP-6D0DBMB (36), DESKTOP-A8CALR3 (30), Agamemnon (30), evtx-PC (19)
informational: WinDev2310Eval (39), DESKTOP-6D0DBMB (38), DESKTOP-A8CALR3 (37), WIN-TKC15D7KHUR (35), WIN-FPV0DSIC9O6.sigma.fr (32)

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                                               Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                                                File Creation Date Changed to Another Year (15,884)            │
│ n/a                                                                Windows Shell/Scripting Application File Write to Sus... (991) │
│ n/a                                                                EVTX Created In Uncommon Location (986)                        │
│ n/a                                                                Proc Exec (Non-Exe Filetype) (60)                              │
│ n/a                                                                File Download with Headless Browser (60)                       │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                                                 Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (12,311)                                           Proc Access (1,613,391)                                        │
│ Process Ran With High Privilege (7,673)                            Possible Timestomping (71,065)                                 │
│ Potential Credential Dumping Activity Via LSASS (6,135)            Scheduled Task Created - Registry (8,185)                      │
│ LSASS Access From Program In Potentially Suspicious F... (2,396)   Shell Context Menu Command Tampering (4,283)                   │
│ Uncommon New Firewall Rule Added In Windows Firewall ... (918)     System Drawing DLL Load (1,038)                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                                         │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (542,438)                                             Net Conn (14,234)                                              │
│ File Deleted (94,703)                                              Pipe Created (10,602)                                          │
│ Pipe Conn (39,460)                                                 DNS Query (7,251)                                              │
│ Proc Exec (23,695)                                                 WMI Provider Started (706)                                     │
│ Proc Terminated (14,674)                                           Suspicious High IntegrityLevel Conhost Legacy Option (322)     │
╰──────────────────────────────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: big.csv (2.3 GB)

Elapsed time: 00:07:48.1457

Please report any issues with Hayabusa rules to: https://github.com/Yamato-Security/hayabusa-rules/issues
Please report any false positives with Sigma rules to: https://github.com/SigmaHQ/sigma/issues
Please submit new Sigma rules with pull requests to: https://github.com/SigmaHQ/sigma/pulls
Errors were generated. Please check ./logs/errorlog-20240616_151108.log for details.

Rule Parse Processing Time: 00:00:01.587
Analysis Processing Time: 00:07:47.737
Output Processing Time: 00:00:00.105

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     3.1 GiB     3.1 GiB   108.0 MiB     3.0 GiB
 committed:     1.0 GiB     3.1 GiB   814.8 GiB  -811.7 GiB                          ok
     reset:     0
    purged:    48.4 GiB
   touched:   128.5 KiB    28.5 MiB    92.2 GiB   -92.2 GiB                          ok
  segments:    20         457         445          12                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           1.1 Mi     -1.1 Mi                           ok
-abandoned:     2           2           2           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     3
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    33.7 Ki
   threads:    17          17           1          16                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:   469.481 s
   process: user: 3171.723 s, system: 34.336 s, faults: 316, rss: 1.1 GiB, commit: 1.0 GiB

[fukusuket]

Memory usage increased by 1GB with version 2.16.0. Thus, it appears to be an effect of the changes in version 2.16.0🤔

• v2.15.0...v2.16.0

[fukusuket]

At the time the following PRs were merged, memory usage had already increased.

• feat: add support for Data[x] notation and Provider_Name(field mapping feature's config) #1352

[fukusuket]

memo:
memory usage:

• 3.1GB 94e8e19
• 2.1GB 8bc08d2
• 2.1GB 8cde8f8
• 2.1GB 36bf6cf
• 2.1GB 05ad734

[YamatoSecurity]

@fukusuket Are you making sure you are testing with the same rules? I just tried with the current rules but version 2.14.0 does not support windash so many parsing errors which results in less rules being loaded which is probably why less memory is being used. You might need to convert windash rules to something compatible for 2.14.0 like we did in the past in order to do a proper comparison.

[YamatoSecurity]

Also, when I am testing on my intel mac, the total committed amount of memory will change about 1GB so running 2.15.0 and 2.16.0 several times will sometimes result in 2GB and sometimes 3GB for each version. Sometimes 2.15.0 uses less memory but sometimes it will use 3GB and 2.16.0 uses only 2GB of memory. So I do not think this is reliable.
In my experiences, comparing the rss and commit values in the last line: process: user: 3171.723 s, system: 34.336 s, faults: 316, rss: 1.1 GiB, commit: 1.0 GiB is usually more reliable for me.

[fukusuket]

@YamatoSecurity

Are you making sure you are testing with the same rules?

Yes! I am comparing using version 2.14.0 rules, so there is no difference in the number of rules when comparing.
In my environment, the peak/reserved value is stable and I have been able to compare with this value without problems so far, but I will check!

[YamatoSecurity]

@fukusuket I see! In that case, no problem. Thanks for looking into this!

[fukusuket]

Memory usage has increased by 1GB since the commit 94e8e19
It appears that our code change is not the cause, but a version change in one of the dependent libraries...

There are so many changes in the dependent libraries that it is difficult to identify which library is responsible...😇

[YamatoSecurity]

@fukusuket That's too bad. In my environment I do not notice a difference in memory usage so it may depend on the CPU architecture. If there is a regression, it might be nice to report it to the crate owner but I would rather stick to using the latest crate versions than use old crates unless there is really a significant degradation in performance or quality.

[fukusuket]

@YamatoSecurity
Yes, I agree! Since it was not an implementation issue here, I will close it as an issue for now :)