Improved speed to v2 #847

hitenkoku · 2022-12-23T22:43:11Z

What Changed

removed unnecessary collect process to improve speed, from v2

Evidence

Analyzed evtx files that total size is 6.1GB

prev this pr (main)

>.\main.exe csv-timeline -d ..\all-evtx\ -o main.csv --debug -q

...

Start time: 2022/12/24 07:16

Analyzing event files: 1858
Total file size: 6.1 GB

Loading detections rules. Please wait.

Excluded rules: 15
Noisy rules: 7 (Disabled)

Experimental rules: 1951 (59.50%)
Stable rules: 218 (6.65%)
Test rules: 1110 (33.85%)

Hayabusa rules: 145
Sigma rules: 3134
Total enabled detection rules: 3279

1858 / 1858 [=============================================================================================================================================================================] 100.00 %

Analysis finished. Please wait while the results are being saved.

Rule Authors:

...

Results Summary:

Events with hits / Total events: 1,593,652 / 4,817,181 (Data reduction: 3,223,529 events (66.92%))

Total | Unique detections: 1,625,936 | 142
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,057 (0.74%) | 20 (14.08%)
Total | Unique medium detections: 9,955 (0.61%) | 34 (23.94%)
Total | Unique low detections: 1,053,425 (64.79%) | 38 (26.76%)
Total | Unique informational detections: 550,499 (33.86%) | 50 (35.21%)

Dates with most total detections:
critical: n/a, high: 2022-09-18 (4,441), medium: 2022-02-08 (4,793), low: 2022-09-18 (911,662), informational: 2022-03-02 (206,158)

Top 5 computers with most unique detections:
critical: n/a
high: evtx-PC (8), Agamemnon (8), DESKTOP-6D0DBMB (8), DESKTOP-A8CALR3 (6), WIN-FPV0DSIC9O6.sigma.fr (4)
medium: Agamemnon (21), DESKTOP-6D0DBMB (18), DESKTOP-A8CALR3 (16), evtx-PC (10), WIN-FPV0DSIC9O6 (9)
low: DESKTOP-6D0DBMB (22), Agamemnon (19), DESKTOP-A8CALR3 (15), evtx-PC (12), WIN-FPV0DSIC9O6.sigma.fr (10)
informational: DESKTOP-6D0DBMB (37), DESKTOP-A8CALR3 (36), WIN-TKC15D7KHUR (34), Agamemnon (31), WIN-FPV0DSIC9O6.sigma.fr (31)

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                        Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                         File Creation Date Changed to Another Year (10,490)            │
│ n/a                                         Windows Shell File Write to Suspicious Folder (991)            │
│ n/a                                         DLL Load By System Process From Suspicious Locations (406)     │
│ n/a                                         Proc Exec (Non-Exe Filetype) (45)                              │
│ n/a                                         SysmonEnte Usage (33)                                          │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                          Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (8,544)                     Proc Access (1,020,252)                                        │
│ Proc Injection (673)                        Possible Timestomping (31,784)                                 │
│ Process Ran With High Privilege (191)       Creation of an Executable by an Executable (411)               │
│ Use Short Name Path in Command Line (133)   Modified Rule in Windows Firewall with Advanced Security (254) │
│ Potentially Malicious PwSh (127)            Modification of IE Registry Settings (231)                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (422,037)                      Pipe Created (9,044)                                           │
│ File Deleted (53,696)                       Net Conn (8,755)                                               │
│ Proc Exec (18,780)                          DNS Query (5,108)                                              │
│ Pipe Conn (17,062)                          WMI Provider Started (681)                                     │
│ Proc Terminated (12,388)                    Suspicious Load of Advapi31.dll (420)                          │
╰───────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: main.csv (574.7 MB)
Elapsed time: 00:07:34.361
Rule Parse Processing Time: 00:00:02.311
Analysis Processing Time: 00:07:20.648
Output Processing Time: 00:00:11.400

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    6.1 GiB    6.1 GiB   56.0 MiB    6.1 GiB                        not all freed!
 committed:    4.8 GiB    6.5 GiB    1.7 GiB    4.7 GiB                        not all freed!
     reset:      0          0          0          0                            ok
   touched:  128.5 KiB   28.2 MiB   54.3 GiB  -54.2 GiB                        ok
  segments:     19        226        220          6                            not all freed!
-abandoned:      0          0          0          0                            ok
   -cached:      0          0          0          0                            ok
     pages:      0          0      741.2 Ki  -741.2 Ki                         ok
-abandoned:      0          0          0          0                            ok
 -extended:      0
 -noretire:      0
     mmaps:      0
   commits:    2.0 Ki
   threads:     32         32          0         32                            not all freed!
  searches:     0.0 avg
numa nodes:       1
   elapsed:     454.364 s
   process: user: 3090.375 s, system: 74.203 s, faults: 1217953, rss: 4.5 GiB, commit: 4.8 GiB

this pr

 >.\pr847.exe csv-timeline -d ..\all-evtx\ -o pr847.csv --debug -q
Start time: 2022/12/24 08:21

Analyzing event files: 1858
Total file size: 6.1 GB

Loading detections rules. Please wait.

Excluded rules: 15
Noisy rules: 7 (Disabled)

Experimental rules: 1951 (59.50%)
Stable rules: 218 (6.65%)
Test rules: 1110 (33.85%)

Hayabusa rules: 145
Sigma rules: 3134
Total enabled detection rules: 3279

1858 / 1858 [===============================================================================================] 100.00 %

Analysis finished. Please wait while the results are being saved.

Rule Authors:

...

Results Summary:

Events with hits / Total events: 1,593,652 / 4,817,181 (Data reduction: 3,223,529 events (66.92%))

Total | Unique detections: 1,625,936 | 142
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,057 (0.74%) | 20 (14.08%)
Total | Unique medium detections: 9,955 (0.61%) | 34 (23.94%)
Total | Unique low detections: 1,053,425 (64.79%) | 38 (26.76%)
Total | Unique informational detections: 550,499 (33.86%) | 50 (35.21%)

Dates with most total detections:
critical: n/a, high: 2022-09-18 (4,441), medium: 2022-02-08 (4,793), low: 2022-09-18 (911,662), informational: 2022-03-02 (206,158)

Top 5 computers with most unique detections:
critical: n/a
high: evtx-PC (8), Agamemnon (8), DESKTOP-6D0DBMB (8), DESKTOP-A8CALR3 (6), WIN-FPV0DSIC9O6.sigma.fr (4)
medium: Agamemnon (21), DESKTOP-6D0DBMB (18), DESKTOP-A8CALR3 (16), evtx-PC (10), WIN-FPV0DSIC9O6 (9)
low: DESKTOP-6D0DBMB (22), Agamemnon (19), DESKTOP-A8CALR3 (15), evtx-PC (12), WIN-FPV0DSIC9O6.sigma.fr (10)
informational: DESKTOP-6D0DBMB (37), DESKTOP-A8CALR3 (36), WIN-TKC15D7KHUR (34), Agamemnon (31), WIN-FPV0DSIC9O6.sigma.fr (31)

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Top critical alerts:                        Top high alerts:                                               │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ n/a                                         File Creation Date Changed to Another Year (10,490)            │
│ n/a                                         Windows Shell File Write to Suspicious Folder (991)            │
│ n/a                                         DLL Load By System Process From Suspicious Locations (406)     │
│ n/a                                         Proc Exec (Non-Exe Filetype) (45)                              │
│ n/a                                         SysmonEnte Usage (33)                                          │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top medium alerts:                          Top low alerts:                                                │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Raw Access Read (8,544)                     Proc Access (1,020,252)                                        │
│ Proc Injection (673)                        Possible Timestomping (31,784)                                 │
│ Process Ran With High Privilege (191)       Creation of an Executable by an Executable (411)               │
│ Use Short Name Path in Command Line (133)   Modified Rule in Windows Firewall with Advanced Security (254) │
│ Potentially Malicious PwSh (127)            Modification of IE Registry Settings (231)                     │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ Top informational alerts:                                                                                  │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ File Created (422,037)                      Pipe Created (9,044)                                           │
│ File Deleted (53,696)                       Net Conn (8,755)                                               │
│ Proc Exec (18,780)                          DNS Query (5,108)                                              │
│ Pipe Conn (17,062)                          WMI Provider Started (681)                                     │
│ Proc Terminated (12,388)                    Suspicious Load of Advapi31.dll (420)                          │
╰───────────────────────────────────────────╌────────────────────────────────────────────────────────────────╯

Saved file: pr847.csv (574.7 MB)
Elapsed time: 00:07:24.609
Rule Parse Processing Time: 00:00:01.827
Analysis Processing Time: 00:07:11.826
Output Processing Time: 00:00:10.954

Memory usage stats:
heap stats:    peak      total      freed    current       unit      count
  reserved:    6.1 GiB    6.1 GiB   56.0 MiB    6.1 GiB                        not all freed!
 committed:    4.7 GiB    6.5 GiB    1.8 GiB    4.7 GiB                        not all freed!
     reset:      0          0          0          0                            ok
   touched:  128.5 KiB   28.7 MiB   54.3 GiB  -54.2 GiB                        ok
  segments:     19        230        224          6                            not all freed!
-abandoned:      0          0          0          0                            ok
   -cached:      0          0          0          0                            ok
     pages:      0          0      741.2 Ki  -741.2 Ki                         ok
-abandoned:      0          0          0          0                            ok
 -extended:      0
 -noretire:      0
     mmaps:      0
   commits:    2.0 Ki
   threads:     32         32          0         32                            not all freed!
  searches:     0.0 avg
numa nodes:       1
   elapsed:     444.612 s
   process: user: 3083.781 s, system: 69.156 s, faults: 1256446, rss: 4.5 GiB, commit: 4.7 GiB

…ctString

YamatoSecurity · 2022-12-24T01:36:54Z

@hitenkoku 結構改善して下さっていますね！ありがとうございます！
変更が多いので、時間をかけてゆっくり検証したいと思います。(年末の勉強会でバタバタしていて、年明けになりそうなので、急ぐ必要はありません。）
今日は今のmainブランチのv2をリリースしたいと思います。＾＾ｖ

fukusuket · 2022-12-24T13:45:18Z

M1 Mac(メモリ8GB, 8コア)でもいつもの6.1GB evtxで以下の改善(と出力ファイルサイズの一致)を確認いたしました💪
LGTMです！！🚀

検証コマンド

 ./hayabusa-main csv-timeline -d ../all-evtx -o o.csv -p super-verbose

1回目 7秒改善

rev	time
2.0.0	Elapsed time: 00:10:03.001
3c931b7	Elapsed time: 00:09:56.566

2回目 11秒改善

rev	time
2.0.0	Elapsed time: 00:10:22.525
3c931b7	Elapsed time: 00:10:11.353

(Macでベンチマークを安定して取る方法は現在模索中です... 今のところWindowsほど安定させられず...)

YamatoSecurity · 2022-12-25T01:48:41Z

検証がもっと時間がかかると思いましたが、できました。LGTM! マージしましょう。
取り敢えずバージョンを2.0.1に上げました。

ベンチマークは通りです:

1.8.1: 00:08:02.760
1.9.0: 00:07:56.696
2.0.0: 00:09:10.591
2.0.1: 00:08:53.038

YamatoSecurity

LGTM!

hitenkoku · 2022-12-25T11:51:09Z

ありがとうございます。それではマージしておきます。

hitenkoku added 17 commits December 18, 2022 00:22

to improve speed, removed unnecessary to_string call

74fe5b3

to improve performance, removed unnecessary to_string

320f5c6

to improve speed, removed unnecessary to_string

75ff713

cargo fmt

44ed4b6

to improve speed, removed unnecessary collect

1c19a28

to improve performance, changed record_info type from String to Compa…

90d179b

…ctString

to imporve performance, removed unnecessary variable assignment

393a8cd

to improve performance, replaced if statement with match statement

57f71f6

to improve performance, removed unnecessary to_string

faac347

to improve process, removed unnecessary collect

6b5cae2

to improve performance, replaced collect with iterator

670b8bf

to improve performance removed unnecessary to_string

ba2ba2d

removed unnecessary variable assignment

8919805

removed unnecessary to_owned

2189a23

cargo fmt

76ba2ac

Merge branch 'main' into to_improve_speed_656

94aa985

fix: compile error after merge

cffdadf

hitenkoku requested review from fukusuket and YamatoSecurity December 23, 2022 22:43

hitenkoku added 2 commits December 24, 2022 08:04

to improve speed, removed unnecessary collect processing

5345a63

to need collect for parallel processing in detection

3d43caa

hitenkoku self-assigned this Dec 23, 2022

hitenkoku added the enhancement New feature or request label Dec 23, 2022

hitenkoku marked this pull request as ready for review December 23, 2022 23:32

hitenkoku changed the title ~~To improve speed 656~~ Improved speed to v2 Dec 23, 2022

hitenkoku mentioned this pull request Dec 23, 2022

enhancement separate help menu by commands #832

Merged

hitenkoku added 2 commits December 24, 2022 08:40

fixed separator typo

9e0b63d

fix(pkl): to improve speed, removed collect #847

7920d35

fix(pkl): wrong size_hint check

3c931b7

fukusuket approved these changes Dec 24, 2022

View reviewed changes

YamatoSecurity added 2 commits December 25, 2022 10:37

update cargo version

70578dd

update changelog

e23f61e

YamatoSecurity approved these changes Dec 25, 2022

View reviewed changes

hitenkoku merged commit 4709f1e into main Dec 25, 2022

hitenkoku deleted the to_improve_speed_656 branch December 25, 2022 11:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved speed to v2 #847

Improved speed to v2 #847

hitenkoku commented Dec 23, 2022 •

edited

Loading

YamatoSecurity commented Dec 24, 2022

fukusuket commented Dec 24, 2022 •

edited

Loading

YamatoSecurity commented Dec 25, 2022

YamatoSecurity left a comment

hitenkoku commented Dec 25, 2022

Improved speed to v2 #847

Improved speed to v2 #847

Conversation

hitenkoku commented Dec 23, 2022 • edited Loading

What Changed

Evidence

YamatoSecurity commented Dec 24, 2022

fukusuket commented Dec 24, 2022 • edited Loading

YamatoSecurity commented Dec 25, 2022

YamatoSecurity left a comment

Choose a reason for hiding this comment

hitenkoku commented Dec 25, 2022

hitenkoku commented Dec 23, 2022 •

edited

Loading

fukusuket commented Dec 24, 2022 •

edited

Loading