Reduct DetectInfo fields memory #806

fukusuket · 2022-11-14T01:04:04Z

What Changed

Reduced memory usage of DetectInfo Memory usage has almost doubled after introducing custom profiles #789
added Profile Enum in option/profile.rs and remove lazy_static PRELOAD_xx.
- to store the following data.
  - profiles.yaml key-value (before convertion)
  - converted value (ex. %Timestamp% -> 2013-10-24 01:16:13.843 +09:00)
- for the pattern matching of profile feature
changed DetectInfo fileds and behaviors as follows
- change ext_field type Nested<Vec<CompactString>> -> Vec<(CompactString, Profile)>(tuple)
- add is_condition to check whether the detected rule was [condition] based.
- clear details and record_information when the moment stored details string in ext_field
  - I think this change has the most affected on memory reduction in this PR.

Evidence

I got the benchmark with this environment and procedure.
Compared to version 1.8.0, I confirmed reduction of memory usage : -200MB(avg) -510MB(max peak).

version	avg mem	max mem	Elapsed Time
v180	3.13GB	peak 5.64GB	00:13:22.948
v190-dev	2.93GB	peak 5.13GB	00:13:18.767

v190-dev output is as follows.

Results Summary:

Events with hits / Total events: 1,593,663 / 4,817,181 (Data reduction: 3,223,518 events (66.92%))

Total | Unique detections: 1,629,377 | 143
Total | Unique critical detections: 0 (0.00%) | 0 (0.00%)
Total | Unique high detections: 12,041 (0.74%) | 22 (15.38%)
Total | Unique medium detections: 9,941 (0.61%) | 35 (24.48%)
Total | Unique low detections: 1,056,244 (64.83%) | 40 (27.97%)
Total | Unique informational detections: 551,151 (33.83%) | 46 (32.17%)
...
Elapsed time: 00:13:18.767
Saved file: out.csv (576.3 MB)

- add: Profile Enum for profile feature - remove: lazy_static value LOADED_PROFILE_ALIAS, PRELOAD_PROFILE, PRELOAD_PROFILE_REGEX. - chg: DetectInfo.ext_field Nested<Vec<CompactString>> to Vec<(CompactString, Profile)> - chg: clear DetectInfo.detail when convert detail in message.rs::insert

YamatoSecurity · 2022-11-14T07:28:11Z

@fukusuket
対応ありがとうございます！
Intel Macで14GBのデータに対して、-P super-verboseプロファイルで比較してみました。

1.8.0:
メモリ: 20 GB
処理時間: 00:30:04.921

1.9.0-dev:
メモリ: 18.5 GB (7.5%削減)
処理時間: 00:27:24.418 (9%加速)

検知数は一緒ですが、Events with hitsとData reductionの合計が変わっています。
1.8.0:

Events with hits / Total events: 2,705,252 / 14,213,054 (Data reduction: 11,507,802 events (80.97%))

Total | Unique detections: 3,925,540 | 749
Total | Unique critical detections: 104 (0.00%) | 21 (2.80%)
Total | Unique high detections: 451,919 (11.51%) | 303 (40.45%)
Total | Unique medium detections: 185,392 (4.72%) | 249 (33.24%)
Total | Unique low detections: 1,201,680 (30.61%) | 105 (14.02%)
Total | Unique informational detections: 2,086,445 (53.15%) | 71 (9.48%)

1.9.0-dev:

Events with hits / Total events: 2,706,807 / 14,213,054 (Data reduction: 11,506,247 events (80.96%))

Total | Unique detections: 3,925,540 | 749
Total | Unique critical detections: 104 (0.00%) | 21 (2.80%)
Total | Unique high detections: 451,919 (11.51%) | 303 (40.45%)
Total | Unique medium detections: 185,392 (4.72%) | 249 (33.24%)
Total | Unique low detections: 1,201,680 (30.61%) | 105 (14.02%)
Total | Unique informational detections: 2,086,445 (53.15%) | 71 (9.48%)

確認して頂けますか？

YamatoSecurity · 2022-11-14T09:07:19Z

-P minimalプロファイルでも比較してみました。

1.8.0:
メモリ: 9.26 GB
処理時間: 00:27:20.853

1.9.0-dev:
メモリ: 8 GB (約15%削減)
処理時間: 00:26:57.496 (約2%加速)

minimalプロファイルの方がメモリ使用が削減されますが、処理時間はそんな変わらないようですね。

参考に1.4.3も同じルールで試しました:
メモリ: 100%時に5.3GB、スキャン後に6.75GBに増えた (37%削減)
処理時間: 00:33:47.368 (25%減速)

Total events: 10012719
Data reduction: 7306619 events (72.97%)

Total detections: 3929895
Total critical detections: 104
Total high detections: 452326
Total medium detections: 189474
Total low detections: 1201564
Total informational detections: 2086427

Unique detections: 751
Unique critical detections: 21
Unique high detections: 305
Unique medium detections: 250
Unique low detections: 104
Unique informational detections: 71

fukusuket · 2022-11-14T11:22:39Z

@YamatoSecurity

検証いただき、ありがとうございます🙇 Events with hitsのリグレッションすみません、修正いたします！
現在の修正は、%Details%のメモリ削減を見込んでいるのですが(そのため、minimalのほうが効果が大きく見えると思われます)、%AllFieldInfo%も同じ手法で削減できないか確認いたします！

fukusuket · 2022-11-15T00:31:57Z

Events with hits件数不具合修正と、%AllFieldInfo%改善版作成いたしました！
%AllFieldInfo%でも%Details%と同様に途中でStringをクリアする処理を入れたところ、修正前後の結果は以下の通りでした。

version	profile	avg mem	max mem	elasped time
v180	default	3.13GB	5.64GB	00:13:22.948
v190-dev 03411da (%AllFieldInfo%改善後)	default	2.92GB	4.85GB	00:13:07.053
v190-dev 3dc9e80 (%AllFieldInfo%改善前)	super-verbose	4.91GB	9.88GB	00:13:08.920
v190-dev 03411da (%AllFieldInfo%改善後)	super-verbose	4.33GB	8.66GB	00:13:08.769

また%Details%の処理ももう少し改善(不要なvec/hashmapアクセスの削除)を加えたところ、少し速度改善したようです :)
手元で機能面リグレッションないか、もう少しテストいたします！

YamatoSecurity · 2022-11-15T12:12:49Z

@fukusuket あれ、Githubチェックは無事のようですが、cargoをアップデートしたら、ローカル環境で以下の警告が出ていました：

warning: use of deprecated associated function `chrono::TimeZone::timestamp`: use `timestamp_opt()` instead
   --> src/afterfact.rs:955:32
    |
955 |         let offset_sec = Local.timestamp(0, 0).offset().local_minus_utc();
    |                                ^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated associated function `chrono::TimeZone::ymd`: use `with_ymd_and_hms()` instead
   --> src/detections/detection.rs:258:32
    |
258 |         let default_time = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);
    |                                ^^^

warning: use of deprecated associated function `chrono::Date::<Tz>::and_hms`: Use and_hms_opt() instead
   --> src/detections/detection.rs:258:48
    |
258 |         let default_time = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);
    |                                                ^^^^^^^

warning: use of deprecated associated function `chrono::TimeZone::ymd`: use `with_ymd_and_hms()` instead
  --> src/detections/rule/count.rs:31:28
   |
31 |     let default_time = Utc.ymd(1977, 1, 1).and_hms(0, 0, 0);
   |                            ^^^

warning: use of deprecated associated function `chrono::Date::<Tz>::and_hms`: Use and_hms_opt() instead
  --> src/detections/rule/count.rs:31:44
   |
31 |     let default_time = Utc.ymd(1977, 1, 1).and_hms(0, 0, 0);
   |                                            ^^^^^^^

warning: use of deprecated associated function `chrono::TimeZone::timestamp`: use `timestamp_opt()` instead
   --> src/options/update.rs:223:44
    |
223 |         let mut latest_update_date = Local.timestamp(0, 0);
    |                                            ^^^^^^^^^

warning: `hayabusa` (lib) generated 6 warnings

念の為、確認をお願いできますか？

YamatoSecurity · 2022-11-15T12:28:55Z

-P super-verboseでもう一回試したら、20GB -> 18.5 -> 更に 16GBまで減りました！
とても良い感じです！

Events数も直りました。ありがとうございました！

YamatoSecurity

LGTM!
@hitenkoku どうですか？

fukusuket · 2022-11-15T13:36:28Z

@YamatoSecurity
パフォーマンスさっそく見ていただきありがとうございます🙇

warning: use of deprecated associated function `chrono::TimeZone::timestamp`

Chrono 0.4.23リリース(3days ago)で現在使っているいくつかの関数がdeprecatedになったことが原因のようです :(
別issueでChronoのdeprecate関数の削除を対応するのがよさそうかと思いました！
https://docs.rs/chrono/0.4.23/chrono/offset/trait.TimeZone.html#method.ymd
https://github.com/chronotope/chrono/releases/tag/v0.4.23

YamatoSecurity · 2022-11-15T13:53:02Z

なるほど、良いと思います。では、とりあえずChronoを0.4.22に固定しました。

Chrono 0.4.23リリース(3days ago)で現在使っているいくつかの関数がdeprecatedになったことが原因のようです :( 別issueでChronoのdeprecate関数の削除を対応するのがよさそうかと思いました！ https://docs.rs/chrono/0.4.23/chrono/offset/trait.TimeZone.html#method.ymd https://github.com/chronotope/chrono/releases/tag/v0.4.23

fukusuket · 2022-11-15T14:13:08Z

すみません...🙇 1点まだ不具合があり、%AllFieldInfo%に値ないときは、- (ハイフン)を出力するのがもともとの仕様だったため、そちら修正いたします！

…to 789-reduct-detectinfo-memory

fukusuket · 2022-11-15T15:50:57Z

%AllFieldInfo%に値ないときは、- (ハイフン)を出力する

↑の修正および
v180と比較して、-j, -J, -P minimal,-P (standard),-P all-field-info オプションの出力結果に不要な差分がないことを確認しました。手元でのテストは完了です。

@YamatoSecurity @hitenkoku d5636ca にてレビュー頂けますと幸いです、よろしくお願いいたします🙇

hitenkoku · 2022-11-15T19:38:16Z

この件は既にmainにマージされている 7266abf で対応済みです。

@fukusuket あれ、Githubチェックは無事のようですが、cargoをアップデートしたら、ローカル環境で以下の警告が出ていました：

warning: use of deprecated associated function `chrono::TimeZone::timestamp`: use `timestamp_opt()` instead
   --> src/afterfact.rs:955:32
    |
955 |         let offset_sec = Local.timestamp(0, 0).offset().local_minus_utc();
    |                                ^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated associated function `chrono::TimeZone::ymd`: use `with_ymd_and_hms()` instead
   --> src/detections/detection.rs:258:32
    |
258 |         let default_time = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);
    |                                ^^^

warning: use of deprecated associated function `chrono::Date::<Tz>::and_hms`: Use and_hms_opt() instead
   --> src/detections/detection.rs:258:48
    |
258 |         let default_time = Utc.ymd(1970, 1, 1).and_hms(0, 0, 0);
    |                                                ^^^^^^^

warning: use of deprecated associated function `chrono::TimeZone::ymd`: use `with_ymd_and_hms()` instead
  --> src/detections/rule/count.rs:31:28
   |
31 |     let default_time = Utc.ymd(1977, 1, 1).and_hms(0, 0, 0);
   |                            ^^^

warning: use of deprecated associated function `chrono::Date::<Tz>::and_hms`: Use and_hms_opt() instead
  --> src/detections/rule/count.rs:31:44
   |
31 |     let default_time = Utc.ymd(1977, 1, 1).and_hms(0, 0, 0);
   |                                            ^^^^^^^

warning: use of deprecated associated function `chrono::TimeZone::timestamp`: use `timestamp_opt()` instead
   --> src/options/update.rs:223:44
    |
223 |         let mut latest_update_date = Local.timestamp(0, 0);
    |                                            ^^^^^^^^^

warning: `hayabusa` (lib) generated 6 warnings

念の為、確認をお願いできますか？

hitenkoku

確認しました。chronoのlintエラーに関連したバージョン固定は不要ですのでそちらを戻していただければ他は問題ありません

Cargo.toml

hitenkoku

LGTM。ご対応ありがとうございます。

fukusuket added 5 commits November 13, 2022 00:02

fix failed unit tests.

fafbd09

fixed output json output format with details

defcfdd

fixed aggregation condition detection profile

2261d61

fix: output constant string values in profiles.yml

44f8efd

hitenkoku assigned fukusuket Nov 14, 2022

hitenkoku added the enhancement New feature or request label Nov 14, 2022

hitenkoku linked an issue Nov 14, 2022 that may be closed by this pull request

Memory usage has almost doubled after introducing custom profiles #789

Closed

fukusuket added 2 commits November 14, 2022 12:18

fixed clippy error.

697b412

fixed clippy error.

c1053a5

YamatoSecurity requested review from hitenkoku and YamatoSecurity November 14, 2022 05:59

fukusuket added 2 commits November 14, 2022 21:24

fixed Events with hits count issue

3dc9e80

clear unnecessary detect_info.record_infomation

03411da

YamatoSecurity added 2 commits November 15, 2022 20:56

update cargo and changelog

8801ab1

Merge branch 'main' into 789-reduct-detectinfo-memory

261a235

YamatoSecurity approved these changes Nov 15, 2022

View reviewed changes

fix chrono to 0.4.22

5e23977

fukusuket added 3 commits November 16, 2022 00:14

fix output "-" when %AllFieldInfo% is empty.

885f070

Merge remote-tracking branch 'origin/789-reduct-detectinfo-memory' in…

5cb1e01

…to 789-reduct-detectinfo-memory

apply cargo fmt

d5636ca

hitenkoku requested changes Nov 15, 2022

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

deprecated chrono function has been deleted, so revert version

9ec7d68

hitenkoku approved these changes Nov 16, 2022

View reviewed changes

hitenkoku merged commit bbbe65f into main Nov 16, 2022

hitenkoku deleted the 789-reduct-detectinfo-memory branch November 16, 2022 00:30

fukusuket mentioned this pull request Feb 18, 2023

Create developer's guide for performance enhancements #928

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduct DetectInfo fields memory #806

Reduct DetectInfo fields memory #806

fukusuket commented Nov 14, 2022 •

edited

Loading

YamatoSecurity commented Nov 14, 2022

YamatoSecurity commented Nov 14, 2022 •

edited

Loading

fukusuket commented Nov 14, 2022

fukusuket commented Nov 15, 2022 •

edited

Loading

YamatoSecurity commented Nov 15, 2022

YamatoSecurity commented Nov 15, 2022

YamatoSecurity left a comment

fukusuket commented Nov 15, 2022 •

edited

Loading

YamatoSecurity commented Nov 15, 2022

fukusuket commented Nov 15, 2022

fukusuket commented Nov 15, 2022 •

edited

Loading

hitenkoku commented Nov 15, 2022

hitenkoku left a comment

hitenkoku left a comment

Reduct DetectInfo fields memory #806

Reduct DetectInfo fields memory #806

Conversation

fukusuket commented Nov 14, 2022 • edited Loading

What Changed

Evidence

YamatoSecurity commented Nov 14, 2022

YamatoSecurity commented Nov 14, 2022 • edited Loading

fukusuket commented Nov 14, 2022

fukusuket commented Nov 15, 2022 • edited Loading

YamatoSecurity commented Nov 15, 2022

YamatoSecurity commented Nov 15, 2022

YamatoSecurity left a comment

Choose a reason for hiding this comment

fukusuket commented Nov 15, 2022 • edited Loading

YamatoSecurity commented Nov 15, 2022

fukusuket commented Nov 15, 2022

fukusuket commented Nov 15, 2022 • edited Loading

hitenkoku commented Nov 15, 2022

hitenkoku left a comment

Choose a reason for hiding this comment

hitenkoku left a comment

Choose a reason for hiding this comment

fukusuket commented Nov 14, 2022 •

edited

Loading

YamatoSecurity commented Nov 14, 2022 •

edited

Loading

fukusuket commented Nov 15, 2022 •

edited

Loading

fukusuket commented Nov 15, 2022 •

edited

Loading

fukusuket commented Nov 15, 2022 •

edited

Loading