Skip to content

Conversation

@UgnineSirdis
Copy link
Collaborator

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

We need to know if something goes wrong with audit events process - for example, if someone (occasionally or on their purpose) disabled audit logging. At the same time, it can be normal that for some cluster there are no records for a long time. With this setting we can setup alerts on audit and not to get false positive on them.

@github-actions
Copy link

github-actions bot commented Sep 7, 2025

🟢 2025-09-16 15:48:19 UTC The validation of the Pull Request description is successful.

@github-actions
Copy link

github-actions bot commented Sep 7, 2025

2025-09-07 10:36:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for d172c6f has started.
2025-09-07 10:37:11 UTC Artifacts will be uploaded here
2025-09-07 10:41:02 UTC ya make is running...
🔴 2025-09-07 11:06:42 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Sep 7, 2025

2025-09-07 10:37:20 UTC Pre-commit check linux-x86_64-release-asan for d172c6f has started.
2025-09-07 10:37:34 UTC Artifacts will be uploaded here
2025-09-07 10:41:28 UTC ya make is running...
🔴 2025-09-07 10:56:54 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Sep 7, 2025

2025-09-07 19:52:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4c695b0 has started.
2025-09-07 19:52:36 UTC Artifacts will be uploaded here
2025-09-07 19:56:36 UTC ya make is running...
🟡 2025-09-07 21:56:41 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40139 37381 0 1 2721 36

2025-09-07 22:00:06 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-07 22:13:56 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
389 (only retried tests) 365 0 0 0 24

🟢 2025-09-07 22:14:05 UTC Build successful.
🟢 2025-09-07 22:14:21 UTC ydbd size 2.3 GiB changed* by +86.1 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 6a4cad3 merge: 4c695b0 diff diff %
ydbd size 2 443 672 488 Bytes 2 443 760 656 Bytes +86.1 KiB +0.004%
ydbd stripped size 511 587 976 Bytes 511 593 096 Bytes +5.0 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 7, 2025

2025-09-07 19:52:42 UTC Pre-commit check linux-x86_64-release-asan for 4c695b0 has started.
2025-09-07 19:52:58 UTC Artifacts will be uploaded here
2025-09-07 19:56:48 UTC ya make is running...
🟡 2025-09-07 22:31:23 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17381 16877 0 144 331 29

🟢 2025-09-07 22:32:51 UTC Build successful.
🟡 2025-09-07 22:33:14 UTC ydbd size 4.0 GiB changed* by +150.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 6a4cad3 merge: 4c695b0 diff diff %
ydbd size 4 294 917 520 Bytes 4 295 071 576 Bytes +150.4 KiB +0.004%
ydbd stripped size 1 488 922 776 Bytes 1 488 944 024 Bytes +20.8 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 8, 2025

2025-09-08 16:45:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for ac3a89e has started.
2025-09-08 16:45:20 UTC Artifacts will be uploaded here
2025-09-08 16:49:24 UTC ya make is running...
🟡 2025-09-08 18:53:36 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40167 37412 0 1 2719 35

2025-09-08 18:57:09 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-08 19:09:16 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
266 (only retried tests) 244 0 0 0 22

🟢 2025-09-08 19:09:25 UTC Build successful.
🟡 2025-09-08 19:09:41 UTC ydbd size 2.3 GiB changed* by +100.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f679329 merge: ac3a89e diff diff %
ydbd size 2 444 247 840 Bytes 2 444 350 872 Bytes +100.6 KiB +0.004%
ydbd stripped size 511 665 800 Bytes 511 676 936 Bytes +10.9 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 8, 2025

2025-09-08 16:45:06 UTC Pre-commit check linux-x86_64-release-asan for ac3a89e has started.
2025-09-08 16:45:20 UTC Artifacts will be uploaded here
2025-09-08 16:49:15 UTC ya make is running...
🟡 2025-09-08 19:33:29 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17405 16999 0 112 265 29

🟢 2025-09-08 19:34:57 UTC Build successful.
🟡 2025-09-08 19:35:21 UTC ydbd size 4.0 GiB changed* by +169.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: f679329 merge: ac3a89e diff diff %
ydbd size 4 295 775 976 Bytes 4 295 949 064 Bytes +169.0 KiB +0.004%
ydbd stripped size 1 489 086 232 Bytes 1 489 115 480 Bytes +28.6 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

#define AUDIT_PART_COND(key, value, condition) \
do { \
if (condition && !value.empty()) { \
if (condition && !TStringBuf(value).empty()) { \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it nessasary?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It allows to specify in AUDIT_LOG macro literals.
Now we can write:

AUDIT_PART("status", "SUCCESS")

While before this "feature" we had to write:

AUDIT_PART("status", TString("SUCCESS"))

And if we call the macro in this way, the code will be less effective, because it creates TString twice: during the check and while inserting to the auditParts

optional TFileBackend FileBackend = 2;
optional TUnifiedAgentBackend UnifiedAgentBackend = 3;
repeated TLogClassConfig LogClassConfig = 4;
optional uint32 HeartbeatIntervalSeconds = 5; // Make heartbeat records to audit log every HeartbeatInterval seconds. 0 means that heartbeat is disabled
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better create new field in ELogClass (HeartBeat as example) and verify condition 'HeartbeatIntervalSeconds > 0'?

Copy link
Collaborator Author

@UgnineSirdis UgnineSirdis Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it will not be very convenient and will not work as expected. If I were a person that sets up the cluster, I would expect that specifying HeartbeatIntervalSeconds setting should run the heartbeat feature.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the one hand, you are right. But on the other hand, I don't like the loss of uniformity in the config. We have all types of logs listed in the ELogClass enum, except for one special one.

Of course, HeartBeat audit is special case, but other types can have special setting in future too

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed. It was decided that there will be new protobuf message with options for HeatBeat logs. Moreover, we agreed that adding new field in ELogClass is better.

@github-actions
Copy link

github-actions bot commented Sep 9, 2025

2025-09-09 15:27:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for d1e51cd has started.
2025-09-09 15:28:00 UTC Artifacts will be uploaded here
2025-09-09 15:32:31 UTC ya make is running...
🟡 2025-09-09 17:48:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40205 37440 0 5 2724 36

2025-09-09 17:51:42 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-09 18:12:38 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
397 (only retried tests) 371 0 0 3 23

🟢 2025-09-09 18:12:48 UTC Build successful.
🟡 2025-09-09 18:13:06 UTC ydbd size 2.3 GiB changed* by +126.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 3d8ed99 merge: d1e51cd diff diff %
ydbd size 2 444 544 712 Bytes 2 444 673 832 Bytes +126.1 KiB +0.005%
ydbd stripped size 511 737 544 Bytes 511 752 968 Bytes +15.1 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 9, 2025

2025-09-09 15:32:43 UTC Pre-commit check linux-x86_64-release-asan for d1e51cd has started.
2025-09-09 15:33:17 UTC Artifacts will be uploaded here
2025-09-09 15:37:47 UTC ya make is running...
🟡 2025-09-09 18:22:27 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17437 16980 0 154 271 32

🟢 2025-09-09 18:23:59 UTC Build successful.
🟡 2025-09-09 18:24:23 UTC ydbd size 4.0 GiB changed* by +218.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 8d0894e merge: d1e51cd diff diff %
ydbd size 4 296 467 304 Bytes 4 296 690 848 Bytes +218.3 KiB +0.005%
ydbd stripped size 1 489 328 536 Bytes 1 489 378 328 Bytes +48.6 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link
Collaborator

@flown4qqqq flown4qqqq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's worth doing the following:

  1. add a check that IntervalSeconds > 0;
  2. set the default IntervalSeconds value, for example, 30

@github-actions
Copy link

github-actions bot commented Sep 9, 2025

2025-09-09 18:27:14 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1c3703a has started.
2025-09-09 18:27:38 UTC Artifacts will be uploaded here
2025-09-09 18:32:02 UTC ya make is running...
🟡 2025-09-09 20:02:51 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40208 37431 0 4 2738 35

2025-09-09 20:36:40 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-09 20:50:02 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
439 (only retried tests) 410 0 0 3 26

🟢 2025-09-09 20:50:12 UTC Build successful.
🟡 2025-09-09 20:50:30 UTC ydbd size 2.3 GiB changed* by +120.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a3a095c merge: 1c3703a diff diff %
ydbd size 2 444 541 488 Bytes 2 444 665 336 Bytes +120.9 KiB +0.005%
ydbd stripped size 511 737 832 Bytes 511 752 008 Bytes +13.8 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 9, 2025

2025-09-09 18:27:59 UTC Pre-commit check linux-x86_64-release-asan for 1c3703a has started.
2025-09-09 18:28:31 UTC Artifacts will be uploaded here
2025-09-09 18:33:12 UTC ya make is running...
🟡 2025-09-09 20:39:16 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17439 16920 0 174 320 25

🟢 2025-09-09 20:40:44 UTC Build successful.
🟡 2025-09-09 20:41:08 UTC ydbd size 4.0 GiB changed* by +209.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: a3a095c merge: 1c3703a diff diff %
ydbd size 4 296 464 928 Bytes 4 296 678 976 Bytes +209.0 KiB +0.005%
ydbd stripped size 1 489 332 504 Bytes 1 489 378 584 Bytes +45.0 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 10, 2025

2025-09-10 13:13:40 UTC Pre-commit check linux-x86_64-release-asan for 63e29ca has started.
2025-09-10 13:14:15 UTC Artifacts will be uploaded here
2025-09-10 13:18:50 UTC ya make is running...
🟡 2025-09-10 15:52:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17450 16992 0 172 256 30

🟢 2025-09-10 15:53:46 UTC Build successful.
🔴 2025-09-10 15:54:09 UTC ydbd size 4.0 GiB changed* by +5.5 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 747eb09 merge: 63e29ca diff diff %
ydbd size 4 312 200 248 Bytes 4 317 940 496 Bytes +5.5 MiB +0.133%
ydbd stripped size 1 491 570 296 Bytes 1 493 078 872 Bytes +1.4 MiB +0.101%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 10, 2025

2025-09-10 13:14:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 63e29ca has started.
2025-09-10 13:14:34 UTC Artifacts will be uploaded here
2025-09-10 13:18:46 UTC ya make is running...
🟡 2025-09-10 15:29:07 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40239 37453 0 3 2740 43

2025-09-10 15:32:51 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-09-10 15:48:53 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
923 (only retried tests) 894 0 1 4 24

2025-09-10 15:49:05 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-09-10 15:59:23 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
244 (only retried tests) 221 0 1 0 22

🟢 2025-09-10 15:59:46 UTC Build successful.
🔴 2025-09-10 16:00:04 UTC ydbd size 2.3 GiB changed* by +3.2 MiB, which is >= 2.0 MiB vs main: Alert

ydbd size dash main: 747eb09 merge: 63e29ca diff diff %
ydbd size 2 465 086 184 Bytes 2 468 438 952 Bytes +3.2 MiB +0.136%
ydbd stripped size 515 205 672 Bytes 515 753 704 Bytes +535.2 KiB +0.106%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@flown4qqqq flown4qqqq self-requested a review September 10, 2025 13:16
@UgnineSirdis UgnineSirdis added the rebase-and-check Rebase PR with the current base branch and check label Sep 10, 2025
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Sep 10, 2025
@github-actions
Copy link

github-actions bot commented Sep 10, 2025

2025-09-10 21:12:19 UTC Pre-commit check linux-x86_64-release-asan for c9aba69 has started.
2025-09-10 21:12:55 UTC Artifacts will be uploaded here
2025-09-10 21:17:39 UTC ya make is running...
🟡 2025-09-10 23:59:14 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17409 16959 0 123 298 29

🟢 2025-09-11 00:00:42 UTC Build successful.
🟡 2025-09-11 00:01:07 UTC ydbd size 4.0 GiB changed* by +207.2 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d938a53 merge: c9aba69 diff diff %
ydbd size 4 318 494 664 Bytes 4 318 706 808 Bytes +207.2 KiB +0.005%
ydbd stripped size 1 493 258 616 Bytes 1 493 301 272 Bytes +41.7 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Sep 10, 2025

2025-09-10 21:15:09 UTC Pre-commit check linux-x86_64-relwithdebinfo for c9aba69 has started.
2025-09-10 21:15:22 UTC Artifacts will be uploaded here
2025-09-10 21:19:15 UTC ya make is running...
🟡 2025-09-10 23:28:32 UTC Some tests failed, follow the links below. Going to retry failed tests...

Details

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40200 37440 0 1 2729 30

2025-09-10 23:32:06 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-09-10 23:58:07 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
281 (only retried tests) 260 0 0 0 21

🟢 2025-09-10 23:58:15 UTC Build successful.
🟡 2025-09-10 23:58:31 UTC ydbd size 2.3 GiB changed* by +121.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d938a53 merge: c9aba69 diff diff %
ydbd size 2 468 779 184 Bytes 2 468 903 544 Bytes +121.4 KiB +0.005%
ydbd stripped size 515 821 192 Bytes 515 835 336 Bytes +13.8 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@UgnineSirdis UgnineSirdis merged commit 884b117 into ydb-platform:main Sep 11, 2025
16 of 17 checks passed
@UgnineSirdis UgnineSirdis deleted the heartbeat-audit branch September 11, 2025 05:01
UgnineSirdis added a commit to UgnineSirdis/ydb that referenced this pull request Sep 11, 2025
UgnineSirdis added a commit to UgnineSirdis/ydb that referenced this pull request Sep 11, 2025
UgnineSirdis added a commit that referenced this pull request Sep 15, 2025
UgnineSirdis added a commit that referenced this pull request Sep 16, 2025
@alexv-smirnov alexv-smirnov changed the title Heartbeat setting for audit logging EXT-1486 Heartbeat setting for audit logging Sep 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants