[ML] Report the "actual" memory usage of the autodetect process #2846

edsavage · 2025-04-04T02:41:11Z

Determine the actual memory usage of the autodetect process as reported by the OS, e.g. on Linux this would be the value of the maximum resident set size returned by a call to getrusage.

Add this value to the model size stats record returned to the ES Java process so it can be included in the job counts tab for anomaly detection jobs.

Relates elastic/elasticsearch#131981

Determine the actual memory usgae of the autodetect process as reported by the OS, e.g. on Linux this mould be the value of the maximum resident set size returned by a call to `getrusage`. Add this value to the model size stats record returned to the ES Java process so it can be included in the `job counts` tab for anomaly detection jobs.

valeriy42

Hi Ed. I did the first pass.

We should discuss the naming of the new field. While "actual" conveys the intention of the value, it is confusing to the user.

Also, does maximum resident set size actually correspond to the actual current memory usage or is it the historical peak process memory usage?

valeriy42 · 2025-04-08T14:04:37Z

include/model/CResourceMonitor.h

@@ -180,6 +181,8 @@ class MODEL_EXPORT CResourceMonitor {
    //! Returns the sum of used memory plus any extra memory
    std::size_t totalMemory() const;

+    std::size_t actualMemoryUsage() const;


I think we can come up with something better than actualMemoryUsage. Maybe: systemMemoryUsage?

bin/autodetect/Main.cc

edsavage · 2025-04-08T22:30:32Z

Also, does maximum resident set size actually correspond to the actual current memory usage or is it the historical peak process memory usage?

The resident set size (RSS) represents the process's current RAM usage (so not counting pages that have been swapped out etc.), and the max RSS is the high water mark of that value. I think that reporting both would be useful for our purposes.

* ActualMemory -> SystemMemory * Report current resident set size as well as max

bin/autodetect/Main.cc

include/model/CResourceMonitor.h

valeriy42 · 2025-04-10T12:01:24Z

include/model/ModelTypes.h

+    E_AssignmentBasisSystemMemoryBytes = 4, //!< Use the current system memory size
+    E_AssignmentBasisMaxSystemMemoryBytes = 5 //!< Use the highest ever system memory size


I don't think we need this assignment basis reasons (at least so far)

valeriy42 · 2025-04-10T12:02:26Z

bin/autodetect/Main.cc

+        ml::counter_t::E_TSADResidentSetSize,
+        ml::counter_t::E_TSADMaxResidentSetSize};


Should we call these counters SystemMemoryUsage for consistency?

valeriy42 · 2025-04-10T12:05:21Z

lib/model/ModelTypes.cc

+    case E_AssignmentBasisSystemMemoryBytes:
+        return "system_memory_bytes";
+    case E_AssignmentBasisMaxSystemMemoryBytes:
+        return "max_system_memory_bytes";


Are those necessary?

Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>

…mem_usage # Conflicts: # bin/autodetect/Main.cc # include/model/CResourceMonitor.h

elastic-sonarqube · 2025-04-11T03:31:21Z

Quality Gate failed

Failed conditions
274 New issues
23.0% Duplication on New Code (required ≤ 3%)
164 New Major Issues (required ≤ 0)
10 New Critical Issues (required ≤ 0)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

prodsecmachine · 2025-05-25T22:01:30Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

* Address failing unit tests * More accurate, meaningful description of new program counters

valeriy42

Looks good. I think the last piece that is missing now is that we use the system memory usage when in CResourceMemory when calculating if allocations are allowed and that we report it back to Java as "model memory usage" and "peak model memory usage" instead of the estimated values on Linux.

… set size) for the "model memory usage" and "peak model memory usage" fields reported to Java.

…nto ad_real_mem_usage

On Linux both adjusted usage and adjusted peak usage are set to system memory usage (max resident set size) These are the values reported back to the Java process, they are not used for any other purpose.

…ml-cpp into ad_real_mem_usage" This reverts commit 13fa5cf, reversing changes made to e8c00e7.

The ml-cpp PR elastic/ml-cpp#2846 introduces changes to how memory values are calculated and reported for Linux platforms. This PR adjusts test case values accordingly.

valeriy42

LGTM

valeriy42 · 2025-07-29T13:54:19Z

docs/CHANGELOG.asciidoc

@@ -33,6 +33,7 @@
 === Enhancements

 * Track memory used in the hierarchical results normalizer. (See {ml-pull}2831[#2831].)
+* Report the actual memory usage of the autodetect process. (See {ml-pull}2846[#2846])


You need to move it to 9.2.0 now

The ml-cpp PR elastic/ml-cpp#2846 introduces changes to how memory values are calculated and reported for Linux platforms. This PR adjusts test case values accordingly.

elastic-sonarqube · 2025-07-31T02:24:36Z

Quality Gate passed

Issues
0 New issues
1 Fixed issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

…#131981) The ml-cpp PR elastic/ml-cpp#2846 introduces changes to how memory values are calculated and reported for Linux platforms. This PR adjusts test case values accordingly.

edsavage added >enhancement :ml v9.1.0 labels Apr 4, 2025

edsavage added 4 commits April 4, 2025 15:53

Update changelog

cb957ca

Formatting

8f73f02

Appease SonarQube

d3a39ae

Tweak unit test for platform portability

e2d1bf5

edsavage marked this pull request as ready for review April 8, 2025 02:03

edsavage mentioned this pull request Apr 8, 2025

[ML] Accommodate changes to calculation of "model bytes" and "peak mode bytes" values.. elastic/elasticsearch#126256

Closed

edsavage requested a review from valeriy42 April 8, 2025 02:05

valeriy42 reviewed Apr 8, 2025

View reviewed changes

Attend to review comments

1a9a99a

* ActualMemory -> SystemMemory * Report current resident set size as well as max

valeriy42 reviewed Apr 10, 2025

View reviewed changes

edsavage and others added 6 commits April 11, 2025 11:08

Update bin/autodetect/Main.cc

5ae22cb

Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>

Update include/model/CResourceMonitor.h

fe6f1fa

Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>

Update include/model/CResourceMonitor.h

582430e

Co-authored-by: Valeriy Khakhutskyy <1292899+valeriy42@users.noreply.github.com>

Attend to review comments

9476ede

Merge remote-tracking branch 'origin/ad_real_mem_usage' into ad_real_…

e29d8f7

…mem_usage # Conflicts: # bin/autodetect/Main.cc # include/model/CResourceMonitor.h

Formatting

475fef1

Merge branch 'main' of github.com:elastic/ml-cpp into ad_real_mem_usage

fc8888c

edsavage added 2 commits May 27, 2025 15:50

Attend to failing unit tests (hopefully)

6945d3f

* Address failing unit tests * More accurate, meaningful description of new program counters

Formatting.. grr

3b69b72

valeriy42 reviewed Jun 2, 2025

View reviewed changes

edsavage added 3 commits June 4, 2025 16:24

On Linux only, use the value of the system memory usage (max resident…

1bafa8c

… set size) for the "model memory usage" and "peak model memory usage" fields reported to Java.

Fix copyright headers

efc311b

Nits in test code

6be6395

edsavage added 18 commits July 4, 2025 12:35

Tidy up of scripts

62dce12

Further tidy up of scripts

55875c0

Tweaks for linux aarch64

9c67353

Tweak for linux aarch64 seccomp test

d265102

Typo

2cf5752

Better isolation of tests

3ace6f4

Formatting

4014a72

Formatting

e3d2f8a

Reworking test case

7b39899

Small tidy up

58fafb4

Minor tweaks to test runner script

033adef

typo

fe71d2a

Merge branch 'main' of github.com:elastic/ml-cpp into ad_real_mem_usage

e8c00e7

Merge branch 'run_tests_individually' of github.com:edsavage/ml-cpp i…

13fa5cf

…nto ad_real_mem_usage

Set both adjusted usage and adjusted peak usage to system memory.

bbe25b1

On Linux both adjusted usage and adjusted peak usage are set to system memory usage (max resident set size) These are the values reported back to the Java process, they are not used for any other purpose.

Formatting

3ab09ba

Revert "Merge branch 'run_tests_individually' of github.com:edsavage/…

39b4f3b

…ml-cpp into ad_real_mem_usage" This reverts commit 13fa5cf, reversing changes made to e8c00e7.

Remove unneded sleep in test case

a536d46

edsavage mentioned this pull request Jul 28, 2025

[ML] Adjust test case values due to changed memory reporting elastic/elasticsearch#131981

Merged

valeriy42 added v9.2.0 and removed v9.1.0 labels Jul 28, 2025

valeriy42 approved these changes Jul 29, 2025

View reviewed changes

edsavage added 2 commits July 31, 2025 12:22

Update changelog

0a6e2b1

Merge branch 'main' into ad_real_mem_usage

befb881

edsavage merged commit e1e93b5 into elastic:main Jul 31, 2025
16 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ML] Report the "actual" memory usage of the autodetect process #2846

[ML] Report the "actual" memory usage of the autodetect process #2846

Uh oh!

edsavage commented Apr 4, 2025 •

edited

Loading

Uh oh!

valeriy42 left a comment

Uh oh!

valeriy42 Apr 8, 2025

Uh oh!

Uh oh!

edsavage commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valeriy42 Apr 10, 2025

Uh oh!

valeriy42 Apr 10, 2025

Uh oh!

valeriy42 Apr 10, 2025

Uh oh!

elastic-sonarqube bot commented Apr 11, 2025

Uh oh!

prodsecmachine commented May 25, 2025 •

edited

Loading

Uh oh!

valeriy42 left a comment

Uh oh!

valeriy42 left a comment

Uh oh!

valeriy42 Jul 29, 2025

Uh oh!

elastic-sonarqube bot commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

		E_AssignmentBasisSystemMemoryBytes = 4, //!< Use the current system memory size
		E_AssignmentBasisMaxSystemMemoryBytes = 5 //!< Use the highest ever system memory size

		ml::counter_t::E_TSADResidentSetSize,
		ml::counter_t::E_TSADMaxResidentSetSize};

[ML] Report the "actual" memory usage of the autodetect process #2846

[ML] Report the "actual" memory usage of the autodetect process #2846

Uh oh!

Conversation

edsavage commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edsavage commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Apr 11, 2025

Quality Gate failed

Uh oh!

prodsecmachine commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

valeriy42 Jul 29, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Jul 31, 2025

Quality Gate passed

Uh oh!

Uh oh!

Uh oh!

edsavage commented Apr 4, 2025 •

edited

Loading

prodsecmachine commented May 25, 2025 •

edited

Loading