You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
User specifies a memory limit up front with no guidance as to what is sensible/realistic
They start the analysis
Analysis chugs away for a while reindexing the source index into the destination index
C++ process is started and checks whether the memory limit is sufficient to do the analysis
If it's not then the process exits, reporting how much memory was required in the error message
Two possible ways to solve it are:
We duplicate that logic that calculates the memory requirement from the C++ code into the Java and UI code
We add a mode of operation to the C++ process where you just supply the spec and instead of actually doing the analysis it just tells you:
i. What you'd have to set the memory limit to to do the analysis entirely in RAM
ii. What the minimum memory limit is that will enable the analysis to run at all (using disk)
Although option 2 is a lot of work, the memory calculations done by the C++ are now so complex that it is impractical to duplicate them. Therefore we should implement option 2. The work will be something along the lines of:
Call the endpoint as part of the configuration process in the UI so that the UI can present a sensible default model_memory_limit, advise on sensible upper and lower bounds and validate any value the user chooses themselves [ML] Improve UX regarding df analytics model memory limit kibana#43740
The current workflow for memory limits is:
Two possible ways to solve it are:
i. What you'd have to set the memory limit to to do the analysis entirely in RAM
ii. What the minimum memory limit is that will enable the analysis to run at all (using disk)
Although option 2 is a lot of work, the memory calculations done by the C++ are now so complex that it is impractical to duplicate them. Therefore we should implement option 2. The work will be something along the lines of:
Introduce memory usage estimation mode in data_frame_analyzer ml-cpp#584
Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188
model_memory_limit
is so low that the C++ process will fail immediately when runCall the new _estimate_memory_usage API endpoint on df analytics _start #45536
model_memory_limit
, advise on sensible upper and lower bounds and validate any value the user chooses themselves[ML] Improve UX regarding df analytics model memory limit kibana#43740
HLRC for memory usage estimation API #45531
Implement ml/data_frame/analytics/_estimate_memory_usage API endpoint #45188 (API reference), Add docs for HLRC for Estimate memory usage API #45538 (HLRC)
The text was updated successfully, but these errors were encountered: