max_agents setting in fleet unclear

[pjbertels]

Kibana version:
8.3.0+

Elasticsearch version:

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
The Fleet integration has a field max_agents and some other advanced settings and they are documented here(https://www.elastic.co/guide/en/fleet/current/fleet-server-scalability.html#recommend-settings-scaling-agents). This setting(max_agents) may or may not affect a bunch of other advanced settings related to the scalability of the Fleet server. In ESS we have not been able to find the logs that Sean thought would give an indication to how these settings are set for a given cluster. It is possible in ESS that the decision was made not to change the settings at all in favor of other scaling strategies.
Changing the setting to 12,000 and doing a test with enrolling 10,000 Horde drones(enrolling at 20 drones /per sec ) does not seem to make a noticeable difference.

Steps to reproduce:
1.
2.
3.

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[joshdover]

We need to provide some documentation on this setting, including what it sets, how the default is calculated (or not), etc. We should also consider automatically configuring this value in Cloud if possible (either auto-calculated at runtime or set via kibana.yml)

[nimarezainia]

@joshdover I do have a draft to mend documentation. However last week we realized that these defaults may not be set correctly. i don't know how to confirm if these variables are set or set to the correct value.

@pierrehilbert We need someone to take this topic up urgently. My suggestion would be to have the yaml box show the configuration (it used to have that) when the max agents value is set. thanks.

[pjbertels]

I noticed that the 8.3 documentation(current) and the master documentation don't refer to max_agents but refer to max_connections in step 3.
https://www.elastic.co/guide/en/fleet/current/fleet-server-scalability.html
https://www.elastic.co/guide/en/fleet/master/fleet-server-scalability.html
but in the 8.3.0-SNAPSHOTS we have an indication to effect that max_connections is being deprecated.

And the plot thickens ...

[joshdover]

Some action items we identified in our sync today:

• Need to clarify when the “max agent” field is not set, what is the behavior.
• Add documentation about this behavior
• Add a link to the “max agent” section in the integration to our documentation to explain how this works.
• Add a log to Fleet Server that would dump all the information that is related to the server.runtime configuration. Log all of fleet server configuration at startup #1626

[joshdover]

Close in favor of #1860