[Search] Load test search sessions vs no search sessions

[Dosant]

Currently, search sessions in Kibana are implemented in a way that they add some performance overhead even if the search session is not saved. We are working on improving that aiming to tend search session impact to 0 when they are not saved.
But as of now, these are some load testing results with search sessions on and off:

Conditions

• Using default configuration and default scenarios from https://github.com/elastic/kibana-load-testing
• Running against a cloud instance: https://github.com/elastic/kibana-load-testing/blob/main/docs/KIBANA_CI.md#cloud-testing

Results

Focusing just on a dashboard scenario

Single Kibana, 2 es nodes (default cloud config)

Scenario	Total Requests	Events/second	Min	50th pct	75th pct	95pct	99th pct	Max	Mean	Std dev
Search sessions enabled	23120	68.2	21	3968	6031	9568	12414	18137	4047	3004
Search sessions disabled	23477	70.7	22	3651	5574	8389	11726	15007	3843	2725

fill gatling report multiple es nodes.zip

Then I reduced the number of elasticsearch nodes to a single node and the difference in the same testing scenario even more significant:

Single Kibana, singles es node

Scenario	Total Requests	Events/second	Min	50th pct	75th pct	95pct	99th pct	Max	Mean	Std dev
Search sessions enabled	19227	44.9	23	7351	11527	21453	27288	40196	8003	6171
Search sessions disabled	21165	54.3	21	5658	8915	16068	19690	23451	6055	4652

full gatling report a single es nodes.zip

[rayafratkina]

cc @crob611 @jb1b84 @sixstringcode
I wonder if fixing time interval to round per #94280 will end up hitting the cached search session and improving the performance?

[Dosant]

I wonder if fixing time interval to round per #94280 will end up hitting the cached search session and improving the performance?

This would help with starting fewer new searches in some Kibana usage scenarios, so I agree it would help with overall cluster load in some cases.
But with search sessions, we know that they are implemented in a way that when they are enabled, but not actually used, they still add additional overhead to Kibana and Elasticsearch (what this comparison shows). We are working on an RFC to change it and work towards the minimal impact on performance until the search session is saved

[rayafratkina]

This would help with starting fewer new searches in some Kibana usage scenarios, so I agree it would help with overall cluster load in some cases.

I think this is the scenario we should be optimizing for at this point - most real-world usage of solutions means dashboards are created by a subset of users and/or are rarely modified but many users need to view them and do so often. Anything we can do to help a dashboard "hit cache" by default is important

[Dosant]

I think this is the scenario we should be optimizing for at this point - most real-world usage of solutions means dashboards are created by a subset of users and/or are rarely modified but many users need to view them and do so often. Anything we can do to help a dashboard "hit cache" by default is important

For general performance, I agree that elasticsearch "cache hit" is definitely something we should optimize for (#94280)

But I think it is unrelated to the problem that current search session implementation adds additional load for every search run even until the session is saved by the user.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)