Downsampling performance analysis and improvement

[salvatore-campagna]

Description

We would like to measure performances of downsampling operations using Rally. For this purpose we need to include a new challenge to the existing tsdb Rally track. The new challenge will measure latency for a limited set of downsampling operations using different values for the fixed_interval parameter. As part of the analysis we need to collect JFR recordings and flame graphs so that we can spot areas of the code we can improve.

Right now the tsdb track uses a dataset including more than 116M documents for a total JSON file size of more than 120 GB, which results in a 32.5 GB index. The plan is to measure downsampling latency with a single thread implementation, a single node Elasticsearch cluster and a single shard.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[salvatore-campagna]

Attaching flame graph collected while running two downsample operations and JFR recording collected for the whole challenge:

• downsample-1h: using fixed_interval: 1h
• downsample-1d:using fixed_interval: 1d

downsample-flamegraph-d6e36b58-52ff-495f-b64d-a765c368f7ad.html.zip

profile-d6e36b58-52ff-495f-b64d-a765c368f7ad.jfr.zip

[salvatore-campagna]

As a result of analyzing both the JFR recording and the flame graph I see two improvements which are worth working on:

1. making sure we don't decode keyword fields (BytesRef to UTF) and just use the BytesRef (see RollupShardIndexer)
2. making sure we get rid of hash map access and iteration while collecting fields (see RollupShardIndexer)

Other than that, time is spent reading doc values which is expected.

NOTE: after merging PR #90088 we see consistent and significant improvements in latency. Latest tests show that both the downsampling operations (1h and 1d fixed interval) take around 30 minutes to complete.

[salvatore-campagna]

Downsampling the same source index using 1m fixed interval took about 1.5 hours producing an index with about 7M documents. Attaching JFR recording and flames graph.

profile-9c51f1f9-816b-4fa7-8739-aa140dd6d3e6.jfr.zip

downsample-flamegraph-9c51f1f9-816b-4fa7-8739-aa140dd6d3e6.html.zip

[salvatore-campagna]

Closing after the following PR has been merged #92494