Replace ingest pipeline woraround in tsdb track

[martijnvg]

The workaround is replaced by a workaround that uses fixed start / end time range in the template.

This way there is no need to update document's @timestamp value before indexing with an ingest pipeline.
The @timestamp field values as they are can directly be used. This is beneficial because this doesn't
add noice of an ingest pipeline to the challenge. Other tsdb challenges don't use a pipeline.

This should allow for running with ingest_mode of value data_stream by default. Also in rally nightlies.

[pquentin]

Thanks!

Can you please apply your index-template.json changes to index.json too?

Also, is this possible thanks to improvements in Elasticsearch? What is the latest Elasticsearch branch where this will work?

[pquentin]

Removing those lines is not necessary since #375.

And we have #377 to rely on Elasticsearch defaults. That said, we want to keep this to allow testing synthetic source with standard indices, which is tested in our nightly benchmarks.

[pquentin]

What is the point of this change?

[martijnvg]

This is a mistake. I will revert this. This search is ran when running the downsample challenge and in that case the tsdb-1d index exists.

[martijnvg]

This is required otherwise when running with ingest_mode data_stream this search returns no results.

[martijnvg]

Can you please apply your index-template.json changes to index.json too?

Done.

Also, is this possible thanks to improvements in Elasticsearch? What is the latest Elasticsearch branch where this will work?

Yes, tsdb indices use synthetic source by default. Starting from 8.7.0.

[pquentin]

Also, is this possible thanks to improvements in Elasticsearch? What is the latest Elasticsearch branch where this will work?

Yes, tsdb indices use synthetic source by default. Starting from 8.7.0.

To be clear, "this" was referring to the fact that we can now remove the ingest pipeline. time_series start_time/end_time were introduced in 8.4 I think? I was wondering if something more recent allowed removing that pipeline. I'm assuming it's not thanks to the synthetic source default since it was possible to configure synthetic source with TSDB before 8.7.

Anyway, that's only my curiosity. The changes look great now, thank you!

[martijnvg]

I'm assuming it's not thanks to the synthetic source default since it was possible to configure synthetic source with TSDB before 8.7.

Yes, this is why it can't be back ported beyond 8.7