Support for overlapping "buckets" in the date histogram

[tobiasstadler]

I would like create overlapping "buckets" in the date histogram aggregation(or a new aggregation). E.g I would like to create buckets for every hour of the last 12 hours, but each bucket should also contain the documents 3 hours prior the bucket.

bucket 1 should contain everything between now and now-4h,
bucket 2 should contain everything between now-1h and now-5h,
bucket 3 should contain everything between now-2h and now-6h,
...

It should then be possible to calculate a metric for each bucket. E.g. I should be able to calculate the average of the last 3 hours for each hour.

This is similar to what one can do with Prometheus range queries (https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries)

[tobiasstadler]

I know I can use date_range aggregation by manual specifying the ranges, but I was hoping for more automatic range creation.

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[nik9000]

I've labeled this for triage by aggs folks.

I know I can use date_range aggregation by manual specifying the ranges, but I was hoping for more automatic range creation.

For what it is worth, come 7.11 Elasticsearch internally rewrites date_histogram into a date_range aggregation in lots of cases (#63643). So, at least in 7.11, your work around isn't going to execute any slower.

[bradyasana]

Going to pile on with a similar feature request: it'd be great if you could accumulate the documents in the buckets of a date_histogram aggregation, such that bucket 2 would contain all documents from bucket 1, etc. While you can accumulate inner metrics across date_histogram buckets, you can't accumulate the documents themselves.

This would be particularly beneficial if you could access the bucket keys in the sub-aggregations (#56392).

[imotov]

We are going to address that as a part of #74660 (sliding window aggregation). We can discuss if we want to make it applicable to non-TSDB indices or not.

[tobiasstadler]

My use case only involves time series data, so I am fine with TSDB indices (only).

[tobiasstadler]

@imotov Is there any timeline when sliding window aggregation will be available?

[imotov]

@tobiasstadler we are actively working on it and there is an internal timeline for it. Unfortunately, I cannot share it externally. I can only suggest watching the public issue #74660 to see how this work is progressing.

[tobiasstadler]

I am looking forward for it

[wchaparro]

Based on our recent internal discussion on this, we plan on introducing this as a sliding window aggregation focused on TSDB indicies.

We also had a discussion to see how we might introduce this as a sliding window aggregation for non-TSDB indicies as a general use case ( i.e. show me logs where we had 404 errors, over a 3 hour time window, and then show it to me for the next hour). We would introduce this as a separate distinct aggregation, supporting a fixed time interval.

[wchaparro]

Closing per prior comment.