[Lens] Time offset using filters

[flash1293]

Rough POC

Create a chart in Lens with a count metric with label "shifted" - it will get shifted by one day

Mechanism

This works by doing the "base request" for the actually selected time range first, then serializing the bucket hierarchy into a flat list of filters (which time shifts applied) and doing a second request to fetch the shifted time:

Configuration:

First request:

{
    "index": "kibana_sample_data_logs",
    "body": {
        "aggs": {
            "2bc261dc-92c1-48f9-b117-af885fb6539c": {
                "date_histogram": {
                    "field": "@timestamp",
                    "calendar_interval": "1d",
                    "time_zone": "Europe/Berlin"
                },
                "aggs": {
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "terms": {
                            "field": "extension.keyword",
                            "order": {
                                "_count": "desc"
                            },
                            "size": 2
                        }
                    }
                }
            }
        },
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "match_all": {}
                    },
                    {
                        "range": {
                            "@timestamp": {
                                "gte": "2021-03-25T23:00:00.000Z",
                                "lte": "2021-03-27T23:00:00.000Z",
                                "format": "strict_date_optional_time"
                            }
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    },
}

First response:

{
    "aggregations": {
        "2bc261dc-92c1-48f9-b117-af885fb6539c": {
            "buckets": [
                {
                    "key_as_string": "2021-03-26T00:00:00.000+01:00",
                    "key": 1616713200000,
                    "doc_count": 230,
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 113,
                        "buckets": [
                            {
                                "key": "",
                                "doc_count": 79
                            },
                            {
                                "key": "css",
                                "doc_count": 38
                            }
                        ]
                    }
                },
                {
                    "key_as_string": "2021-03-27T00:00:00.000+01:00",
                    "key": 1616799600000,
                    "doc_count": 230,
                    "fa2b4f03-c718-4131-8d4b-8fe69e779abe": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 102,
                        "buckets": [
                            {
                                "key": "",
                                "doc_count": 88
                            },
                            {
                                "key": "gz",
                                "doc_count": 40
                            }
                        ]
                    }
                }
            ]
        }
    }
}

Time-shift request (additional shifts would happen in parallel)

{
    "index": "kibana_sample_data_logs",
    "body": {
        "aggs": {
            "rootSerialization": {
                "filters": {
                    "filters": {
                        "serialized-row-0": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-24T23:00:00.000Z",
                                                            "lte": "2021-03-25T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": ""
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-1": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-24T23:00:00.000Z",
                                                            "lte": "2021-03-25T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": "css"
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-2": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-25T23:00:00.000Z",
                                                            "lte": "2021-03-26T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": ""
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        },
                        "serialized-row-3": {
                            "bool": {
                                "must": [],
                                "filter": [
                                    {
                                        "bool": {
                                            "filter": [
                                                {
                                                    "range": {
                                                        "@timestamp": {
                                                            "gte": "2021-03-25T23:00:00.000Z",
                                                            "lte": "2021-03-26T23:00:00.000Z"
                                                        }
                                                    }
                                                },
                                                {
                                                    "match_phrase": {
                                                        "extension.keyword": "gz"
                                                    }
                                                }
                                            ]
                                        }
                                    }
                                ],
                                "should": [],
                                "must_not": []
                            }
                        }
                    }
                }
            }
        },
        "query": {
            "bool": {
                "must": [],
                "filter": [
                    {
                        "match_all": {}
                    },
                    {
                        "range": {
                            "@timestamp": {
                                "gte": "2021-03-24T23:00:00.000Z",
                                "lte": "2021-03-26T23:00:00.000Z",
                                "format": "strict_date_optional_time"
                            }
                        }
                    }
                ],
                "should": [],
                "must_not": []
            }
        }
    },
}

Time shift response

{
    "aggregations": {
        "rootSerialization": {
            "buckets": {
                "serialized-row-0": {
                    "doc_count": 84
                },
                "serialized-row-1": {
                    "doc_count": 31
                },
                "serialized-row-2": {
                    "doc_count": 79
                },
                "serialized-row-3": {
                    "doc_count": 38
                }
            }
        }
    }
}

Limitations

• Other bucket doesn't work because of agg config internals (easily fixable)
• Only date histogram and terms implemented so far (easy to extend to histogram, filters, ...)
• Parent pipeline aggs won't work at all (because the are only allowed on date histograms / histograms
  • No way around that
  • Wouldn't be a problem for Lens because we don't expose them anyway
• It's not possible to sort by a shifted metric
  • this is partially fixable by making the shift we want to sort by the base request
  • It's not easily fixable if different sorts are operating on different shifts
  • We could also leave it up to the user to change the time range and use a different shift, so the metric they want to sort by becomes the one of the actual time range
• This requires two requests in series (which means it doesn't work well with send-to-background, something we might be able to fix Kibana side)
• If there's an "other" bucket, it requires three requests in series

[skip ci]

[flash1293]

Discussed with @ppisljar how this could get implemented:

• Add time shift property to all metrics
• Add time shift property to createFilter (not used for everything except for date histogram and probably date range)
• Introduce new field timeFilter - it's merged into the actual request like filter, but is shadowing existing timeFilter fields of parent search sources
  • This is used to set the time range, so we can override it
  • Just takes fields array and time range, builds the filter internally
• Move other bucket stuff from search source into agg configs (global onPostFlightRequest hook)
• Add time shift request building and response parsing into this hook as well
  • Will walk the response of the base request like other bucket logic, building large filters array
  • Sends out the filters request
  • Will walk the response same way as other bucket, putting the shifted metrics into the first response so tabify will catch them with the default mapping logic and constructure the final table as a last step

[flash1293]

See #98781