Add "Filter Ratio" metric aggregation

[fbaligand]

Describe the feature:
In TSVB visualization, there is a very useful aggregation named "Filter Ratio", where we can get the ratio between a numerator filter and a denominator filter.
This would be great that such an aggregation is added to metric aggregations available in all other visualizations.

Describe a specific use case for the feature:
In a line chart, it allows to display the evolution of a percentage over the time.
In a goal/gauge chart, it lets to see the percentage of error requests among all requests, and to fix an acceptable goal percentage.

[timroes]

This would be solveable with bucket scripts (#4707). Nevertheless I wouldn't count this a duplicate, since it would make sense to maybe have an "easier" UI to calculate the ratio between two filters. Maybe we could still create a good bucket script UI that would also be very easy to use for "Filter Ratio" kind of aggregations. Under the hood this should imho still use the bucket script aggregation to calculate those ratio.

[fbaligand]

The 2 options you suggest would be great solutions !
Then, as you said, a "Filter Ratio" aggregation would be more simple to use and to implement, than a bucket scripts aggregation.
And maybe, using the existing "Filter Ratio" code in TSVB, it could be quick to implement.

[timroes]

@fbaligand It unfortunately isn't that easy, since TSVB (which is an experiment) bypasses a lot of central querying infrastructure (Courier, index patterns, etc.), so there is basically up to no shared code between TSVB and other visualizations. They already use completely different means to query ES (serverside vs browserside).

To give you a very tentative plan: We are currently working on using the Canvas rendering pipeline for all Kibana visualizations (#19813). Once this is done (targeted for 7.0) we will start implementing more of the outstanding aggregations (bucket script, scripted metrics, missing pipeline aggs, etc.) into the pipeline functions.

Also we are working on creating a new editor (#17076), which will replace the current editor and be designed from scratch. We also won't add any complex UI like bucket scripts anymore to the old editor, since we rather want to have that designed in a proper usable way (e.g. that we can have bucket script and "filter ratio" within one properly function UI).

So it's very unlikely any new aggregations will still happen during the 6.x roadmap before the new pipeline is finished and the new editor has started.

[fbaligand]

Thanks for the « tentative » roadmap.
I love the planned future editor !
It looks promising !

As I understand, this feature is not planned for tomorrow :)

[JackRyanson]

@timroes my understanding is that canvas is xpack, does this mean the entire kibana is not going to be apache any longer?

[timroes]

@JackRyanson No, don't worry about that. We will actually move the parser/interpreter of the pipeline expressions into the OSS part of Kibana. So Canvas as the UI and layouting engine will still stay X-pack, but the actual interpreter for the expressions (that at that point Canvas and Kibana will then be using) will be OSS Apache 2 licensed.

[JackRyanson]

Thanks for the clarification Tim! sounds like a big job. but should be worth it.

[JackRyanson]

Ah another thing i guess timelion will be discontinued too? havent seen much movement there for a while..

[timroes]

A lot of those features are actually currently kind of "stalled" as you might describe it due to capacity issues. Refactoring to the pipeline, building a new editor (and what wasn't mentioned in this thread right now, building a new chart implementation), takes quiet some time. So we only sparsely find time to add new features to the existing visualizations, especially since we know that after we switched e.g. to the new pipeline a lot of features will become way easier to implement, why we also on purpose rather postpone them for now (one example here would be the said bucket script aggregation, which we don't want to add in the current old infrastructure anymore).

But to give a short outlook for timelion (though this might be the same tentative as everything, since we have not put that on our roadmap yet): We want to remove (most likely somewhere over the 7.x versions) the rendering capabilities and timelion as an standalone application, but we don't want to remove timelion for it's data gathering and calculating part. But that will in the end just be one data providing function inside the pipeline, that you then could use to render into any chart (not just line charts as nowadays), e.g. (simplified example)

timelion exp=".es(q=foo:bar, offset=-1d).divide(2)" | pie_chart colors=['orange', 'hotpink']

That way you can use everything that is not styling related in timelion and use it as input for any chart you like. That gives a couple of advantages:

• Way more flexibility in using timelion with different chart types.
• Get rid of the separate chart implementation in timelion.

The new chart implementation should also offer some of the until now missing features the timelion chart had (like synced crosshairs, when hovering over a chart).

To remove the timelion application itself, we try to look at what it currently benefits and bring that in for all charts in Kibana. We currently see the huge advantage in timelion is the ability to quickly edit multiple charts besides each other. But maybe we could instead give the possibility on a dashboard to edit any chart in place? Wouldn't that be way better for all visualizations in the end :-)

[timroes]

Until then most likely we will rather have only minor changes in the existing infrastructure, but e.g. Kibana 6.4 will have a new feature in timelion, to use the whole chart time range as an offset: #19709

[JackRyanson]

@timroes yes the ability to change settings of the visualizations (each and every) from dashboard view vs an "edit" view is key for analysts.

if you're doing such design you should make the foolscreen option super fast e.g. getting a visualization fullscreens should not reload data, and allow anything to be change in dashboard mode (either fullscreen or not) . This is standard for analysts in other analytics products (zoomdata/tableau etc) thanks for considering!

[fbaligand]

Waou, this is an impressive roadmap...
Sounds like a huge work to do !

[timroes]

Indeed it is, thanks for the understanding there and the patience while it might feel not much is going on. Also since it's so much work, and someone feels like contributing more time to Kibana: https://www.elastic.co/about/careers/engineering/jobs/1143849 :-)

[fbaligand]

Thanks for the proposal :)

[stratoula]

This is not possible with Lenms formulas!