Include and exclude patterns in visualizations throw an error

[epixa]

Elasticsearch version: 5.2.0

As of 5.2.0, using an object syntax for exclude in a terms aggregation will result in a 400 illegal_argument_exception error. The exact same query works in 5.1.2.

Example _msearch request body (taken directly from Kibana):

{"index":["logstash-0"],"ignore_unavailable":true,"preference":1486048204582}
{"size":0,"query":{"bool":{"must":[{"query_string":{"analyze_wildcard":true,"query":"*"}},{"range":{"@timestamp":{"gte":1486047452495,"lte":1486048352495,"format":"epoch_millis"}}}],"must_not":[]}},"_source":{"excludes":[]},"aggs":{"2":{"terms":{"field":"referer","exclude":{"pattern":"apollo"},"size":5,"order":{"_count":"desc"}}}}}

Response body:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"[terms] exclude doesn't support values of type: START_OBJECT"}],"type":"illegal_argument_exception","reason":"[terms] exclude doesn't support values of type: START_OBJECT"},"status":400}

This was originally filed against Kibana: elastic/kibana#10153

@javanna believes it may have been caused by #22048

[jpountz]

Good catch. It seems that this exclude/pattern syntax has been undocumented since 2.0, causing it to not be tested, but in the meantime we did not deprecate it either. I think we should add it back to 5.x to not break users in a minor release with a deprecation notice recommending to just do exclude: apollo instead.

[epixa]

++

[GuruRajaY]

removing exclude/include patterns is not fix. Are we getting any fix for this ?

[epixa]

@Guruindya The proposal wasn't to remove the feature entirely but to remove the older, outdated syntax that causes this issue. If you use the exclude/include syntax that you see in the docs right now, that should be fine.

[0asp0]

Is there any workaround available? Some of our vizualisations are using it, causing whole dashboards in kibana to break.

[maxenced]

+1 , if I upgrade my "client" nodes to 5.2, almost all our visualization breaks. The whole cluster is in 5.2 now, except the 2 clients nodes (the ones used by Kibana), which are stuck in 5.0. Also kibana can't be upgraded in 5.2 (as it will fail because some of the cluster's node are in 5.0).

[danielmotaleite]

please consider releasing a urgent update to elasticsearch or kibana to fix this issue, we can not downgrade ES nor kibana and now we are stuck with broken visualization/dashboards. Having a visible problem like this for a long time will for sure generate a lot noise

[danielmotaleite]

@epixa

If you use the exclude/include syntax that you see in the docs right now, that should be fine.

How can i do that in kibana? From what i understand from the kibana bug , there is no workaround in kibana 5.2.0

[GuruRajaY]

Is there any ETA for this fix ? So that we can plan accordingly

[clintongormley]

@Guruindya sorry, but we do not announce release dates ahead of time

[mormor1971]

I have just updated my cluster to v5.2 and hit the same issue with the include / exclude patterns breaking a lot of visualisations so would like to know if there is a 'proper' way to use the include / exclude filters and where they are documented and kept up to date.

One of the things I have struggled a lot with in the past is the lack of documentation for some parts of the ES stack so if there is a proper place to hut this down then please someone point me there.

[epixa]

A fix for this has just been released in 5.2.2 for both Kibana and Elasticsearch.