Elasticsearch creates index templates with deprecated types

[chrisdavies]

We're trying to get Kibana to work with ES7, but are getting errors like this:

Cannot put multiple mappings: [_doc, doc]

The reason is that we're modifying Kibana to no longer specify the root type for a document, but there are existing index templates which are interfering with us.

Here's a list of template names and their root doc types:

• .ml-meta [ 'doc' ]
• .ml-notifications [ 'audit_message' ]
• .ml-anomalies- [ 'doc' ]
• .ml-config [ 'doc' ]
• .ml-state [ 'doc' ]
• .logstash-management [ 'doc' ]
• .watches [ 'doc' ]
• .triggered_watches [ 'doc' ]
• .watch-history-9 [ 'doc' ]
• .monitoring-alerts [ 'doc' ]
• .monitoring-es [ 'doc' ]
• .monitoring-kibana [ 'doc' ]
• .monitoring-logstash [ 'doc' ]
• .monitoring-beats [ 'doc' ]
• security_audit_log [ 'doc' ]

I saw that there are some individual issues to track this (#37442) but I thought it might be good to have a more comprehensive issue.

Feel free to close if this is redundant!

[elasticmachine]

Pinging @elastic/es-search

[gwbrown]

Per @chrisdavies on Slack, clarifying the issue:

What happens is that Kibana's test suite loads in archives which create indices + mappings and import a bunch of docs, then the tests run against that data.

We've upgraded our archives to match ES7 expectations (e.g. no root type specified). But this causes a conflict with the existing index templates most of which have a root type of 'doc'.

There may be another way around this for the Kibana test suite, but we should still clear all of these up for the reasons detailed in #37442.

/cc @jtibshirani for awareness

[jtibshirani]

Thank you for flagging this @chrisdavies, and sorry that you ran into these issues!

We had actually planned to keep the type name doc for these internal templates for the 7.0 release. There are a few points behind this reasoning:

• In almost all cases we don’t expect deprecation warnings from interacting with these templates, because warnings are only issued at the REST layer and these internal products use the transport client.
• Typeless APIs like 'index' and 'bulk' work against indices with a custom type name, so there is not a strong reason to migrate these index templates right now to be typeless.
• Moving off the type name doc would be a breaking change for those internal products, and also potentially Kibana, as Kibana is still in the process of moving off of the typed APIs.

We will certainly plan to move to typeless templates after the 7.0 release, when we begin to update more server-side code to be typeless, and work towards the goal of removing types completely in elasticsearch.

While I don’t think there’s immediate action in terms of upgrading templates, the error you ran into highlights a bug in our current approach: it’s not possible to issue a typeless 'create index' request when it triggers an index template that contains a custom type. I will look into fixing this bug, as it’s something I think we should be able to support.

Lastly, @jpountz or @bleskes would you be able to confirm if the reasoning above around templates above makes sense to you as well?

[jpountz]

@jtibshirani Agreed.

[bleskes]

@jtibshirani I'm ok with the approach. I'm not 100% sure I understand why we can't update all the templates (it's just json, right?) to remove the types in 7.0.0 but I trust your judgement.

[chrisdavies]

Yeah. This actually makes Kibana's process trickier, I think, because this will mean we have "doc" in some places and "_doc" / default in some places.

That said, what I'm doing right now is just removing the mappings for these indices from our test archives, and that seems to be working out so far. It may be that we don't have any production code in Kibana that explicitly specifies the root type for these indices. I'll know more in a few days once the Eleasticsearch 7 update is complete on our end.

[jtibshirani]

@bleskes The main issue is all the indices generated from the templates would switch from having type doc to being typeless. We would then need to update all the code in internal products to use the type name _doc/ the typeless APIs for 7.0, and also reindex any existing internal indices that were created using the old type doc.

@chrisdavies After we fix the bug mentioned above, I wouldn't think that there'd be a situation where you still need to refer to the old doc type name. If you run into a situation like this or continue to have any issues with the templates, please let us know.

[jpountz]

@chrisdavies Can you tell us what API call you did exactly to run into this error?

[chrisdavies]

It was es_archiver in Kibana, which basically is doing a create index with mappings.

[jpountz]

Ok, so it is creating an index explicitly that matches a template, rather than letting the index be created automatically, this is what I needed to know. Thank you.