Add require_data_stream feature

[eyalkoren]

Closes #97032

Adding the ability to set require_data_stream parameter (boolean) on several APIs.
For document indexing and update, this flag affects also auto creation of underlying index - if set to true, an index will be created only if a matching index template is found and it contains a data stream template.

[elasticsearchmachine]

Hi @eyalkoren, I've created a changelog YAML for you.

[dakrone]

This looks like a good general direction to me. I think we should check whether writing to an alias that points to a data stream will still work with the require_data_stream flag (and potentially whether it should or should not.) I left a few minor comments since this is still a draft PR

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[dakrone]

Alright, I've pulled out the update support, and made this flag mean "if require_data_stream is set, the indexing operation needs to be targeting a data stream, or a data stream that will be created by a template". I've also added more tests for this. Hopefully it's clearer and not too bad to review.

[jbaiera]

LGTM! 🚢

[dakrone]

@elasticmachine update branch

[axw]

🎉 thank you @dakrone and @eyalkoren!

[leandrojmp]

Hello @dakrone and @eyalkoren

Does this change have any impact when using Logstash elasticsearch output to index data on data streams, but on data streams that have a custom naming pattern?

We are planning the upgrade to 8.13 and checking the release notes, we have Logstash writing to a custom data stream and since logstash does not support custom data stream names, we use data_stream => false and on the index option we point to the data stream alias name and use the action as a create.

Our logstash outputs are like this:

output {
    elasticsearch {
        hosts => ["HOSTS"]
        index => "data-stream-name"
        action => "create"
        http_compression => true
        data_stream => false
        manage_template => false
        ilm_enabled => false
        cacert => 'ca.crt'
        user => 'USER'
        password => 'PASSWORD'
    }
}

Not sure if this change here will break this.

[axw]

@leandrojmp this is an opt-in feature, so it will have no effect on your use case. You would need to reconfigure Logstash to pass ?require_data_stream=true in the _bulk API requests for it to have any effect.