Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add auto create action #56122

Merged
merged 2 commits into from
May 4, 2020
Merged

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented May 4, 2020

Backport of #55858 to 7.x branch.

Currently the TransportBulkAction detects whether an index is missing and
then decides whether it should be auto created. The coordination of the
index creation also happens in the TransportBulkAction on the coordinating node.

This change adds a new transport action that the TransportBulkAction delegates to
if missing indices need to be created. The reasons for this change:

  • Auto creation of data streams can't occur on the coordinating node.
    Based on the index template (v2) either a regular index or a data stream should be created.
    However if the coordinating node is slow in processing cluster state updates then it may be
    unaware of the existence of certain index templates, which then can load to the
    TransportBulkAction creating an index instead of a data stream. Therefor the coordination of
    creating an index or data stream should occur on the master node. See Auto create data streams using index templates v2 #55377

  • From a security perspective it is useful to know whether index creation originates from the
    create index api or from auto creating a new index via the bulk or index api. For example
    a user would be allowed to auto create an index, but not to use the create index api. The
    auto create action will allow security to distinguish these two different patterns of
    index creation.
    This change adds the following new transport actions:

AutoCreateAction, the TransportBulkAction redirects to this action and this action will actually create the index (instead of the TransportCreateIndexAction). Later via #55377, can improve the AutoCreateAction to also determine whether an index or data stream should be created.

The create_index index privilege is also modified, so that if this permission is granted then a user is also allowed to auto create indices. This change does not yet add an auto_create index privilege. A future change can introduce this new index privilege or modify an existing index / write index privilege.

Relates to #53100

Backport of elastic#55858 to 7.x branch.

Currently the TransportBulkAction detects whether an index is missing and
then decides whether it should be auto created. The coordination of the
index creation also happens in the TransportBulkAction on the coordinating node.

This change adds a new transport action that the TransportBulkAction delegates to
if missing indices need to be created. The reasons for this change:

* Auto creation of data streams can't occur on the coordinating node.
Based on the index template (v2) either a regular index or a data stream should be created.
However if the coordinating node is slow in processing cluster state updates then it may be
unaware of the existence of certain index templates, which then can load to the
TransportBulkAction creating an index instead of a data stream. Therefor the coordination of
creating an index or data stream should occur on the master node. See elastic#55377

* From a security perspective it is useful to know whether index creation originates from the
create index api or from auto creating a new index via the bulk or index api. For example
a user would be allowed to auto create an index, but not to use the create index api. The
auto create action will allow security to distinguish these two different patterns of
index creation.
This change adds the following new transport actions:

AutoCreateAction, the TransportBulkAction redirects to this action and this action will actually create the index (instead of the TransportCreateIndexAction). Later via elastic#55377, can improve the AutoCreateAction to also determine whether an index or data stream should be created.

The create_index index privilege is also modified, so that if this permission is granted then a user is also allowed to auto create indices. This change does not yet add an auto_create index privilege. A future change can introduce this new index privilege or modify an existing index / write index privilege.

Relates to elastic#53100
@martijnvg martijnvg added :Data Management/Indices APIs APIs to create and manage indices and templates backport v7.8.0 labels May 4, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label May 4, 2020
martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request May 4, 2020
Also use auto create action if minimum node version if 7.8.0 after backing porting elastic#56122
@martijnvg
Copy link
Member Author

@elasticmachine run elasticsearch-ci/1

@martijnvg martijnvg merged commit 6d03081 into elastic:7.x May 4, 2020
martijnvg added a commit that referenced this pull request May 4, 2020
Also use auto create action if minimum node version if 7.8.0 after backing porting #56122
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport :Data Management/Indices APIs APIs to create and manage indices and templates Team:Data Management Meta label for data/management team v7.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants