Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bundle by default miminal composable templates for the new indexing strategy #56709

Closed
ph opened this issue May 13, 2020 · 7 comments · Fixed by #57629
Closed

Bundle by default miminal composable templates for the new indexing strategy #56709

ph opened this issue May 13, 2020 · 7 comments · Fixed by #57629
Assignees
@dakrone
Labels
:Data Management/Indices APIs APIs to create and manage indices and templates Team:Data Management Meta label for data/management team

Comments

@ph
Copy link
Contributor

ph commented May 13, 2020 •
edited
Loading

As part of the new indexing strategy proposed in this doc we would like to add by default a few v2 templates that will be installed by default.

That will match the following index patterns:

logs-*-*
metrics-*-*
events-*-*

This is to ensure that the Elastic Agent just works out of the box before Kibana is started.

The downside of this indices matching the above patterns will potentially have a conflict. One of the ideas was that we provide a config flag in Elasticsearch to disable this setup in 7.x so users can opt out.

With the flag we didn't need this setting as the v1 template was still honored even if a v2 template was around and only had priority if the flag was set which was the case for data sent from the agent. Now that we don't have the flag anymore I assume we need such a setting or other ideas how to help users that will face this conflict in 7.x.
@jasontedor jasontedor changed the title Bundle by default miminal v2 templates for the new indexing strategy Bundle by default miminal composable templates for the new indexing strategy May 13, 2020
@jakelandis jakelandis added the :Data Management/Indices APIs APIs to create and manage indices and templates label May 13, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Indices APIs)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label May 13, 2020
@ph
Copy link
Contributor Author

ph commented May 14, 2020

Changed the name of the index_pattern in the above issues for

logs-*-*
metrics-*-*
events-*-*

This expand into ${type}-${dataset}-${namespace}

cc @ruflin

@dakrone dakrone self-assigned this May 27, 2020
@dakrone
Copy link
Member

dakrone commented May 27, 2020

Marked this as team-discuss so we can discuss in @elastic/es-core-features where in the codebase this will live.

@dakrone
Copy link
Member

dakrone commented May 28, 2020

We discussed this and decided to make a new "stack" module in the x-pack directory for these templates and other common infrastructure (to be added to in the future).

@ruflin
Copy link
Contributor

ruflin commented Jun 2, 2020

We soon need to have a conversation about the exact content of these templates. I started a draft PR here elastic/package-registry#481 but no real progress yet. It will be important that the ES team also chimes in especially on the setting sides for recommendations. I will ping here as soon as we need input.

@dakrone
Copy link
Member

dakrone commented Jun 2, 2020

@ruflin I am almost done with this work already, should we have the discussion on the PR itself once I open it?

@ruflin
Copy link
Contributor

ruflin commented Jun 2, 2020

@dakrone That would be great! Can you ping me on it? BTW lets only add logs-- and metrics-- for now as we might not need events-- for now.

dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 3, 2020
@dakrone
Add default composable templates for new indexing strategy
9f3af21 
This commit adds the component and composable templates, as well as ILM policies, for the new
default indexing strategy. It installs:

- logs-default-mappings (component)
- logs-default-settings (component)
- logs-default-policy (ilm policy)
- logs-default-template (composable template)
- metrics-default-mappings (component)
- metrics-default-settings (component)
- metrics-default-policy (ilm policy)
- metrics-default-template (composable template)

These templates and policies are managed by a new x-pack module, `stack`, and can be disabled by
setting `stack.templates.enabled` to `false`.

These ensure that patterns for the `logs-*-*` and `metrics-*-*` indices are set up to create data
streams with the proper mappings and settings.

This also makes changes to the `IndexTemplateRegistry` to support installing component and
composable templates (previously it supported only legacy templates).

Resolves elastic#56709
@dakrone dakrone mentioned this issue Jun 3, 2020
Merged
dakrone added a commit that referenced this issue Jun 30, 2020
@dakrone
Add default composable templates for new indexing strategy (#57629)
3b68df2 
This commit adds the component and composable templates, as well as ILM policies, for the new
default indexing strategy. It installs:

- logs-default-mappings (component)
- logs-default-settings (component)
- logs-default-policy (ilm policy)
- logs-default-template (composable template)
- metrics-default-mappings (component)
- metrics-default-settings (component)
- metrics-default-policy (ilm policy)
- metrics-default-template (composable template)

These templates and policies are managed by a new x-pack module, `stack`, and can be disabled by
setting `stack.templates.enabled` to `false`.

These ensure that patterns for the `logs-*-*` and `metrics-*-*` indices are set up to create data
streams with the proper mappings and settings.

This also makes changes to the `IndexTemplateRegistry` to support installing component and
composable templates (previously it supported only legacy templates).

Resolves #56709
dakrone added a commit to dakrone/elasticsearch that referenced this issue Jun 30, 2020
@dakrone
Add default composable templates for new indexing strategy (elastic#5…
39c27d6 
…7629)

This commit adds the component and composable templates, as well as ILM policies, for the new
default indexing strategy. It installs:

- logs-default-mappings (component)
- logs-default-settings (component)
- logs-default-policy (ilm policy)
- logs-default-template (composable template)
- metrics-default-mappings (component)
- metrics-default-settings (component)
- metrics-default-policy (ilm policy)
- metrics-default-template (composable template)

These templates and policies are managed by a new x-pack module, `stack`, and can be disabled by
setting `stack.templates.enabled` to `false`.

These ensure that patterns for the `logs-*-*` and `metrics-*-*` indices are set up to create data
streams with the proper mappings and settings.

This also makes changes to the `IndexTemplateRegistry` to support installing component and
composable templates (previously it supported only legacy templates).

Resolves elastic#56709
dakrone added a commit to dakrone/elasticsearch that referenced this issue Sep 9, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data
af938d8 
For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 elastic#56709 and elastic#57629.

Resolves elastic#61665
@dakrone dakrone mentioned this issue Sep 9, 2020
Merged
dakrone added a commit that referenced this issue Sep 14, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data (#62193)
d21ddb5 
* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 #56709 and #57629.

Resolves #61665
dakrone added a commit to dakrone/elasticsearch that referenced this issue Sep 14, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data (elastic#62193)
9f84054 
* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 elastic#56709 and elastic#57629.

Resolves elastic#61665
dakrone added a commit that referenced this issue Sep 14, 2020
@dakrone
[7.x] Add "synthetics-*-*" templates for synthetics fleet data (#62193)…
6b2af30 
… (#62346)

* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 #56709 and #57629.

Resolves #61665
@jalvz jalvz mentioned this issue Oct 22, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakrone dakrone

Labels
:Data Management/Indices APIs APIs to create and manage indices and templates Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add default composable templates for new indexing strategy dakrone/elasticsearch
5 participants
@ph @dakrone @ruflin @jakelandis @elasticmachine