[Discuss] Default mappings for dynamic fields

[ruflin]

Today all the templates are setup with the following dynamic templates

            "dynamic_templates" : [
              {
                "strings_as_keyword" : {
                  "mapping" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "match_mapping_type" : "string"
                }
              }
            ],

By default all the unknown fields are mapped by Elasticsearch and all matching string to keyword. This issue is to discuss what our defaults should be moving forward.

Mapping all fields by default comes at a storage cost. In recent releases Elasticsearch has introduced runtime fields and dynamic: runtime as an option. An other option is to disable indexing completely and only allow runtime fields on query time.

Related issue: #39

[ruflin]

@jpountz This also ties into what we should use for the ES base templates. Would be great to get your thoughts on this.
@tsg @simitt @urso @andrewkroh Pinging you as you also have quite a bit of history on this.

[tsg]

I think we'd want the "custom" fields (anything outside of ECS) to be either runtime or index: false. This would mean that we'd need to again specify explicitly all ECS keyword fields that we want to be indexed on write.

On the alerts Alerts side (elastic/kibana#100884) we're having the advantage that all custom fields are not top-level, but nested under alert.original_event

Could we do something similar, in the sense that all custom fields must go under a given prefix? Probably not, right?

[ruflin]

Do we really need to index all the ECS fields? In case of a package, I would expect the fields that should be really index are specified. To cover better for ECS fields there might be the middle ground, that we defined all the ECS prefixes like user.* to be mapped as a runtime field. Everything else is not indexed. If users add fields inside the ECS prefixes, these will also be index.

Having all ECS fields in each mapping even as runtime fields I would assume would mean lots of data in the cluster state.

[ruflin]

I would like to revive this discussion. The Elasticsearch templates for logs-*-*, metrics-*-* etc. still map by default to keyword, ip etc. Same is true for the templates created by Fleet.

In general, I think we should stop indexing all fields by default. The question is, should we go for runtime by default or index false? What are the pros and cons from an Elasticsearch perspective? @jpountz

If we make the change, it should apply in Elasticsearch and Fleet (@joshdover ). Would we consider this a breaking change?

[jpountz]

@ruflin In short,

• runtime fields would be super slow to search and slow to aggregate
• fields with index: false would be slow to search (still much faster than runtime fields) and fast to aggregate
• fields with default options would be fast to search and fast to aggregate

In terms of disk usage, fields with index: false are in-between runtime fields and fields with default options.

Would we consider this a breaking change?

I would not consider this a breaking change. We are free to change the performance characteristics of mappings produced by our integrations. The same queries that work with today's mappings would still work after this change.

[jpountz]

My intuition is that index: false (doc-value-only field) is a good compromise, because it should still perform fast enough so that users could leverage these fields in dashboards, while reducing index-time CPU usage and disk usage compared to default mappings.

Maybe it would help to use a concrete example. Would you only make this change to dynamically-mapped changes, which I expect to be application-specific fields, or also to some of the fields that are populated by Agent like host.name, etc.?

[ruflin]

I had a follow up discussion on this with @jpountz . Some general thoughts:

• index: false This is especially useful for metrics / numbers.
• runtime true: This works for non numbers.

If we would expose the dynamic_mapping to packages, we could start experiment with some of the packages and later make a more fundamental change in ES itself.

I would not consider this a breaking change. We are free to change the performance characteristics of mappings produced by our integrations. The same queries that work with today's mappings would still work after this change.

It sounds like before we do the change, there should be a mechanism in Fleet for users to potentially overwrite the mapping to have the old performance back. @joshdover My understand is that this is possible with the @custom template? So if we set foo to a runtime field by default but a user wants to have it as a keyword indexed, the @custom template could be used?

For the examples:

• The system.cpu dataset ships a lot of fields by default but I challenge if we should index all of them by default.
• json logs: There can be lots of fields in json logs we don't have mappings for as we don't know in advance what fields are in. Mapping these as default to runtime would save storage.

[jpountz]

Some general thoughts:
index: false This is especially useful for metrics / numbers.
runtime true: This works for non numbers.

For clarity, index:false also works with keywords with a similar trade-off: little or no impact on aggregations, only slower filtering (to a much lesser extent than runtime fields). The point about metrics and index:false is that metric fields are likely a good place to get started changing our defaults to be more space-efficient and less CPU-intensive at index time, since metric fields are less likely used for filtering than dimensions. This will likely be the recommendation for TSDB.

[ruflin]

@joshdover Assigned to you for now but likely best driven forward by someone in the ecosystem or Fleet team.