ecs@mappings: support all date fields when date_detection is disabled

[zmoog]

Situation

The ecs@mappings component template supports all the fields in ECS.

For date fields, it supports the following naming conventions:

elasticsearch/x-pack/plugin/core/template-resources/src/main/resources/ecs@mappings.json

Lines 138 to 162 in 212fe03

	{
	"ecs_date": {
	"mapping": {
	"type": "date"
	},
	"path_match": [
	"*.timestamp",
	"*_timestamp",
	"*.not_after",
	"*.not_before",
	"*.accessed",
	"created",
	"*.created",
	"*.installed",
	"*.creation_date",
	"*.ctime",
	"*.mtime",
	"ingested",
	"*.ingested",
	"*.start",
	"*.end"
	],
	"unmatch_mapping_type": "object"
	}
	},

Problem

The mapping works in all circumstances for the date fields that match the above naming convention.

However, other data fields do not match this naming convention:

threat.indicator.first_seen
threat.indicator.last_seen 
threat.indicator.modified_at
threat.enrichments.indicator.modified_at
threat.enrichments.matched.occurred
threat.enrichments.indicator.first_seen 
threat.enrichments.indicator.last_seen

These fields are generally mapped as dates thanks to the date_detection dynamic field mapping option, enabled by default.

If date_detection is disabled, Elasticsearch will not map these fields as date, creating unexpected mapping problems.

Conslusion

ecs@mappings should support all the data fields in ECS by extending the naming convention, even if integration devs or end users disable date_detection for any reason.

References

• Relates Verify mapping problems after migrating to ecs@mappings integrations#10848

[eyalkoren]

When adding these fields, we should update EcsDynamicTemplatesIT as well to run with date_detection: false, to make sure we capture all existing and future fields that are affected by this setting.

[eyalkoren]

These fields are generally mapped as dates thanks to the date_detection dynamic field mapping option, enabled by default.

Note that the last dynamic template in ecs@mappings acts as a fallback to all string values, mapping then to keyword, which is effectively disabling date detection for strings. So what I am not sure about is how come we miss that.
Our tests (should) cover all ECS fields, generate mock String values for fields that are mapped to date, index documents with these values and validate that they are mapped correctly.

[eyalkoren]

I can verify that when setting date_detection: false, these fields get the wrong mapping. I think this means that the automatic date detection is applied somehow before all dynamic templates are analyzed against the input fields, which I don't think is intentional.
Maybe there is a different explanation to that.

Either way, I proposed a fix to mitigate this issue.

[ruflin]

Here is related issue: #109381