Clarification for type vs event.type

[ceeeekay]

Hi there,

We're currently trying to align as closely as possible with ECS as part of a major version upgrade.

I notice from #242 that event.type has recently been set to reserved when previously it seemed the intent was to replace any root type field, (e.g., for Logstash conditionals, etc) with event.type.

We are already some way along with converting all our type fields to event.type and need clarification on what we should be doing at this point. What is the best place to move any type fields to?

Thanks.

[ruflin]

Can you share what kind of values you have in the type field today?

[ceeeekay]

@ruflin sure thing: anything like: apache, apache-[app_name]-error, auditd, postfix, etc.

Just the regular stuff you would expect to find in a root type field for a long-existing elastic stack.

We're fine with moving it (if we do it soon), but we really need to know where to put it to match ECS. The original ECS fields.yml gave the impression the event.type was the right place for this.

[MikePaquette]

@ceeeekay great question that pulls on two related threads 1) event categorization and 2)source types.

1. The ECS fields: event.kind, event.category, event.type, and event.outcome are reserved so that we can define a canonical ECS event categorization scheme that can be used across all data sources.

2. There are two ECS fields, that although they may have some overlap with the above, should be used for mapping your existing data sources (fka types) to ECS. These are event.module and event.dataset. We expect that event.module would be used as a higher-level field, while event.dataset would be used to provide more details. Also, it may be possible that there could be multiple datasets in a given event. The Elastic team expects to use these fields similarly, populating the event.module field with the Beats or Logstash module name, and the event.dataset field with the name(s) of the dataset, metricset, or fileset associated with the module.

For example, an apache error log event could populate these fields like:
event.module: apache2
event.dataset: error

Another example, an apache access log event could populate these fields like:
event.module: apache2
event.dataset: apache_access <-- you may want the dataset name to include the module name

Another example, auditd file events could populate these fields like:
event.module: auditd
event.dataset: file_integrity

Another example, auditd system call events could populate these fields like:
event.module: auditd
event.dataset: system_call

Another example, Palo Alto pan_traffic events could populate these fields like:
event.module: traffic
event.dataset: scan

Another example, Palo Alto pan_threat events could populate these fields like:
event.module: threat
event.dataset: wildfire_virus

Hope this helps.

[ruflin]

I think what @ceeeekay describes could fit well into event.dataset. What is important about event.dataset is it describes a unique data. Taking event.dataset: apache.status as an example, a JSON example can be found here: https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-apache-status.html

I expect all your postfix events look the same as an example?

[ceeeekay]

@MikePaquette @ruflin Sorry about the radio silence - the Xmas break got in the way.

We were right in the middle of the upgrade (5.x -> 6.x) when we came across this change in the specification, and decided to namespace our type field away from anywhere that may conflict with ECS fields.

Part of the reason for this is we also use a subtype field for processing events like apache logs which have non-standard formats, but there was no clear place to put these in ECS, and we really needed to get our new config in place and pull the trigger on a reindex to get the upgrade finished.

While event.module and event.dataset may have somewhat fit, we weren't comfortable using event.module for events that weren't actually generated by a Beats module.

[jamiehynds]

Closing, as question was answered by Mike and Nicolas.