Transition Beats to ECS

[ruflin]

With 7.0 Beats will transition to ECS: https://github.com/elastic/ecs This meta issue is to track all changes needed in Beats. The list will be extended over time

Migration Strategy

The overall migration strategy is to add a alias layer to 7.x which is opt-in to be backward compatible with 6.x data if needed. For some of the core fields used in the Infra / Logging UI aliases are introduced in 6.x for the 7.x data.

• Introduce migration file to track migration Update migration file to new structure #9381

6.x (6.6 / 6.7)

• Add field aliases for 1-1 for Infrastructure UI and Logs UI Introduce aliases for 7.x fields in 6.x #9283
  • beat.hostname -> host.hostname
  • docker.container.name -> container.name
  • docker.container.image -> container.image
  • meta.cloud.availability_zone -> cloud.availability_zone
  • meta.cloud.machine_type -> cloud.machine.type
  • meta.cloud.project_id -> cloud.project.id
  • meta.cloud.provider -> cloud.provider
  • metricset.module -> dataset.module
  • fileset.module -> dataset.module
• Add breaking changes fields
  • event.dataset for Metricbeat Introduce event.duration and event.dataset for backward compatiblity #9393
  • event.dataset for Filebeat Add event.dataset to filebeat #9457
  • event.duration for metricbeat (unit change) Introduce event.duration and event.dataset for backward compatiblity #9393
  • log.source.ip & log.path Introduce log.source.address and log.file.path for 7.x compatiblity #9435
• Correctly represent aliases in the asciidoc
  • Backport Cherry-pick #9269 to 6.x: Fix alias field generation in docs #9498

7.0

• Remove old fields which had a 1-1 mapping like beat.hostname
• Make agent.* overwritable for apm-server move agent metadata to a processor #9952
• Make sure all alias from the migration contain the migrate: * flag

Fields changes

• Introduce structured file to document field migration for later automated migration: Add tracking file for ECS migration #8878
• Import ECS 1.0.0 Beta 2 field definitions
  • Direct attempt [WIP] Import ECS 1.0.0 Beta 1 field definitions #9014 (may be discarded)
  • Update http field set Update the HTTP field set with ECS definitions as of beta 2 #9645
  • Import various small changes from ECS Beta 2 Finish importing ECS Beta 2 differences #9738
  • Update os field set in all relevant places Importing changes to os file set from ECS Beta 2 #9763
  • Import the geo field set
• Some ECS field definitions casually refer to other ECS Readme sections in the Beats docs. We need to address this better Moved to later.
• Import Make user.group a nesting of the group field set ecs#308, where user.group becomes the group field set Import user.group changes from ECS #10275
• Review all fields using text indexing. All fields except message and error.message should be keyword. If a field requires text indexing, it should be added as a multi-field under the keyword field.
  • Review or adjust Auditbeat. Suggest fields as keywords #10518
  • Review or adjust Filebeat. [Filebeat] List of text fields to be converted #10372
  • Review or adjust Heartbeat. Suggest fields as keywords #10518
  • Review or adjust Journalbeat. Suggest fields as keywords #10518
  • Review or adjust libbeat.
  • Review or adjust Metricbeat. Change text fields to keyword for Metricbeat #10318
  • Review or adjust Packetbeat. Suggest fields as keywords #10518
  • Review or adjust Winlogbeat. [Winlogbeat] Fix duplicated type entry #10373

Libbeat adjustments

• Rename beat.name to agent.type Migrate beat.* to agent.* #8873
• Rename beat.version to agent.version Migrate beat.* to agent.* #8873
• Rename beat.hostname to agent.hostname Migrate beat.* to agent.* #8873
• Introduce config option to enable / disable alias Add migration.enabled config option #9805
• Add agent.id and agent.ephemeral_id Add agent.{id,ephemeral_id} to all beat events #9404

Beats processors

• Review all processors for necessary changes, and list changes required below
• add_cloud_metadata Move add-cloud-metadata to ECS #9265
  • Remove nesting under meta.*, cloud.* should be at the top level.
  • Rename to cloud.instance.id
  • Rename to cloud.machine.type
  • ECS doesn't have cloud.project_id or project.id. Should we address this in ECS or leave project_id as is? No
• add_docker_metadata Migrate add_docker_metadata to ECS #9412
  • Use the field set container.* at the top level, not nested under docker.*, and make sure to review field names vs ECS Migrate add_docker_metadata to ECS #9412
• Decide what to do with docker.container.labels as alias does not work here (object)
  • Decision is we don't migrate the labels, it's a breaking change.
• add_host_metadata (fields already in ECS schema, some additional fields like build or codename exist which is ok)
  • Use host.hostname Rename host.name to host.hostname and add config option for name #9943
  • Add host.os.name Add OS Name into add-host-metadata #9405
• add_locale Rename beat.timezone to event.timezone in add_locale processor #9458
  • We've taken out timezones as offsets in seconds from ECS. But we still need to add the tz name to ECS. We may want to change the default? Rename beat.timezone to event.timezone in add_locale processor #9458
• add_process_metadata Rename process.exe to process.executable for ECS #9949
  • the names seem to match very well, but some fields are missing from ECS. We should add them for 7.0.0 and make sure Beats is in sync. (additional fields are ignored for now)
• dns (no option default values to change)

Auditbeat

• Add missing ECS field defs used by Auditbeat Add Auditbeat system module fields to fields.ecs.yml #9318
• Review current Auditbeat GA modules for ECS compatibility [Auditbeat] Review Auditbeat's auditd and FIM modules with regards to conforming to ECS #10111
  • Perform minor changes outlined in this review
• Review new SecOps Auditbeat modules for ECS compatibility

Filebeat

• The redis input has a read_timestamp which should be changed to event.created Rename Redis input read_timestamp to event.created #9924

Filebeat modules

• Rename fileset.name to event.dataset Migrate fileset to ECS #8879
• Rename fileset.module to event.module Migrate fileset to ECS #8879
• Convert source field to ECS Rename source field in Filebeat  #8902
• Rename offset to log.offset Rename field offset to log.offset #8923
• Rename source_ecs to source Rename source_ecs to source #8983
• How do we migrate lower case for http.request.method?
• Changes likely to affect multiple modules at once
  • Output timestamp when Filebeat read an event to event.created, and not read_timestamp Replace read_timestamp with event.created in all remaining Filebeat modules #10139
  • Use [source|destination].address for the ambiguous address (prior to parsing an IP, socket, domain) everywhere Use the .address fields for ambiguous address prior to extracting IP & domain #10141
  • Transition HTTP size and timing metrics to use ECS fields Migrate to ECS Filebeat modules populating http size and duration metrics #10188
  • [Optional] Finish event duration migration: remove all old fields, mention them in ecs-migration as alias: false and with scale:, use the shared Ingest Node code, to reduce compilations. Finish migration to event.duration for 4 Filebeat modules #10274
  • Finish transition to ECS of the user_agent output: get rid of all the field renames in the pipelines. 10472, Finalize user_agent migration to ECS #10441
  • Remove deprecated field url.hostname. Remove field url.hostname. #10469

Filebeat Module migrations

• apache2
  • access Convert apache2.access to ECS - Take 2 #9245
  • error Migration of apache2 error fileset to ECS #8963
• auditd: log Convert the Filebeat auditd module to ECS #10192
• elasticsearch: audit, deprecation, gc, server, slowlog Convert Filebeat elasticsearch.* to ECS #9293
• haproxy: log Convert Filebeat haproxy.log to ECS #9117
• icinga: debug, main, startup Convert Filebeat icinga.* to ECS #9294
• iis
  • access Convert Filebeat iis.access to ECS #9084
  • error Convert Filebeat iis.error to ECS and add IPv6 zone support for IIS access logs #9955
• kafka: log Convert Filebeat kafka.* to ECS #9297
• kibana: log Convert Filebeat kibana.log to ECS #9301
• logstash: log, slowlog Convert Filebeat logstash.* to ECS #9935
• mongodb: log Convert Filebeat mongodb.log to ECS #10009
• mysql: error, slowlog Convert Filebeat mysql.* to ECS #10008
• nginx
  • access Convert Filebeat nginx.access to ECS #9081
  • error Convert Filebeat nginx.error to ECS #10007
• osquery: result Non-breaking adjustment of osquery Filebeat module to ECS #10088
• postgresql: log Convert Filebeat postgresql.log to ECS #9308
• redis: log, slowlog Convert Filebeat redis.log to ECS #9315
• system:
  • auth Convert Filebeat system.auth to ECS #9138
  • syslog Convert Filebeat system.syslog to ECS #9135
• Update Suricata vs Mike's spreadsheet Populate more ECS fields in the Suricata module #10006
• traefik: access Convert Filebeat's traefik.access to ECS. #9005
• Revisit all modules doing int coercions in Grok, to see if we need to coerce using :long instead
  Fix recently translated Filebeat modules where integer coercions were added #9598
• Add service.type to modules: Introduce service.type field for all Filebeat modules #10042

Metricbeat modules

• Rename metricset.name to event.dataset Migration of metricset.* fields to ECS #8941

• Rename metricset.module to event.module Migration of metricset.* fields to ECS #8941

• Add service.type to modules. Introduce service.type for all Metricbeat modules #8965

• Decide on metricset.namespace on where the field should go.

  • We keep this for now in metricset.namespace as metricset.* still exists and does not fit ECS:

• Transition container and kubernetes fields to use container fields?

• Update all data.json files

• Map system metricsets (and others) to process in ECS Move Metricbeat module fields to ECS #10218

  • Check hostname fields
  • Check network data fields
  • network metricset Migration of system network metricset to ECS #10325
  • process metricset Migrate system process metricset fields to ECS #10332
  • Zookeeper module [Metricbeat] Zookeeper ecs #10286
  • Kubernetes events: [Metricbeat] Copy k8s.event.message to message field for ECS #10284
  • Mongodb module: [Metricbeat] Move mongodb.status metricset to ECS #10368
  • Elastic stack modules: Updating Metricbeat stack modules to ECS #10350
  • Http module: [Metricbeat] Rename http.request.body to http.request.body.content for ECS #10315

Packetbeat

• Convert Packetbeat basic fields to ECS Convert Packetbeat Flows to ECS #9121
• Convert protocols to ECS [Packetbeat] - Change field names to follow ECS #7968

Journalbeat

• Convert read_timestamp to event.created Convert read_timestamp to event.created in Journalbeat #10043

Heartbeat

• Add event.dataset: Add event.dataset field #9408

Winlogbeat

• Any changes to Winlogbeat needed? Migrate Winlogbeat to ECS, take 2 #10333

Varia

• Temporary fix for dashboards Temporary fix for dashboards under 7.0 #9031
• Populate ecs.version in all relevant places Add ecs.version to each event #9284
• Finish the transition of user_agent parsing to ECS for all web access logs.
• Better representation of field aliases in the documentation Fix alias field generation in docs #9269
• Part 2 to improve alias representation in docs Improve fields view in Docs #9288 (can also happen later)

See also all issues tagged "ecs"

Others

• Apply renaming to all Kibana files Introduce migration script for data in Kibana files #9998
• Apply renaming to ML jobs (ouf of the scope of this issue, ML team will handle it)
• Check where docs need updating Update configs for old beat.* fields #10370
• Introduce add_labels / add_tags processor Introduce add_labels and add_tags processors #9973
• Introduce script to show breaking field changes: Script to generate breaking field changes list #10405

Open questions:

• Should we rename co.elastic.logs/fileset to co.elastic.logs/dataset for autodiscovery (@exekias )
• Should we change the metricsets config option in Metricbeat?
• Proposal by @ruflin Keep it for now as we keep also the field fileset and metricset around

Notes

• The code side is not changed as part of this migration.
• The filebeat generated files must often be updated. Use the following to commands: INTEGRATION_TESTS=1 GENERATE=1 nosetests tests/system/test_modules.py -v, x-pack: MODULES_PATH=./module INTEGRATION_TESTS=1 GENERATE=1 nosetests tests/system/test_xpack_modules.py -v.

[webmat]

@ruflin

• I've created a section listing all modules. We can expand modules to one line per fileset only when needed. I get the feeling we'll end up doing some modules with 1 PR for the whole module, rather than per fileset.
• I've also checked the tasks that were listed and have been merged (e.g. beat.name). If any of those required follow-up work, please add subtasks.

[exekias]

@ruflin about fileset -> dataset. This relies in Filebeat docs, they name these fileset: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html

In my opinion that makes sense, as we are talking about files. They will generate datasets, and that's correct too, but as as long as we name this fileset in Filebeat docs I think the annotation should keep that nomenclature.

[ruflin]

@exekias We must change it in the docs too. Would this solve the issue?

[exekias]

If we completely rename the thing, I would say yes, annotations must follow

[webmat]

@ruflin I've updated the "Beats processors" section. The list and fields to change should be pretty comprehensive now. Please take a look, to confirm I haven't missed something.

cc @roncohen

[webmat]

@ruflin Just added this to the "Field changes" section. I think this would be best solved by moving ECS docs to asciidoc on the doc website:

• Some ECS field definitions casually refer to other ECS Readme sections in the Beats docs. We need to address this better

[sophiec20]

For our UI ML Module automated testing, we do the following:

• restore a data snapshot
• call the Kibana API data recogniser
• call the Kibana API module setup - this creates ML jobs, datafeeds and visualisations and dashboards
• the run the ML jobs
• then check ML results are visible
• then check you can click through to the newly created dashboards (this bit is manual)

We currently use

• filebeat-* where fileset.module": "nginx" AND "fileset.name": "access"
• filebeat-* where fileset.module": "apache2" AND "fileset.name": "access"
• auditbeat-* where "event.type": "syscall" for docker containers and hosts (tbc if ECS changes will affect ML in 7.0)

As we start with data snapshots in our existing test framework, is the beats team able to supply snapshots of indices containing
a) pure new 7.0 ECS data and b) backward compatible indices?
With (a) being the priority.

[ruflin]

With 7.0 you will be able for the above queries to just rely on event.dataset: nginx.access as an example. BTW we also renamed apache2 to apache to be in line with the metricbeat module.

I assume the data you are looking for is nginx and apache data for the logs. What I could produce is a few lines of example data based on our test suite logs. Would that be enough? Or you need larger logs? If you have larger log files for nginx and apache I can easily create the data.

[sophiec20]

@ruflin Can we please start with some example data snapshots?
(We do have larger logs, but I'm not sure if we retained them in their original raw format as they were anonymised. Will need to check and will share with you if we can).

[ruflin]

Our tests logs can be found here:

• apache.acess: https://github.com/elastic/beats/tree/master/filebeat/module/apache/access/test
• nginx.access: https://github.com/elastic/beats/tree/master/filebeat/module/nginx/access/test

I initially thought I provide you with a snapshot or es_archiver zip file from ES. But I think it's easier if the one that works on these files ingests the data himself. Like this also your apache files can be used and it does not have to go through me anymore.

To make the module work with any file path, var.paths must be adjusted in the module config: https://github.com/elastic/beats/blob/master/filebeat/modules.d/apache.yml.disabled

For testing use the snapshot builds:

• OS X https://beats-ci.elastic.co/job/elastic+beats+master+multijob-package-darwin/gcsObjects/
• Linux & Windows: https://beats-ci.elastic.co/job/elastic+beats+master+multijob-package-linux/gcsObjects/

[ruflin]

@webmat Above I did check the checkbox around http.request.method to normalise it. I suggest we skip this for now.

[webmat]

@ruflin Understood. If I can get around to it in time would you have any objections, though?

Not 100% sure I can (e.g. if we don't have what we need in field generation), but I'd like to get it done if possible.

[ruflin]

@webmat No objections :-)

[ruflin]

Closing this issue as all the checkboxes have been done except the following 3:

• http method lower case: We will figure out something at a later stage
• osquery module: According to @webmat conversion to ECS does not make sense
• Breaking changes files: PR is open and will be merged soonish.

A big thank you to everyone that contributed to getting this massive effort done.