Audit logging: phase 1

[kobelb]

Phase 1 as outlined in #52125

Completed

Phase 1 has completed. If all goes to plan, this will be available in the upcoming 7.11.0 release.

Tasks:

• Define new event types, add these events to the Audit Trail service
  https://docs.google.com/document/d/1TwLjsmLxlLfmtCS86Mwk6S1SVjxfq4RrHCYEuuaj4Ns/edit
• Enhance new Auditor plugin (currently named "Audit Trail"):
  • License awareness (ECS audit logging #74640)
  • Enrich Audit Trail service events with info required for records (such as user/session info)
  • Set up appender to write to separate transport/file, with configuration options to control
  • Additional configuration by YAML, with options to include or exclude event types and attributes
• Deprecate legacy audit events in existing audit service
• Implement core middleware for "top-level" audit events (inbound HTTP requests)
  • Generate unique X-Opaque-Id header if it doesn't already exist? (Add support for reading request ID from X-Opaque-Id header #71019)
• Ensure parity on ESS with ES Audit logs
  • Update Kibana's filebeat module to ingest audit logs. (ES module for reference)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[legrego]

We met with @s-nel today to ensure that ESS could ingest our audit logs. It sounds like we "just" need to update Kibana's filebeat module to accommodate the new audit logs that we write.

Additionally, we will want to make sure that the fields we write to within the ECS format don't collide with any fields that ESS itself is expecting to write to:

service.name
service.id
service.node.name
service.version
service.type
cloud.availability_zone
cloud.region

[legrego]

Work on Phase 1 of audit logging has completed. If all goes to plan, this will be available in the upcoming 7.11.0 release.