Add correlation section to audit logging docs (#123757)

(cherry picked from commit 6e4c311)

docs/user/security/audit-logging.asciidoc

@@ -55,8 +55,9 @@ For information on how to configure `xpack.security.audit.appender`, refer to
 
 Refer to the table of events that can be logged for auditing purposes. 
 
-Each event is broken down into <<field-event-category, category>>, <<field-event-type, type>>, <<field-event-action, action>> and <<field-event-outcome, outcome>> fields
-to make it easy to filter, query and aggregate the resulting logs. 
+Each event is broken down into <<field-event-category, category>>, <<field-event-type, type>>, <<field-event-action, action>> and
+<<field-event-outcome, outcome>> fields to make it easy to filter, query and aggregate the resulting logs. The <<field-trace-id, trace.id>>
+field can be used to correlate multiple events that originate from the same request.
 
 Refer to <<xpack-security-ecs-audit-schema>> for a table of fields that get logged with audit event. 
 
@@ -448,7 +449,57 @@ Example: `https`
 | *Field*
 | *Description*
 
-| `trace.id`
+| [[field-trace-id]] `trace.id`
 | Unique identifier allowing events of the same transaction from {kib} and {es} to be be correlated.
 
 |======
+
+[[xpack-security-ecs-audit-correlation]]
+==== Correlating audit events
+
+Audit events can be correlated in two ways:
+
+1. Multiple {kib} audit events that resulted from the same request can be correlated together.
+2. If {ref}/enable-audit-logging.html[{es} audit logging] is enabled, {kib} audit events from one request can be correlated with backend
+   calls that create {es} audit events.
+
+NOTE: The examples below are simplified, many fields have been omitted and values have been shortened for clarity.
+
+===== Example 1: correlating multiple {kib} audit events
+
+When "thom" creates a new alerting rule, five audit events are written:
+
+[source,json]
+-------------
+{"event":{"action":"http_request","category":["web"],"outcome":"unknown"},"http":{"request":{"method":"post"}},"url":{"domain":"localhost","path":"/api/alerting/rule","port":5601,"scheme":"https"},"user":{"name":"thom","roles":["superuser"]},"kibana":{"space_id":"default","session_id":"3dHCZRB..."},"@timestamp":"2022-01-25T13:05:34.449-05:00","message":"User is requesting [/api/alerting/rule] endpoint","trace":{"id":"e300e06..."}}
+{"event":{"action":"space_get","category":["database"],"type":["access"],"outcome":"success"},"kibana":{"space_id":"default","session_id":"3dHCZRB...","saved_object":{"type":"space","id":"default"}},"user":{"name":"thom","roles":["superuser"]},"@timestamp":"2022-01-25T13:05:34.454-05:00","message":"User has accessed space [id=default]","trace":{"id":"e300e06..."}}
+{"event":{"action":"connector_get","category":["database"],"type":["access"],"outcome":"success"},"kibana":{"space_id":"default","session_id":"3dHCZRB...","saved_object":{"type":"action","id":"5e3b1ae..."}},"user":{"name":"thom","roles":["superuser"]},"@timestamp":"2022-01-25T13:05:34.948-05:00","message":"User has accessed connector [id=5e3b1ae...]","trace":{"id":"e300e06..."}}
+{"event":{"action":"connector_get","category":["database"],"type":["access"],"outcome":"success"},"kibana":{"space_id":"default","session_id":"3dHCZRB...","saved_object":{"type":"action","id":"5e3b1ae..."}},"user":{"name":"thom","roles":["superuser"]},"@timestamp":"2022-01-25T13:05:34.956-05:00","message":"User has accessed connector [id=5e3b1ae...]","trace":{"id":"e300e06..."}}
+{"event":{"action":"rule_create","category":["database"],"type":["creation"],"outcome":"unknown"},"kibana":{"space_id":"default","session_id":"3dHCZRB...","saved_object":{"type":"alert","id":"64517c3..."}},"user":{"name":"thom","roles":["superuser"]},"@timestamp":"2022-01-25T13:05:34.956-05:00","message":"User is creating rule [id=64517c3...]","trace":{"id":"e300e06..."}}
+-------------
+
+All of these audit events can be correlated together by the same `trace.id` value `"e300e06..."`. The first event is the HTTP API call, the
+next audit events are checks to validate the space and the connectors, and the last audit event is the actual rule creation.
+
+===== Example 2: correlating a {kib} audit event with {es} audit events
+
+When "thom" logs in, a "user_login" {kib} audit event is written:
+
+[source,json]
+-------------
+{"event":{"action":"user_login","category":["authentication"],"outcome":"success"},"user":{"name":"thom","roles":["superuser"]},"@timestamp":"2022-01-25T09:40:39.267-05:00","message":"User [thom] has logged in using basic provider [name=basic]","trace":{"id":"818cbf3..."}}
+-------------
+
+The `trace.id` value `"818cbf3..."` in the {kib} audit event can be correlated with the `opaque_id` value in these six {es} audit events:
+
+[source,json]
+-------------
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,604-0500", "event.action":"access_granted", "user.name":"thom", "user.roles":["superuser"], "request.id":"YCx8wxs...", "action":"cluster:admin/xpack/security/user/authenticate", "request.name":"AuthenticateRequest", "opaque_id":"818cbf3..."}
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,613-0500", "event.action":"access_granted", "user.name":"kibana_system", "user.roles":["kibana_system"], "request.id":"Ksx73Ad...", "action":"indices:data/write/index", "request.name":"IndexRequest", "indices":[".kibana_security_session_1"], "opaque_id":"818cbf3..."}
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,613-0500", "event.action":"access_granted", "user.name":"kibana_system", "user.roles":["kibana_system"], "request.id":"Ksx73Ad...", "action":"indices:data/write/bulk", "request.name":"BulkRequest", "opaque_id":"818cbf3..."}
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,613-0500", "event.action":"access_granted", "user.name":"kibana_system", "user.roles":["kibana_system"], "request.id":"Ksx73Ad...", "action":"indices:data/write/bulk[s]", "request.name":"BulkShardRequest", "indices":[".kibana_security_session_1"], "opaque_id":"818cbf3..."}
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,613-0500", "event.action":"access_granted", "user.name":"kibana_system", "user.roles":["kibana_system"], "request.id":"Ksx73Ad...", "action":"indices:data/write/index:op_type/create", "request.name":"BulkItemRequest", "indices":[".kibana_security_session_1"], "opaque_id":"818cbf3..."}
+{"type":"audit", "timestamp":"2022-01-25T09:40:38,613-0500", "event.action":"access_granted", "user.name":"kibana_system", "user.roles":["kibana_system"], "request.id":"Ksx73Ad...", "action":"indices:data/write/bulk[s][p]", "request.name":"BulkShardRequest", "indices":[".kibana_security_session_1"], "opaque_id":"818cbf3..."}
+-------------
+
+The {es} audit events show that "thom" authenticated, then subsequently "kibana_system" created a session for that user.