[logging] Use Elastic Common Schema (ECS)

[tylersmalley]

Since everything is moving to https://github.com/elastic/ecs and we're looking to ingest logs into Elasticsearch, we should use 8.0 as a chance to migrate to ECS by default. Before 8.0, this format could be opt-in. So, the JSON output would be in ECS format.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[joshdover]

We may need to coordinate here between Operations and Platform. We are currently working on moving to a new logging configuration that more closely matches Elasticsearch's config (#41956). I don't anticipate any conflicts here, I think we'll just need to change the json layout (src/core/server/logging/layouts/json_layout.ts) to match the ECS field names.

@restrry anything else we should consider here?

[mshustov]

@tylersmalley do you know if elasticsearch already adopted ECS format for JSON layout?
Output for ESJsonLayout in 7.5.2:

{"type": "server", "timestamp": "2020-02-05T09:53:14,274+01:00", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch", "node.name": "Mikhails-MBP.Speedport_W723_V_Typ_A_1_01_022", "message": "closed", "cluster.uuid": "nenqSLyJSICGb3D6Z-77hA", "node.id": "VLonIgw2SWOVwbk4xUi8_Q"  }

I can see component, but not log.logger here.

Also, I have some questions regarding the Log ECS format https://www.elastic.co/guide/en/ecs/current/ecs-log.html
Who it the right person to ask?

• Should we prefix all logs with the log prefix?
• Do all the http events, error events, os events, etc. co-exist in the one log record?

@restrry anything else we should consider here?

we need to go through the list of ECS fields to understand how they are matched to the existing format. We might have to add some helpers to formalize what type of data we expect to be passed to the logger. For example for HTTP log record with the format https://www.elastic.co/guide/en/ecs/current/ecs-http.html:

interface HttpLogRecord {
  request: {
    body: {
      bytes: number;
      content: string;
   };
   bytes: number;
   method: HttpMethod;
   referrer: string;
   // ...
}

log.info(message: string, meta: HttpLogRecord | Record<string, any>)

Right now it looks like another breaking change in the logging format. It should be done before GA of NP logging config.

[mshustov]

@jbudz Do you have questions about this issue?
We could start by answering questions from #52226 (comment) Probably, some common ECS questions can be addressed to #ecs channel. Also, we can borrow some ideas from elasticsearch implementation.

After that we can start reformatting existing logs to ECS naming format:
level --> log.level
pid --> process.pid

[jbudz]

do you know if elasticsearch already adopted ECS format for JSON layout?
Output for ESJsonLayout in 7.5.2:

as of 8.0. There's an option to fallback to the older json format.

Should we prefix all logs with the log prefix?

yes the log properties would be prefixed, log.level, etc

Do all the http events, error events, os events, etc. co-exist in the one log record?

circling back on this for a full example. i believe there's room for interpretation, but generally they'll be separate. http events would be one log entry separate from status events

we need to go through the list of ECS fields to understand how they are matched to the existing

on it - combining this with the previous question, i'll add a group by group example for review/discussion

[mshustov]

@jbudz just for reference: elasticsearch doesn't define a specific type of serializable object passed to logs, but
all the properties abide by ECS format:
https://github.com/elastic/elasticsearch/blob/aa0148735e3384b23d504d44ed4f2e20f64f1bca/x-pack/plugin/security/src/main/java/org/elasticsearch/xpack/security/audit/logfile/LoggingAuditTrail.java#L261-L274

[jbudz]

This is going to evolve over time so I'm going to reopen this for discussions.
Currently the JSON formatter is ECS compliant. At a low level there's a couple changes that need to happen

1. rename ECS fields object from meta
2. use internal merge instead od lofash merge
3. apply formatters for logs

There's no end users of the logger yet (the new platform logger in general). I'll be taking a first pass with elasticsearch query logging, and audit logging will probably be a parallel effort. This is going to overlap with other logging tasks e.g. log sanitization.

For now, the API is log(message, ecs_fields.) ECS fields are optional, and each entry will get a timestamp, pid, log level. I'm hoping to keep the top level API the same but the internals are going to evolve.

Here's a general list of additions:

1. type checking
2. revisit default fields
3. log grouping and a template for how, e.g. what fields do we include in an http request and done by passing req to a subclass of log

[mshustov]

revisit default fields is it related to formatters naming? #71138 (comment)

[jbudz]

Ah as in by default each log will message will include....will include fields x y and z and we can probabbly extend it to log subclasses too

[mshustov]

@jbudz any progress with this?

[jbudz]

There was a commit merged recently which did 1 and 2 (not from me), and 3 makes sense with an implementation. We can reopen this or go for specifics in a new issue, but going to close this for now.

[mshustov]

There was a commit merged recently which did 1 and 2 (not from me),

@jbudz could you link the PR? I don't see it's fixed.

rename ECS fields object from meta

uses meta:

kibana/src/core/server/logging/layouts/json_layout.test.ts

Lines 99 to 104 in ab92bbb

	meta: {
	version: {
	from: 'v7',
	to: 'v8',
	},
	},

use internal merge instead od lofash merge

useslodash

kibana/src/core/server/logging/layouts/json_layout.ts

Line 57 in ab92bbb

merge(

apply formatters for logs

Could you elaborate on makes sense with an implementation ?

[jbudz]

I opened #86330 for the lodash merge fix. Apologies, at one point I had a PR up that had merge conflicts on the imports and I must have wrongly followed the thread.

How do you feel about custom instead of meta for adding additional fields?