Change Fleet Server over to use ECS logger to ensure logging format and ECS compliance

[EricDavisX]

From the below comment:
#358 (comment)

we see that Fleet sever had a bug where 2 fields were not ECS compliant. Instead of attempting to test / validate that the current logging is compliant, we can instead switch to the Elastic ECS logger.

This is a ticket for that switch-over work.

More info about ECS logging here: https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html

The APM team owns the ECS logging libraries, fyi.

[simitt]

For go we only support zap and logrus at the moment.

[urso]

Beats/Agent use zap. I have no strong feelings regarding zap vs. zerolog, but given that we have better support for zap and use it in other code bases as well, I think we should standardize Fleet Server on zap as well and start using https://github.com/elastic/ecs-logging-go-zap.

@scunningham Any objections?

[scunningham]

In my opinion, zerolog is a better logger implementation, both from an API and a performance perspective. Our time is better spent fleshing out ECS support for zerolog rather than convert to zap.

I've already spent some time making the fleet-server logging ECS compliant. Much of the work has to do with changing the field names and has little to do with the logger itself. I don't think we'd be buying much by switching to zap frankly. That said, if somebody wants to rewrite all the logging code, more power to them.

[simitt]

There is an open issue for adding a formatter to zerolog similar to what we did for logrus and zap. I don't think there are any immediate plans to work on that, but if anyone had the bandwith to contribute, that would be more than welcome.

[urso]

fyi; elastic/ecs-logging-go-zerolog#1

[michel-laterman]

elastic/ecs-logging-go-zerolog has been released at v0.1.0, it's available as

import "go.elastic.co/ecszerolog"

[EricDavisX]

I'll update this to 7.15 label for now, I don't see it getting any more action during 7.14. We can re-review later.

[urso]

@michel-laterman @scunningham Didn't we introduce ecszerolog in 7.14 already? Can we close this issue?

[scunningham]

@urso Work to switch to ecszerolog has not been done.

[Mpdreamz]

@ruflin @jlind23 similar too: elastic/beats#15544 can we ensure this makes 8.0.0?
Would be a fantastic quick win.