Allow JSON logging for consul-k8s

[paksu]

Is your feature request related to a problem? Please describe.

This is both a problem and a feature request IMO.

We are running consul connect mesh in our GKE cluster and we inject the consul sidecars in each pod of our service. Our service logs in JSON but the consul-k8s logs in default go-hclog format. This translates into log level ERROR in stackdriver even when the logs are info or debug. This creates additional noise to logs of every pod in our services and it's hard to see what are errors and what are not errors.

Naturally you could get around this by restricting the consul-k8s log level to warnings or errors but IMO it would better to try to be compatible with Stackdriver and preserve original log level.

Feature Description

To have better compatibility with Stackdriver you could enable configuring the consul-k8s log format to be JSON. This seems to be already supported by go-hclog but there is no way to pass the parameter to the library.

Use Case(s)

Enabing Stackdriver to parse logs (when they are JSON) should improve the life of the users of Consul Connect.

Contributions

I'm not a very good go developer but with guidance I could take a stab at this. I think this also requires a change in https://github.com/hashicorp/consul-helm to be able to pass log format setting in the first place.

[ishustava]

Thanks for the well-described issue @paksu. We'll see if other folks in the community need this as well and prioritize as needed.

[paksu]

Another setting that could be passed along with the log format would be the log level, currently there is no way to control the log level of the sidecar.

[kasper]

+1, would make it easier to query the logs as well.

[ishustava]

Another setting that could be passed along with the log format would be the log level, currently there is no way to control the log level of the sidecar.

Are you referring to the envoy sidecar log level? You can control the log level with an annotation or a helm value.

[paksu]

@ishustava Nope, I'm talking about the consul-sidecar container, not the envoy container. I think both of these problems could be fixed there was the same kind of mechanism for extra args for the consul sidecar.

Now anything that the consul-sidecar logs look like this in GCP Cloud Logging. Loglevel is info but it's treated as an error.

So I'm asking for a way to a) configure JSON logging so log level gets parser correctly or b) a way to configure minimum log level so we would only log errors and the info logs don't get unnecessarily spammed as errors

[ishustava]

Got it, thanks for clarifying!

[david-yu]

Hi @paksu and @kasper thank you for your feedback. Would the expectation be to expose JSON logging format via Consul Helm as a global setting for all logs? I would think that would be something you would want to set once for all components.

@paksu Could also open up another issue to track log level of sidecar? I assume you are looking to do this on a per sidecar basis as opposed to globally configuring consul-sidecar container log levels.

[paksu]

Hi @david-yu I think having a global control would make sense. When running consul on GKE I have hard time figuring out why someone would want different logging formats from individual services/containers.

I'll create a separate issue for the log level! Being able to control that at the sidecar level makes sense, basically allowing the same kind of control that you have for individual envoy sidecars.

[paksu]

@david-yu Created the log level as a separate issue here #525