Very noisy logs

[barclayadam]

Describe the bug

I have just upgraded Ambassador from Helm chart v6.5.11 to v6.5.18 and have noticed that the logs coming from the ambassador pods are very noisy:

time="2021-02-17 16:49:40" level=warning msg="license_secret_watch: empty decoded license data" func=github.com/datawire/apro/cmd/amb-sidecar.runE.func3 file="github.com/datawire/apro/cmd/amb-sidecar/main.go:259" CMD=amb-sidecar PID=13 THREAD=/license_secret_watch
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"apps", Name:"...n-cf-ca-cert"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"apps", Name:"...-cf-ca-cert"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"apps", Name:"...-cf-ca-cert"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"apps", Name:"...-cf-ca-cert"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"apps", Name:"...-cf-ca-cert"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"ambassador", Name:"ambassador-edge-stack"}
2021/02/17 16:49:43 Taking K8sSecret snapshot.SecretRef{Namespace:"ambassador", Name:"fallback-self-signed-cert"}
2021-02-17 16:49:44 diagd 1.11.1 [P27TAEW] INFO: configuration updated (complete) from snapshot snapshot (S25 L5 G12 C8)

The above lines are repeated every 4 seconds. Before upgrading the log was almost exclusively just HTTP log lines (i.e. `ACCESS [time] "GET /api HTTP...". Now, oddly, I do not see those lines often at all (even though they should be very regular given I have external probes that constantly hit the services)

Expected behavior
A lot less noise in the logs. These lines are offering little value, and especially at the rate I am seeing them.

Versions (please complete the following information):

• Ambassador Helm Chart v6.5.18
• Azure Kubernetes Cluster v1.18.10

[MateuszCzubak]

Same issue on v6.5.17. What's more important, the CPU usage is extremely high when compared to v6.5.11

[kflynn]

@barclayadam Ouch, mea culpa. Thanks for the report!

@MateuszCzubak Could you open a separate issue for the CPU usage you're seeing? Thanks!

[kflynn]

This is fixed in Ambassador 1.12.0.