improve logging for cip-auditor

[listx]

Notes from initial prod deployment of cip-auditor

On 2020-02-25 3:57PM, @listx executed ./deploy.sh k8s-artifacts-prod 5c9a73a90fdfd08adaf3d53ae863144922958911952dfa67cbf9a0d33dedde63 in order to deploy the cip-auditor Cloud Run service to the k8s-artifacts-prod project for the first time.

Shortly thereafter, @thockin pushed up a dummy image to us.gcr.io/k8s-artifacts-prod/suspicious/envelope@sha256:ec3ca3ee90e4dafde96c83232b30f17b5e8992ff35479d0661b8f4ff2f21bf74 in order to test the auditor. The goal was to see at least logs of the auditor detecting a bad change to GCR.

However, the default Cloud Run logs failed to show any meaningful messages:

This was very confusing.

About an hour and a half later, it was discovered that cip-auditor did in fact run and process Pub/Sub messages as intended. However, these log messages did not show up in the Cloud Run instance because their Stackdriver resource.type was set to project and not cloud_run_revision, which is what the Cloud Run "Logs" tab picks up by default. The auditor currently prints interesting bits to the log named projects/k8s-artifacts-prod/logs/cip-audit-log, and is not shown currently in the Cloud Run dashboard.

We can improve this situation by simply making sure that all logs we write to the projects/k8s-artifacts-prod/logs/cip-audit-log log are also written to STDOUT for the cip-auditor process.

[listx]

Aside: the e2e tests that we run for the auditing mechanism did not pick this up because those tests grep through logs in the explicitly-named log cip-audit-log.

[listx]

So, I've discovered that I can set the loggers to be more specific about the labels that are attached to them. Upon creation of the loggers, we can set more options as described here https://godoc.org/cloud.google.com/go/logging#Client.Logger. Specifically, we can probably set resource.type to cloud_run_revision and this should be enough to have logs show up in the Cloud Run "Logs" dashboard, which is what we want. Then there is no need to duplicate log statements every time we write to the log named cip-audit-log.

[bartsmykla]

@listx do you think we can change this issue to be good-first-time-issue maybe?

[listx]

@listx do you think we can change this issue to be good-first-time-issue maybe?

I'm not sure, mainly because the verification requires some jumping around the GCP console which is something not everyone is familiar with.

[bartsmykla]

got it

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[listx]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.