Document how to view gathered metrics from e2e tests

[ncdc]

#2705 added gathering metrics for e2e tests. We should document how to view these metrics and what to look for when tests are flaky.

[stevekuznetsov]

@sttts can we use the OCP solution for metrics from tests here?

[s-urbaniak]

@stevekuznetsov I found the spot where we snapshot metrics in OCP, it's part of the gather-extra suite: https://github.com/openshift/release/blob/0da1c1404ba699a67c653a061947fee7b8dabdd6/ci-operator/step-registry/gather/extra/gather-extra-commands.sh#L208-L240

[s-urbaniak]

OCP though snapshots in-cluster monitoring only, kcp metrics won't be part of that. We have two possibilities:

• enable user workload monitoring (https://docs.openshift.com/container-platform/4.12/monitoring/enabling-monitoring-for-user-defined-projects.html), gather kcp metrics there, and snapshot that prometheus.
• deploy our own Prometheus instance, and snapshot that database

Once we have any prometheus instance running, we can leverage the gather-extra script above to dump metrics data.

[stevekuznetsov]

@s-urbaniak option 2 (our own Prom) sounds like the winner here - we're running in-process, remember, not on any k8s :)

[s-urbaniak]

done:

• 🌱 test/e2e,cmd/test: scrape metrics for test servers and e2e tests #2774
• kcp: enable prometheus metrics scraping openshift/release#36509
• pkg/promecieus: support single step jobs vrutkovs/promecieus#16
• https://kubernetes.slack.com/archives/C021U8WSAFK/p1676975581816679

[s-urbaniak]

oh, not so fast, we need a bit of documentation as well as the title says 😅