🌱 test/e2e,cmd/test: scrape metrics for test servers and e2e tests

[s-urbaniak]

Summary

This adds support for scraping metrics via Prometheus for e2e tests and local shard testing. If the environment variable PROMETHEUS_URL is set, scraping targets are being configured automatically.

For e2e tests, ARTIFACT_DIR needs to be set additionally.

This is the first part of adding scraping support. The second part will add wiring in CI.

Related issue(s)

[stevekuznetsov]

Where are the bits in e.g. make that actually start Prometheus, etc?

[s-urbaniak]

@stevekuznetsov re:

Where are the bits in e.g. make that actually start Prometheus, etc?

this is part 2, as mentioned in the summary 😛 working on that now, wanted to have the code in first.

EDIT: I made all the necessary changes in the Makefile in this PR as well.

[nrb]

I think the make targets as written won't actually stop prometheus when e2e tests fail. Added comments where I think changes are needed.

[sttts]

/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ncdc]

Do we want tests to fail if there are (unrelated) issues with Prometheus?

2023-02-17T11:17:20.9252004Z --- FAIL: TestSyncerTunnel (47.06s)
2023-02-17T11:17:20.9258266Z     tunnels_test.go:59: Saving test artifacts for test "TestSyncerTunnel" under "/tmp/e2e/TestSyncerTunnel/1144179117".
2023-02-17T11:17:20.9259388Z     tunnels_test.go:59: Starting kcp servers...
2023-02-17T11:17:20.9315003Z     kcp.go:687: running: /home/runner/work/kcp/kcp/bin/kcp start --root-directory /tmp/TestSyncerTunnel3196772349/003/kcp/main --secure-port=41259 --embedded-etcd-client-port=43919 --embedded-etcd-peer-port=33053 --embedded-etcd-wal-size-bytes=5000 --kubeconfig-path=/tmp/TestSyncerTunnel3196772349/003/kcp/main/admin.kubeconfig --feature-gates=CustomResourceValidationExpressions=true,KCPSyncerTunnel=true --audit-log-path /tmp/e2e/TestSyncerTunnel/1144179117/artifacts/kcp/main/kcp.audit -v=4 --token-auth-file /tmp/TestSyncerTunnel3196772349/001/auth-tokens.csv --audit-policy-file /tmp/TestSyncerTunnel3196772349/002/audit-policy.yaml
2023-02-17T11:17:20.9322783Z     kcp.go:1190: waiting for readiness for server at https://10.1.0.39:41259
2023-02-17T11:17:20.9323443Z     kcp.go:1233: success contacting https://10.1.0.39:41259/readyz
2023-02-17T11:17:20.9328461Z     kcp.go:1233: success contacting https://10.1.0.39:41259/livez
2023-02-17T11:17:20.9329524Z     kcp.go:1211: server at https://10.1.0.39:41259 is ready
2023-02-17T11:17:20.9339402Z     kcp.go:229: 
2023-02-17T11:17:20.9339988Z         	Error Trace:	kcp.go:229
2023-02-17T11:17:20.9340564Z         	            				kcp.go:446
2023-02-17T11:17:20.9346086Z         	            				kcp.go:139
2023-02-17T11:17:20.9346790Z         	            				tunnels_test.go:59
2023-02-17T11:17:20.9351680Z         	Error:      	Received unexpected error:
2023-02-17T11:17:20.9352643Z         	            	kcp prometheus scrape target never became healthy
2023-02-17T11:17:20.9373765Z         	Test:       	TestSyncerTunnel
2023-02-17T11:17:20.9374264Z     kcp.go:674: cleanup: canceling context
2023-02-17T11:17:20.9383333Z     kcp.go:678: cleanup: waiting for shutdownComplete
2023-02-17T11:17:20.9383929Z     kcp.go:682: cleanup: received shutdownComplete
2023-02-17T11:17:20.9392662Z FAIL
2023-02-17T11:17:20.9393232Z FAIL	github.com/kcp-dev/kcp/test/e2e/syncer	142.785s
2023-02-17T11:17:20.9393768Z ok  	github.com/kcp-dev/kcp/test/e2e/syncer/dns	108.474s
2023-02-17T11:17:20.9405062Z ?   	github.com/kcp-dev/kcp/test/e2e/syncer/dns/workspace1	[no test files]
2023-02-17T11:17:20.9405698Z ?   	github.com/kcp-dev/kcp/test/e2e/syncer/dns/workspace2	[no test files]
2023-02-17T11:17:21.0786016Z ok  	github.com/kcp-dev/kcp/test/e2e/virtual/apiexport	1.374s
2023-02-17T11:17:40.0307249Z ok  	github.com/kcp-dev/kcp/test/e2e/virtual/initializingworkspaces	1.087s
2023-02-17T11:17:40.5568997Z ok  	github.com/kcp-dev/kcp/test/e2e/virtual/syncer	0.977s
2023-02-17T11:18:00.5678430Z ok  	github.com/kcp-dev/kcp/test/e2e/watchcache	0.974s
2023-02-17T11:18:00.7017815Z ok  	github.com/kcp-dev/kcp/test/e2e/workspacetype	1.135s
2023-02-17T11:18:00.7088121Z FAIL
2023-02-17T11:18:02.5264217Z make: *** [Makefile:348: test-e2e-sharded] Error 1
2023-02-17T11:18:02.5511521Z I0217 11:18:02.548097   27348 shard.go:242] "gathering shard metrics" shard="kcp-0"
2023-02-17T11:18:02.5541775Z + EXIT_CODE=2

[sttts]

/retest

[sttts]

"kcp prometheus scrape target never became healthy" - what does this mean exactly?

[s-urbaniak]

looking

[s-urbaniak]

@ncdc @sttts I removed the hard check which verifies if the target is up. I am not sure why it was not detected (it is present in the scraped metrics) but it's not worth checking it. The only risk we have is that we'll miss metrics in case of e2e tests that run very fast.

[ncdc]

Re-tagging as @sttts was happy with it 😄
/lgtm