[Feature] Kind-based druid e2e tests and pipeline integration

[shreyas-s-rao]

Feature (What you would like to be added):
As a developer, I would like be able to easily run the e2e tests for etcd-druid without the need for an external cluster. Ideally, I would like to use a local kind cluster to do this, similar to how gardener/gardener does it.

I would also like to run these e2e tests as part of the CICD pipeline before every PR is merged, to ensure that we aren't required to manually run the e2e tests before every PR/release.

Motivation (Why is this needed?):
Kind-based testing is quite convenient, as already demonstrated by gardener. It allows developers to easily test locally, and also to integrate with the CICD pipeline, which can spin up a kind cluster and run tests against it, rather than trying to procure a k8s cluster from elsewhere.

Approach/Hint to the implement solution (optional):

/area quality
/area testing

[timebertt]

Relevant presubmit and periodic jobs for g/g's default e2e test are configured here:
https://github.com/gardener/ci-infra/blob/7a6196b887987748732e58511de9df1560d06d7f/config/jobs/gardener/gardener-e2e-kind.yaml
I think, it's best to use prow for these jobs as we already have the infrastructure and experience there and can simply copy a lot of configuration, code, etc.

cc @oliver-goetz

[abdasgupta]

Hi, I checked if prow jobs can possibly run e2e tests for druid. What I found is that current images for prow jobs have no provision for fetching AWS secrets from secret server. Moreover, I learned that secret server can't be run in prow jobs as the secret servers can be accessed from concourse only (Please, correct me here if I am wrong.)

[abdasgupta]

So, I checked resorted to concourse to enable running druid e2e tests. I am facing a problem there as well. Currently concourse steps are not able to run KIND. I probably have to make a docker image that will run a KIND cluster and use that cluster to run make e2e.

[shreyas-s-rao]

If that's the case, can we stick to provider Local for e2e tests, which can be run on prow? For it to work for the tests, we will need to run local-path-provisioner to provision the host volumes on the kind node that's running the test etcd.

Either this, or we need to enhance the local provider logic in druid to be able to create a new PVC specifically for the backup store, and mount it onto the backup-restore container and specify the path to the volume correctly, so that etcdbrctl can take local backups to this volume itself. This avoids the need to run local-path-provisioner every time we want to test local provider on druid.

@timuthy @abdasgupta WDYT?

[abdasgupta]

Oliver gave us a way to conduct e2e tests with remote storages and mount secrets in Prow jobs. Currently, I am trying that

[abdasgupta]

I raised #513 and gardener/ci-infra#553 for this

[shreyas-s-rao]

/reopen

Let's keep this open until gardener/ci-infra#553 is merged and the prow job is stable, for correctness of the issue tracking.

[shreyas-s-rao]

Closing since gardener/ci-infra#553 is merged and test runs are stable.
/close