e2e-local "image update" in catalog_e2e_test.go fails

[akihikokuroda]

Bug Report

make e2e-local TEST='image update fails with the following message.

Catalog represents a store of bundles which OLM can use to install Operators
/Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:35
  image update [It]
  /Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:639

  error copying old registry file: failed to exec []string{"skopeo", "--debug", "--insecure-policy=true", "copy", "--dest-tls-verify=false", "--format=v2s2", "--dest-no-creds=true", "docker://quay.io/olmtest/catsrc-update-test:old", "docker://localhost:5000/catsrc-update:xgl57x"}: exit status 1
  Unexpected error:
      <*errors.errorString | 0xc002071410>: {
          s: "failed to exec []string{\"skopeo\", \"--debug\", \"--insecure-policy=true\", \"copy\", \"--dest-tls-verify=false\", \"--format=v2s2\", \"--dest-no-creds=true\", \"docker://quay.io/olmtest/catsrc-update-test:old\", \"docker://localhost:5000/catsrc-update:xgl57x\"}: exit status 1",
      }
      failed to exec []string{"skopeo", "--debug", "--insecure-policy=true", "copy", "--dest-tls-verify=false", "--format=v2s2", "--dest-no-creds=true", "docker://quay.io/olmtest/catsrc-update-test:old", "docker://localhost:5000/catsrc-update:xgl57x"}: exit status 1
  occurred

  /Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:684

What did you do?
Executed make e2e-local TEST='image update

What did you expect to see?

It completes successfully without error.

What did you see instead? Under which circumstances?
Error:

Catalog represents a store of bundles which OLM can use to install Operators
/Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:35
  image update [It]
  /Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:639

  error copying old registry file: failed to exec []string{"skopeo", "--debug", "--insecure-policy=true", "copy", "--dest-tls-verify=false", "--format=v2s2", "--dest-no-creds=true", "docker://quay.io/olmtest/catsrc-update-test:old", "docker://localhost:5000/catsrc-update:xgl57x"}: exit status 1
  Unexpected error:
      <*errors.errorString | 0xc002071410>: {
          s: "failed to exec []string{\"skopeo\", \"--debug\", \"--insecure-policy=true\", \"copy\", \"--dest-tls-verify=false\", \"--format=v2s2\", \"--dest-no-creds=true\", \"docker://quay.io/olmtest/catsrc-update-test:old\", \"docker://localhost:5000/catsrc-update:xgl57x\"}: exit status 1",
      }
      failed to exec []string{"skopeo", "--debug", "--insecure-policy=true", "copy", "--dest-tls-verify=false", "--format=v2s2", "--dest-no-creds=true", "docker://quay.io/olmtest/catsrc-update-test:old", "docker://localhost:5000/catsrc-update:xgl57x"}: exit status 1
  occurred

  /Users/akihikokuroda/development/go/src/operator-lifecycle-manager/test/e2e/catalog_e2e_test.go:684

Environment

• operator-lifecycle-manager version:
  Local system is MAC.
  Running agains latest master. commit id : 4c3c8f1

• Kubernetes version information:

• Kubernetes cluster kind: kind version 0.11.1

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-21T23:01:33Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

I tried to make it work but so far I couldn't.

Additional context

The port-forwarding here probably doesn't work because the context for the kubectl port-forward is not the context of the e2e text execution..

[exdx]

Hi @akihikokuroda, this is a known issue with this test, specifically on kind clusters. The test does pass on minikube however (provided skopeo is installed locally). Currently on our upstream CI this test is skipped, due to the known issue with kind, and downstream it runs differently.

It's not clear how much value there is in getting this test to run in kind and whether it will actually work due to the dependency on an image registry being locally accessible. A solution could be to skip the test if running in kind.

[akihikokuroda]

@exdx Thanks for explanations. It seems that some issues or restrictions in kind cluster. I believe that making skip this test in kind cluster is better than fail always.

[akihikokuroda]

@exdx How can I run the e2e-local with minikube? I don't see any instructions here: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/test/e2e/README.md. Do I have to just set the minikube context as the default context by the kubectl config use-context?

[exdx]

Yes, I think that should work. We supported minikube in the past but recently switched entirely to kind. I setup a minikube instance manually and run make run-local to deploy OLM, then target the specific test via ginkgo.

GO111MODULE=on GOFLAGS="-mod=vendor" go run github.com/onsi/ginkgo/ginkgo -focus "image update" -v --progress ./test/e2e -- -namespace=operators -olmNamespace=olm -dummyImage=bitnami/nginx:latest

[exdx]

I'm not sure why it works with minikube (with the docker driver) and does not with kind. They are both running containerized versions of the cluster so should behave similarly.

[akihikokuroda]

With kind, the test creates a new kind cluster and it doesn't change the default context so the 'kubectl port-forwad` don't work for the test cluster. I also workaounded the port-forward issue but the creation of the registry pod could not pull the image from the image registry pod. I guess that that's a kind cluster.

Would you put the way to run the test with minikube in the /test/e2e/README.md?

Thanks!