scorecard test fails to due to docker rate limiting

[tony-clarke-amdocs]

Bug Report

What did you do?

operator-sdk scorecard bundle

What did you expect to see?

Output from scorecard

What did you see instead? Under which circumstances?

"error running tests context deadline exceeded". This is because the pulling the busybox image from docker.io is rate limited.

"Failed to pull image "docker.io/busybox:1.33.0": rpc error: code = Unknown desc = Error loading manifest for target platform: Error reading manifest sha256:eccadc4fb09194c8163cfb7edcd9727933104e86da2ba8ad076732e5e3702a6a in docker.io/library/busybox: toomanyrequests"

Environment

Operator type:

language go

Kubernetes cluster type:

OpenShift

$ operator-sdk version

operator-sdk version: "v1.7.0", commit: "9291297d89deae7b85a5c2ac4b418a049ba90f7e", kubernetes version: "1.19.4", go version: "go1.15.5", GOOS: "darwin", GOARCH: "amd64"

$ go version (if language is Go)

go version go1.15.8 darwin/amd64

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T11:51:44Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0+7070803", GitCommit:"70708036fc265771f8d0a45598209018a8b9bd3e", GitTreeState:"clean", BuildDate:"2020-12-05T12:01:07Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Provide a configuration option to pull the image from a docker proxy. Presently it's hardcoded into the code

[estroz]

I guess docker rate limits have dropped considerably within the last year. You should be able to pre-load your cluster's registry with the busybox image pulled from a proxy, since the scorecard runner's pod pull policy is IfNotPresent.

/triage support

[tony-clarke-amdocs]

The scorecard test runs as part of a pipeline. That pipeline can target different clusters. Also if you a multi node cluster or a cluster where nodes are dynamically provisioned then I don't think it's practical to pre-load the image.

[estroz]

Ok fair points. Do you already have a proxy set up, or some solution that lets your pull other docker.io images without rate limits?

/ping @jmccormick2001

[tony-clarke-amdocs]

We have a local docker repository (nexus), so we can retag busybox and host it there. Just need a way to make the reference configurable from scorecard.

[jmccormick2001]

if we alllowed scorecard to specify ifNotPresent pull policy would that resolve this?

[tony-clarke-amdocs]

if we alllowed scorecard to specify ifNotPresent pull policy would that resolve this?

No. It would still fail on the first pull.

[jdockter]

@estroz we are seeing this more and more, is there a work around

[estroz]

I'm still in the camp of pre-loading images via a DameonSet, which will provision all nodes with an image:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: prepull
spec:
  selector:
    matchLabels:
      name: prepull
  template:
    metadata:
      labels:
        name: prepull
    spec:
      initContainers:
      - name: prepull
        image: docker
        # Or `bash -c "docker pull <proxy>/busybox && docker tag <proxy>/busybox busybox"`
        command: ["docker", "pull", "busybox"]
        volumeMounts:
        - name: docker-sock
          mountPath: /var/run
        securityContext:
          privileged: true
      volumes:
      - name: docker-sock
        hostPath:
          path: /var/run
      containers:
      - name: pause
        image: gcr.io/google_containers/pause

If that truly is not amenable to your test clusters, then we can talk about a flag to pass some pullable busybox image tag.

[tony-clarke-amdocs]

@estroz This would become part of the SDK? Even still, I don't see how it helps, we will still be rated limited. Surely, the best option is to allow the image to be configured so it can be pulled from a different repository.

[estroz]

You'd apply this DaemonSet to your cluster yourself. Wouldn't this solve rate limiting since docker pull busybox would only be pulled on node join or pod restart [edit: unless you are spinning up a test cluster on each test run see the commented command in my example manifest]? I am trying to find a solution that does not allow changing the image scorecard uses, since we test with a specific image. I realize you only want to proxy busybox but others will use this "feature" to change the image used.

[tony-clarke-amdocs]

@estroz It doesn't seem like a valid solution for two reasons.

1. It's a workaround and users will have to know to do this step
2. It can still be rated limited.

but others will use this "feature" to change the image used.

But there are other places where user can configure the image. For example, the olm.config.yaml also user to specify the scorecard image. image: quay.io/operator-framework/scorecard-test:v1.7.0

I think it's pretty reasonable to allow images to be pulled from a proxy.

[estroz]

If my solution really is that painful, then I am ok with adding an --proxied-init-image-tag flag so you can just set the full image tag (with a warning about support of course).

[estroz]

@jmccormick2001 @theishshah how do you feel about this?

[tony-clarke-amdocs]

@estroz --proxied-init-image-tag seems like a reasonable option.

[jmccormick2001]

I think it would be good to allow images to be overrode via a setting in the config.yaml file. In the same way we let users specify their test images, but this would be a global setting in that config file instead of a per-test setting.

[estroz]

@theishshah do you want to work on this feature? If so, put this in an upcoming release milestone.

[tkrishtop]

@jberkhahn @estroz is somebody already working on this issue?

[estroz]

Nope! Would you like to take this on? If so, feel free to /assign yourself.

[dirgim]

Hi, we've been encountering this problem a lot during operator testing and it's causing false negative results in the operator testing pipelines. Is this issue planned to be handled soon?

[tkrishtop]

hi @dirgim I'm working on somehow similar issue #5285

[tkrishtop]

Since #5306 was merged, we could probably close this issue as well.

[estroz]

/close

[openshift-ci]

@estroz: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.