[scorecard] Default store and untar images using tags in a disconnected environment

[tonyskapunk]

Bug Report

Back in #5285 it was reported the problem of having tags in images in a disconnected environment, also known as air-gapped. To solve that issue a couple of flags were introduced #5306 to allow scorecard to use custom images. The default of those images allowed disconnected environments to work out of the box, i.e. no need to set a custom image.

This PR #6335 introduced back tags instead of the digest of those images

preflight tool makes use of operator-sdk scorecard and recently updated to version 1.28.0 with this new version the issue observed in #5285 has reappeared in a disconnected environment.

What did you do?

One of preflight tests in a disconnected environment failed once it was using the new version where the default images are using tags. redhat-openshift-ecosystem/openshift-preflight#941 (comment)

What did you expect to see?

Expected that the default value of those two images remain using a digest instead of a tag so it will work in both disconnected and connected environments.

What did you see instead? Under which circumstances?

This is the output of the preflight tool that makes use of scorecard, this execution is using a disconnected cluster:

time="2023-04-07T12:42:56Z" level=trace msg="running operator-sdk scorecard check" image="quay.io/telcoci/simple-demo-operator-bundle@sha256:6cfbca9b14a51143cfc5d0d56494e7f26ad1cd3e662eedd2bcbebf207af59c86"
time="2023-04-07T12:42:56Z" level=trace msg="created temporary kubeconfig for use with scorecard at path" name=/tmp/83954228
time="2023-04-07T12:42:56Z" level=debug msg="user provided scorecard test image" image="quay.io/operator-framework/scorecard-test@sha256:f0034b3485c37665a740faf573e101b147afb25fd9550c50e286b31017851478"
time="2023-04-07T12:42:56Z" level=info msg="running scorecard with the following invocation" args="[\"operator-sdk\",\"scorecard\",\"--output\",\"json\",\"--selector=test=basic-check-spec-test\",\"--kubeconfig\",\"/tmp/83954228\",\"--wait-time\",\"240s\",\"--namespace\",\"preflight-testing\",\"--service-account\",\"default\",\"--config\",\"/tmp/scorecard-test-config-4282443852.yaml\",\"--verbose\",\"/tmp/preflight-3825930509/fs\"]"
time="2023-04-07T12:46:56Z" level=debug msg="operator-sdk scorecard failed to run properly"
time="2023-04-07T12:46:56Z" level=debug msg="stderr output" stderr="time=\"2023-04-07T12:42:56Z\" level=debug msg=\"Debug logging is set\"\nError: error running tests context deadline exceeded\nUsage:\n  operator-sdk scorecard [flags]\n\nFlags:\n  -c, --config string            path to scorecard config file\n  -h, --help                     help for scorecard\n      --kubeconfig string        kubeconfig path\n  -L, --list                     Option to enable listing which tests are run\n  -n, --namespace string         namespace to run the test images in\n  -o, --output string            Output format for results. Valid values: text, json, xunit (default \"text\")\n      --pod-security string      option to run scorecard with legacy pod security context (default \"legacy\")\n  -l, --selector string          label selector to determine which tests are run\n  -s, --service-account string   Service account to use for tests (default \"default\")\n  -x, --skip-cleanup             Disable resource cleanup after tests are run\n  -b, --storage-image string     Storage image to be used by the Scorecard pod (default \"quay.io/operator-framework/scorecard-storage:latest\")\n  -t, --test-output string       Test output directory. (default \"test-output\")\n  -u, --untar-image string       Untar image to be used by the Scorecard pod (default \"quay.io/operator-framework/scorecard-untar:latest\")\n  -w, --wait-time duration       seconds to wait for tests to complete. Example: 35s (default 30s)\n\nGlobal Flags:\n      --plugins strings   plugin keys to be used for this subcommand execution\n      --verbose           Enable verbose logging\n\ntime=\"2023-04-07T12:46:56Z\" level=fatal msg=\"error running tests context deadline exceeded\"\n"
time="2023-04-07T12:46:56Z" level=info msg="check completed" check=ScorecardBasicSpecCheck err="failed to run operator-sdk scorecard: exit status 1" result=ERROR
time="2023-04-07T12:46:56Z" level=debug msg="running check" check=ScorecardOlmSuiteCheck

Environment

Kubernetes cluster type:

• OpenShift 4.12.10
• operator-sdk 1.28.0

Possible Solution

Use digests in the default images to keep the support to disconnected environments.