Add loading credentials from docker cli config

[darkowlzz]

This adds a package pkg/runtime/auth with helpers to read the docker cli
config and load the credentials for the host name of a given image. This
is based on nerdctl's dockerconfigresolver, refer: https://github.com/containerd/nerdctl/blob/v0.8.1/pkg/imgutil/dockerconfigresolver/dockerconfigresolver.go#L102 .

When using containerd as the runtime, the credentials from docker cli
config is loaded into a containerd remote resolver and passed to the
containerd remote option used for pulling.

When using docker as the runtime, the credentials from docker cli config
is loaded into the docker image pull options in the required format. Based
on https://docs.docker.com/engine/api/sdk/examples/#pull-an-image-with-authentication .

When not logged in:

$ sudo ./bin/ignite image import darkowlzz/alpine:3.13
INFO[0000] containerd image "darkowlzz/alpine:3.13" not found locally, pulling...
FATA[0002] failed to resolve reference "docker.io/darkowlzz/alpine:3.13": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

After logging in:

$ sudo ./bin/ignite image import darkowlzz/alpine:3.13
INFO[0000] containerd image "darkowlzz/alpine:3.13" not found locally, pulling...
INFO[0003] Starting image import...
INFO[0003] Imported OCI image "darkowlzz/alpine:3.13" (5.7 MB) to base image with UID "f01694878f9ed2f0"
INFO[0003] Created image with ID "f01694878f9ed2f0" and name "darkowlzz/alpine:3.13"

With docker as runtime:

$ sudo ./bin/ignite image import darkowlzz/alpine:3.13 --runtime docker
INFO[0000] docker image "darkowlzz/alpine:3.13" not found locally, pulling...
INFO[0002] Starting image import...
INFO[0003] Imported OCI image "darkowlzz/alpine:3.13" (5.4 MB) to base image with UID "3d630e9936b42849"
INFO[0003] Created image with ID "3d630e9936b42849" and name "darkowlzz/alpine:3.13"

TODO:

• Maybe add flag to the image subcommand to pass the path to docker cli config.
• Maybe allow setting the docker cli config path in the ignite configuration file.
• Update docs.
• Set up automated tests.

Fixes #794

[talhazubairbutt]

@darkowlzz thanks a bunch for working on this. I have tested this with Google Container registry, currently it will only work if I install gcloud on the machine and gcloud is authenticated with an account that has access to the registry:

root@ed-live-vm-g1-small-624d50b5-fab9-4341-98b0-0ad1500fcb01:/home/talha_zubair/ignite-registry-login/bin# ./ignite image import gcr.io/educative-
dev-1-exec-env/author-1-collection-6108886603923456-assessment-6346381115523072-rev-2-container-6627856092233728-docker
INFO[0000] docker image "gcr.io/educative-dev-1-exec-env/author-1-collection-6108886603923456-assessment-6346381115523072-rev-2-container-662785609
2233728-docker:latest" not found locally, pulling... 
WARNING: Could not open the configuration file: [/root/.config/gcloud/configurations/config_default].
ERRO[0000] cannot get auth config for authConfigHostname="gcr.io" (refHostname="gcr.io"): error getting credentials - err: exit status 1, out: `You
 do not currently have an active account selected. See https://cloud.google.com/sdk/docs/authorizing for more information.` 
FATA[0000] Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid cred
entials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication 

However, docker allows to directly import credentials rather than relying on gcloud. This is done via docker login command and the daemon stores credentials at $HOME/.docker/config.json

To authenticate docker i just use the following command:
docker login -u _json_key -p "$(cat keyfile.json)" https://gcr.io

Here is how the config.json looks like:

{
        "auths": {
                "gcr.io": {
                        "auth": "{{TOKEN HERE}}"
                }
        },
        "credHelpers": {
                "asia.gcr.io": "gcloud",
                "eu.gcr.io": "gcloud",
                "gcr.io": "gcloud",
                "marketplace.gcr.io": "gcloud",
                "staging-k8s.gcr.io": "gcloud",
                "us.gcr.io": "gcloud"
        }
}

I did install gcloud for testing and logged into my service account and was able to pull successfully:

./ignite image import gcr.io/educative-dev-1-e
xec-env/author-1-collection-6108886603923456-assessment-6346381115523072-rev-2-container-6627856092233728-docker
INFO[0000] docker image "gcr.io/educative-dev-1-exec-env/author-1-collection-6108886603923456-assessment-6346381115523072-rev-2-container-6627856092233728
-docker:latest" not found locally, pulling... 
INFO[0005] Starting image import...                     
INFO[0008] Imported OCI image "gcr.io/educative-dev-1-exec-env/author-1-collection-6108886603923456-assessment-6346381115523072-rev-2-container-6627856092
233728-docker:latest" (157.5 MB) to base image with UID "32838093c103f269" 
INFO[0008] Created image with ID "32838093c103f269" and name "gcr.io/educative-dev-1-exec-env/author-1-collection-6108886603923456-assessment-634638111552
3072-rev-2-container-6627856092233728-docker:latest" 

It would be best if we could authenticate ignite similar to docker without the need of gcloud installation.

[darkowlzz]

@talhazubairbutt Thanks a lot of testing this.
To verify if gcloud cli tool is really required, I tried to use it in a machine without gcloud installed. I followed the docs from https://cloud.google.com/container-registry/docs/advanced-authentication#json-key, same as what you did with the keyfile.json file.
After doing docker login, I was able to use ignite to import the image.

docker config:

{
        "auths": {
                "gcr.io": {
                        "auth": "<token>"
                }
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/19.03.6 (linux)"
        }
}

I'm got curious about why you have the following error:

WARNING: Could not open the configuration file: [/root/.config/gcloud/configurations/config_default].
ERRO[0000] cannot get auth config for authConfigHostname="gcr.io" (refHostname="gcr.io"): error getting credentials - err: exit status 1, out: `You
 do not currently have an active account selected. See https://cloud.google.com/sdk/docs/authorizing for more information.` 

After some investigation, I found out that such GCP specific errors are due to the cred helpers in your docker cli config. When you have any cred helpers, the docker cli config loader will try to invoke the cred helper to get the auth token, refer https://github.com/docker/cli/blob/v20.10.6/cli/config/configfile/file.go#L326.
So, if you remove the cred helpers section from your config, you wouldn't see any gcloud related errors, just generic unauthorized error.

[talhazubairbutt]

@darkowlzz removing the credential helper from the config section indeed fixed the error and I was able to pull without authenticating gcloud. Can probably document this behaviour for other users as the error itself is a bit misleading. Docker on the other hand somehow worked even with the credential helper added, maybe docker tries all methods before throwing an error or has a different evaluation sequence.

[darkowlzz]

@talhazubairbutt Thanks for verifying. I'll add some docs with a note about credential helpers. 🙂

[stealthybox]

Maybe add flag to the image subcommand to pass the path to docker cli config.
Maybe allow setting the docker cli config path in the ignite configuration file.

+1 to both in future follow-up

Set up automated tests.

discussed test plan:

We want an automated test for success/fail cases importing an image.
It would be good to setup a just-in-time local registry /w any necessary PKI and OS CA bundle updates.
We could also create a trivial shell-script that can be used as a credential-helper test.

Using an actual external registry would incur secrets-management and rate-limiting and third-party reliability concerns