question: does crane copy support multiarch image built with docker buildx?

[ericbl]

I would like to use crane to make the deployment to different registries easier.
I have one gitlab repo, with one pipeline building a multiarch build to the Gitlab registry.

docker buildx build --platform "linux/amd64,linux/arm64" --build-arg REGISTRY=$TARGET_REGISTRY -t $TARGET_REGISTRY_WITH_NAMESPACE/$IMAGE_NAME:$TARGET_IMAGE_TAG --push .

and then, I would like to push this multarch image to AWS ECR registry (actually a job used twice with 2 different target registries).

- crane auth login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- aws ecr get-login-password --region $AWS_REGION | crane auth login --username AWS --password-stdin $TARGET_REGISTRY
- crane cp $CI_REGISTRY_IMAGE/$IMAGE_NAME:$IMAGE_TAG $TARGET_REGISTRY_WITH_NAMESPACE/$IMAGE_NAME:$TARGET_IMAGE_TAG

the auth to the 2 registries (gitlab and aws) seems to be ok, but the crane cp always results with a 400:

2022/03/15 15:28:40 failed to copy index: HEAD <aws-registry>/v2/<aws-namespace>/manifests/sha256:fwed95...: unsupported status code 400

Any idea why this 400? Could be more a question for AWS support.

But at the first place, it is supposed to work ? Does crane cp support copying a multi arch image?

[imjasonh]

crane definitely supports copying multiplatform images.

Can you share more about your manifest? (crane manifest $CI_REGISTRY_IMAGE/$IMAGE_NAME:$IMAGE_TAG) That might help identify what's causing ECR to reject it.

You could also narrow down the issue with crane cp $src $dst --platform=linux/amd64 and then again with --platform=linux/arm64. This won't copy the multiplatform image, but if copying both single-platform images works (or doesn't) that tells us something about the problem.

This might be an issue on ECR's side, but there's stuff we can try to find out.

[imjasonh]

Hmm I just noticed it's getting the 400 on a HEAD request -- does ECR not support HEAD requests?

[ericbl]

thanks Jason for your help.
crane manifest cmd gives nothing unexpected I think:

{
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "schemaVersion": 2,
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:39f60801785f3ea3ef242c628d0a268c834e7575b974d318b2abb057ab32d251",
         "size": 3679,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "digest": "sha256:103659be961702c[38](<gitlab-job>)658edf4caa4bb051650635a6918cb02bb22cc71e6425c54",
         "size": 3679,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

$ crane cp --platform=linux/amd64 $CI_REGISTRY_IMAGE/$IMAGE_NAME:$IMAGE_TAG $TARGET_REGISTRY_WITH_NAMESPACE/$IMAGE_NAME:$TARGET_IMAGE_TAG
2022/03/16 11:59:00 Copying from <CI_REGISTRY_IMAGE>/<image-name> to <TARGET_REGISTRY_WITH_NAMESPACE><image-name>:crane-test
2022/03/16 11:59:01 failed to copy image: HEAD <aws-registry>/v2/<aws-namespace>/<image-name>/blobs/sha256:1c9a8b42b5780ac49c71f[39]()0 Bad Request (HEAD responses have no body, use GET for details)

I actually don't understand why AWS triggers a HEAD request here.

[ericbl]

I am using crane copy to push to AWS ECR in another project successfully, but with images built with kaniko, and using IAM_role from AWS.
here, neither Kaniko nor IAM_Role set.

[imjasonh]

I actually don't understand why AWS triggers a HEAD request here.

crane issues a HEAD request before pushing a manifest or blob, to see whether the registry already has it. The registry is supposed to respond with a 200 or 404 to let us know whether to proceed with a POST to send it up. I'm not sure why it's giving a 400 though -- in this latest example, it's a 400 on the blob HEAD, in your original report it was for a manifest. It's probably something affecting both paths. I suspect auth... 🤔

It's odd that you're using Kaniko, because Kaniko uses the same underlying registry client code in pkg/v1/remote to push, and it's successful. What version of Kaniko are you using?

Just in case it matters, could you make sure you're using the latest crane release, and maybe even try go install github.com/google/go-containerregistry/cmd/crane@main to get the absolute latest version, and see if that works.

[ericbl]

thanks for the explanation with HEAD.
In whilch conditions does the POST occurs? I suppose 404 OR 200 with different sha / different timestamp ?

The other project using Kaniko has different auth indeed.
I would like to build with Kaniko here too but it seems not possible to build multi arch image with Kaniko ( GoogleContainerTools/kaniko#786 )

Yes, the 400 on blob vs manifest is because on manifest occurs on a already existing image on the repo (pushed with docker buildx push) while here on the blob occurs on a different image name. The related aws repo should be already created, but I might add an extra step for that.

So I ll check again my auth here. I though I already fixed it since I used to have a 401 and 403 before,

Ok for getting the lastest crane.

[imjasonh]

In whilch conditions does the POST occurs? I suppose 404 OR 200 with different sha / different timestamp ?

If the response from the HEAD is 404, that means it doesn't currently have the manifest, so we POST it. If it's 200, the registry has the manifest and we can skip uploading it. The registry API is based on content-addressed storage, so if the manifest changes at all -- including new source inputs producing new layers, or non-deterministic build inputs like the current time being different from previous builds -- a new manifest will be produced, which the registry has never seen before, the HEAD will return 404, and the manifest will be POSTed.

The related aws repo should be already created, but I might add an extra step for that.

This might be the cause. Try crane cp --verbose to see if there's a better log message that's being swallowed along the way, that would help us debug that.

[ericbl]

as said earlier, I want to push to 2 different AWS ECR repo.

for the first one,
I use as said earlier
- aws ecr get-login-password --region $AWS_REGION | crane auth login --username AWS --password-stdin $TARGET_REGISTRY

The get-login-password should use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables set as gitlab variables.

for the 2nd one, I have a script that generates a AWS Token based on OAuth2 credentials
and eventually log with
crane auth login $TARGET_REGISTRY -u AWS -p $AWS_TOKEN

both solutions should be ok, isn't it?

edit: apparently not so, I indeed see a 401 with crane cp --verbose before the 400 mentioned above.

[imjasonh]

I don't personally know enough about ECR's auth to know whether one should work and not the other.

Which form of login produces the 401/400? Or do they both fail?

[ericbl]

Both should work, but in my case, none of them!
However I found a solution, via the /.docker/config.json
As said, for my 2nd AWS, I generate a TOKEN, and export it into the config file
echo "{"auths":{"$REGISTRY":{"auth":"$TOKEN"}}}" >~/.docker/config.json

I realized that crane is actually automatically/magically READING this file and use it for auth. Thus I don't need extra auth login.

I also see that it is probably quite new since an older version of crane was not doing so.

Please confirm whether reading .docker/config.json is an expected behavior.
And please enhance the documentation in
https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md
It would help many other user to have a about crane auth login and if expected, mention the /.docker/config.json as alternative.

[ericbl]

and for the first AWS, I still need to find the easiest way to use $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY with
crane auth login without creating an extra IAM_Role.
edit: it is actually working fine with
aws ecr get-login-password --region $AWS_REGION | crane auth login --username AWS --password-stdin $TARGET_REGISTRY
when I set the proper region!

[imjasonh]

crane auth login updates your ~/.docker/config.json to set the username and password you provide. It then uses the auth configured in that file to make requests to registries.

(docker login does the same thing, crane does it for compatibility with docker)

I'll try to make docs describe this interaction better. I'm pretty sure this has been crane's behavior for private registries forever though.

[ericbl]

Here, I did NOT use crane auth login to update config.json BUT did the opposite: I provided the ~/.docker/config.json file and saw crane reading it.

[schurteb]

I would like to build with Kaniko here too but it seems not possible to build multi arch image with Kaniko ( GoogleContainerTools/kaniko#786 )

In case you'd prefer to have the images built with kaniko:
You can compose multi arch manifests from kaniko's single arch builds using github.com/estesp/manifest-tool.

Using this in a job between my build & deploy stages enables me to copy the multi arch image in a single deploy job from gitlab to the appropriate ECR instance, even though kaniko itself builds single arch images only