AWS ECR - Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2

[mantasmatuzas]

Environment:

• Jib version: 3.0.0
• Build tool: Gradle 6.7.1
• OS: Arch Linux (kernel: 5.11.15-arch1-2, 64-bit)

Description of the issue:
Docker image push to AWS ECR fails with Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 error. I'm using AWS ECR Credentials Helper as an authorization mechanism. adoptopenjdk:11-jdk Docker image used in environment where this plugin is used.

Expected behavior:
Plugin successfully pushes Docker image to AWS ECR.

Steps to reproduce:

1. Use 3.0.0 version of com.google.cloud.tools.jib
2. Export env vars: AWS_DEFAULT_REGION, AWS_ECR_REGISTRY_URL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
3. Install https://github.com/awslabs/amazon-ecr-credential-helper version 0.5.0-2
4. Build with: ./gradlew jib -Djib.to.image=${AWS_ECR_REGISTRY_URL}:latest -Djib.to.credHelper=ecr-login

jib-gradle-plugin Configuration:

jib {

}

Log output:

./gradlew jib -Djib.to.image=${AWS_ECR_REGISTRY_URL}:latest -Djib.to.credHelper=ecr-login

> Task :jib

Containerizing application to XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY...
Base image 'adoptopenjdk:11-jre' does not use a specific image digest - build may not be reproducible
Using credential helper docker-credential-ecr-login for XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY
The base image requires auth. Trying again for adoptopenjdk:11-jre...
Using base image with digest: sha256:944587c7efcc21dc419cb838422c23bc9c1748f75b8cc8e8cafdf1d5a555d16d

Container entrypoint set to [java, -cp, /app/resources:/app/classes:/app/libs/*, YYY]
Executing tasks:
[=========================     ] 83.3% complete
> launching layer pushers


> Task :jib FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push image manifest for XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY:latest but failed because: Registry may not support pushing OCI Manifest or Docker Image Manifest Version 2, Schema 2 | If this is a bug, please file an issue at https://github.com/GoogleContainerTools/jib/issues/new

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5s
3 actionable tasks: 1 executed, 2 up-to-date

[mpeddada1]

Hey @mantoshelis, apologies for the delay. This error is only thrown in one place in the Jib code and is due to the server returning MANIFEST_INVALID or TAG_INVALID. We've seen this occur in the past for some registries like Quay.io and Artifactory. I would suggest taking a look at #3201 (comment) and the related issues linked there. I'm tempted to say that #3201 is very similar to what you're experiencing.

[chanseokoh]

The difference from #3201 is that, the build here doesn't seem to be pushing multiple manifests. Otherwise, you must have seen all the tags in the log, but it says only one image (manifest):

Containerizing application to XXX.dkr.ecr.eu-north-1.amazonaws.com/YYY...

However, it's still true that the server is returning MANIFEST_INVALID or TAG_INVALID, and I think it's an issue in AWS ECR, maybe specific to your repository for some reason. (We have not heard any single case of Jib creating and pushing an invalid manifest for many years. And people are regularly pushing to ECR without an issue.)

[mantasmatuzas]

Thank you for the responses. Unfortunately, the problem persisted in AWS ECR registry. Tag immutability was toggled on.

[chanseokoh]

Tag mutability was toggled on.

Aha, thanks for the update. That explains it.

[chanseokoh]

The error is because Jib is pushing the exact same image again (i.e., ECR already has the image). With the tag immutability turned on, ECR won't allow re-uploading existing images.

Obviously, if you hit this issue, there's nothing you need to do; the image is already there. But for some reason you don't want to see Jib fail, one option could be -DskipExistingImages (e.g., ./gradlew jib -DskipExistingImages). Jib will do nothing for the images already existing in a target registry.

[chtpl]

The skipExistingImages option does not work for me :/

This is the profile I'm using

        <profile>
            <id>jib-prod</id>
            <activation>
                <property>
                    <name>jib-prod</name>
                    <value>true</value>
                </property>
            </activation>
            <dependencies>
                <!-- https://mvnrepository.com/artifact/com.google.cloud.tools/jib-maven-plugin -->
                <dependency>
                    <groupId>com.google.cloud.tools</groupId>
                    <artifactId>jib-maven-plugin</artifactId>
                    <version>3.0.0</version>
                </dependency>
            </dependencies>
            <build>
                <defaultGoal>compile test jib:build</defaultGoal>
                <plugins>
                    <plugin>
                        <groupId>com.google.cloud.tools</groupId>
                        <artifactId>jib-maven-plugin</artifactId>
                        <version>3.0.0</version>
                        <configuration>
                            <container>
                                <entrypoint>
                                    <arg>/bin/sh</arg>
                                    <arg>-c</arg>
                                    <arg>echo "PROFILE: $PROFILE"; exec java \
                                        -Dspring.profiles.active=$PROFILE \
                                        $JAVA_OPTS -cp /app/resources:/app/classes:/app/libs/* \
                                        com.company.MyApplication
                                    </arg>
                                </entrypoint>
                            </container>
                            <from>
                                <image>
                                    <ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com/baseimages:openjdk-8
                                </image>
                            </from>
                            <to>
                                <image>
                                    <ACCOUNTID>.dkr.ecr.eu-central-1.amazonaws.com/myapp:${project.version}
                                </image>
                            </to>
                        </configuration>
                    </plugin>
                </plugins>
            </build>
        </profile>

I use this command to run the build mvn -B -Djib-prod=true -Djib.skipExistingImages

[chanseokoh]

@chtpl for documentation purposes: -Djib.skipExistingImages skips pushing images that already exist in a registry (i.e., when trying to push the same image). The "same image" here means the image with the exact same contents, which has the same image digest (SHA). -Djib.skipExistingImages doesn't skip pushing a different image only because a registry already has the same image tag, so it doesn't work as you hoped in your case (#3275).