Push to JFrog fails when using --reproducible in v1.8.0

[meranos]

Actual behavior
With the introduction of version v1.8.0 we started to experience issues with our kaniko based builds which seem to be linked to the flag --reproducible:

INFO[0003] Pushing image to jfrog-registry:12345/testcontainer:1.2.3 
error pushing image: failed to push to destination jfrog-registry:12345/testcontainer:1.2.3: PUT https://jfrog-registry:12345/v2/testcontainer/manifests/1.2.3: MANIFEST_INVALID: manifest invalid; map[description:Circuit Breaker Threshold Reached, Breaking Operation. see log output for manifest details.]

When omitting --reproducible the image ends up being pushed without further issues.
Build & push using docker cli works normally.
With a registry:2 as destination the push also works as expected.

Expected behavior
Pushing the built image to registry works like before v1.8.0.

To Reproduce

docker run \
    -v $PWD/container:/workspace \
    gcr.io/kaniko-project/executor:debug \
    --dockerfile /workspace/Dockerfile \
    --destination jfrog-registry:12345/testcontainer:1.2.3 \
    --context dir:///workspace/ \
    --skip-tls-verify \
    --reproducible
...
INFO[0003] Pushing image to jfrog-registry:12345/testcontainer:1.2.3 
error pushing image: failed to push to destination jfrog-registry:12345/testcontainer:1.2.3: PUT https://jfrog-registry:12345/v2/testcontainer/manifests/1.2.3: MANIFEST_INVALID: manifest invalid; map[description:Circuit Breaker Threshold Reached, Breaking Operation. see log output for manifest details.]

Additional Information

• Dockerfile

FROM alpine
ADD random_stuff.sh /random_stuff.sh

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[bbhagat4]

Facing the same issue.

[MightyDjinn]

This is also the case when not using the "--reproducible" flag when using Artifactory.

[ramonamis]

I can reproduce the same, as soon as the --reproducible flag is not set anymore, I can push to artifactory.

[joaosilva15]

I think I found the root cause(?) and it seems to be similar to GoogleContainerTools/jib#534
I have a simple docker file like

FROM public.ecr.aws/docker/library/alpine:latest
RUN apk add --no-cache git

The reproducible tar for this image would produce a sha256:0ba624e5aee2dbaf34811d62295942438f1dd69fabb345bc1286120ba21e53e6 like

{
  "architecture": "amd64",
  "created": "0001-01-01T00:00:00Z",
  "history": [
    {
      "created": "0001-01-01T00:00:00Z",
      "created_by": "/bin/sh -c #(nop) ADD file:2a949686d9886ac7c10582a6c29116fd29d3077d02755e87e111870d63607725 in / "
    },
    {
      "created": "0001-01-01T00:00:00Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
      "empty_layer": true
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:d0acfd6b536fb10ce96f1bfeb347194baf5c9dc7f307a078af0781c45f40206d",
      "sha256:e09f1ec70c2b627e9135876b9f173f7190ce50d40a8a296199ec86cb9a29c54b"
    ]
  },
  "config": {
    "Cmd": [
      "/bin/sh"
    ],
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Image": "sha256:c0261ca8a4a79627f3e658c0c2b1e3166f56713a58e1411b1e3ab1e378962e75"
  }
}

while an image created without the flag would also have in the history field the entry

    {
      "author": "kaniko",
      "created": "0001-01-01T00:00:00Z",
      "created_by": "RUN apk add --no-cache git"
    }

Adding the entry to the file and pushing the image again artifactory accepts it.

As a side note, the following docker file works with the reproducible flag:

FROM public.ecr.aws/docker/library/alpine:latest

Edit:

It seems like the history always loses the last element when using the reproducible flag

[rseleven]

With version v1.9.1 the problem persists.

[rseleven]

With version v1.6.0 the flag: "--reproducible", works.

[Cagafuego]

It looks like this issue was fixed with the release of 1.9.2