-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Image built with Kaniko claims to be OCI but in reality is not #1836
Comments
Further investigations... This seems to happen regardless of the intermediate image being Multistage/Single stage. So to trigger the bug, all is needed is to build an intermediate image with Kaniko that has the following Dockerfile: > cat Containerfile
FROM $BASE_IMAGE_WITH_MEDIA_TYPE_OCI
RUN mkdir /path
RUN touch /path/file.txt And then > skopeo inspect --raw docker://{IMAGE} | jq .
{
"config": {
...
"mediaType": "application/vnd.oci.image.config.v1+json",
...
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 2896369,
"digest": "sha256:3c4e9198e8c15669838fa75b9fde03039cc4a256d6868d214d966bd8f27b093d"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 209,
"digest": "sha256:d95bec0a2faf35ce091c8b575e61cd11e955f0d3a32444d9f55b3b49972ad6ab"
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 134,
"digest": "sha256:2b75492cbc3ed2684c79591ef48f2421f15ad9244d1bb197b41287f1e7edac12"
}
],
"annotations": {
...
} |
Really nice investigation! We have a similar issue, when it comes to import images build to Kaniko into Harbor. One image can be imported, another one does not. Images are build on GitLab CI, published to GitLab registry and can be used without issue by podman 3. We find kaniko a very efficient way of building images and it would be great if we could keep using it. |
Thanks for investigating! We have images built with Kaniko that can be run by Podman 3.3.1 on CentOS Stream 8 but not pushed to another registry. The error message is slightly different, however it shows it can't handle the format. $ skopeo inspect --raw docker://<kaniko built image> | jq .
{
"schemaVersion": 2,
"mediaType": "",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 2229,
"digest": "sha256:1d9384a1e8cf5636c4c525b02d3eb2c5e2a6300987717cf00b798db54aabd955"
},
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"size": 79848180,
"digest": "sha256:78846bc60c09f099fe07532fd402aff11d9704b96310c7bcf0a7ee20085774a1"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 334659526,
"digest": "sha256:e8430b5e4f32476399aedbebd2bce8e31406d9f53e61bce5149b0f77fc3e11df"
},
... When trying to push:
|
Full disclosure, me, @LajosCseppento and @remivoirin all work in the same organization, but on different projects and teams, it just happens that we found the same issue in similar timeframes. |
@crisbal For us this is present at least since July 2021, when Remi & co. tried to import our image to Harbor. |
Podman was rejecting the image built with Kaniko because of the mix of "oci" and "docker" layers inside a general "oci" image (declared mediaType). |
Have there been any new discoveries on this issue? We just started running into this same issue; trying to add layers to an OCI image with Kaniko makes the image unusable by docker and podman. |
@mn132 what we ended up doing on our side was "walking around the problemi" by publishing/republishing the images we need in non-OCI format. If you are pushing OCI images with podman look into the --format option for |
I have the same issue with
|
FWIW c/image ≥ 5.22.0 is rather more tolerant of unknown MIME types. That should show up in Podman soon, currently there is https://github.com/containers/podman/releases/tag/v4.2.0-rc3 . I’m afraid I can’t now spare the time to test the full scenario, to see if Podman can fully consume (or possibly even correct) these images, or whether that avoiding this failure just runs into another problem soon after. |
With the Ubuntu 22.04 images now in OCI format, this has become a pressing issue for us. I found a workaround: use Same approach appears to work with Podman. |
Is this getting any traction? Otherwise we're going to have to move away from kaniko |
We probably have the same issue as in #2392 |
Side note: This affects usage of https://docs.snyk.io/integrations/ci-cd-integrations/snyk-ci-cd-integration-deployment-and-strategies/snyk-container-specific-ci-cd-strategies#running-pipeline-if-a-docker-daemon-is-not-available
|
from my poking around it looks like gcr's "tarball.LayerFromFile" is being used to convert the snapshots taken at every RUN to the layers: Line 520 in 61312a9
this method is "deprecated" in that lib but apparently is still there. there was an abandoned PR to switch to its preferred replacement: #449 anyways, it seems like what's needed here is for kaniko to pass in WithMediaType => types.OCILayer whenever its working FROM a known-OCI image, because the default is types.DockerLayer: I just pinned all my Dockerfiles' references to ubuntu:* images back to the -20221130 tags because that's the last one published in the docker format and still works properly with both Kaniko and RedHat tools (podman, buildah, quay, etc)
|
quick and dirrty fix for GoogleContainerTools#1836 ?
quick and dirrty fix for GoogleContainerTools#1836 ?
quick and dirrty fix for GoogleContainerTools#1836 ?
We encountered the same problem. Is there any progress on this? |
@kfix I tried your branch (https://github.com/kfix/kaniko/tree/fix_mismatched_oci_layers), but I get the same error message, when pulling the image with docker.
|
@michaelkebe I think you're having a different problem there? I could not reproduce that with Docker 20.10.21. I managed to get the integration tests running for my tweak and confirmed a before-and-after difference with
I was also able to pull the kaniko-built images from the integration-tests' repository into both Docker Engine and Podman 3.4. I could even run them with the mismatched layers but its only when trying to use the
|
I say "tests running" because they appear to be a bit busted. for OCI specifically by GoogleContainerTools/container-diff#390, fixed as a part of #2425 I'm hoping. I think I'm going to have to wait on that stuff before attempting to PR this. |
As soon as GoogleContainerTools/container-diff#390 is merged, I'll change #2425 to use that commit and have green tests again, if you want to test in the meantime you can rebase on top of #2425, install the fixed container-diff and run the integration tests locally or in the CI of your fork (again, note that currently #2425 still loads the old broken container-diff, you'd need to amend that in hte make file until that fix is merged) |
CI fix has been merged, although in the end the container-diff fix was not part of it, since it seems unmaintained. You will have to find a better way to test OCI images (directly using google/go-container-registry might allow you to write much tighter unit tests instead) |
@kfix Was there any progress on this now that the Tests are somewhat back working? |
Same issue here. I'm trying to build an image to use in Fedora Silverblue. rpm-ostree expects a purely OCI image. |
Still a problem, some progress on this issue would be greatly appreciated :) |
@kfix By chance do you have time to try and walk this over the finish line now that tests are back up again? |
@BronzeDeer Yes, I've wrapped up my house move and its very hot outside so maybe I'll chill and attempt to write test code in Go |
Hey guys, where did this ultimately end up? |
Just encountered the same problem. Any progress or workaround would be appreciated. 🙏 |
Also following. Hoping for a fix soon 🤞 |
Thanks @loganprice ! |
Actual behavior
Coming from containers/buildah#3668
I am using kaniko to build an image based on an OCI-image.
The base image has the following manifest: ( notice
mediaType: application/vnd.oci.image.layer.v1.tar+gzip
)>>> skopeo inspect --raw docker://${BASE_IMAGE} | jq .
If I use this image as a base for another image built with Kaniko, I get the following resulting image:
Which claims to be
application/vnd.oci.image.config.v1+json
but indeed has Dockerapplication/vnd.docker.image.rootfs.diff.tar.gzip
layers.This shows as an error when going to use the child image as a base image in
podman build
, which shows (again see issue containers/buildah#3668):Expected behavior
As @vrothberg suggests, the layers should be converted to OCI ones during build or when pushing to the registry.
To Reproduce
Steps to reproduce the behavior:
"mediaType": "application/vnd.oci.image.config.v1+json",
Additional Information
Unfortunately it is quite difficult to find a public image that has
"mediaType": "application/vnd.oci.image.config.v1+json"
, but to build one the following can be achieved withpodman
:Dockerfile for child image:
Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
Triage Notes for the Maintainers
--cache
flagThe text was updated successfully, but these errors were encountered: