Problems trying to load cached images into minikube #655
Comments
|
@afbjorklund Any chance we can get a reproducer without Minikube? It would be a lot easier if we could pull the image it's trying to import out so we can directly inspect it and try and figure out what's going on. Also, could you add |
|
Sure, will try to do that (and the ISO?) Also : pull + save + rmi + load works OK |
|
does it work outside minicube ? |
|
podman is not part of minikube. This example looks like he is prestaging the container with podman and then attempting to run a container with CRI-O. The problem is here. How did you create /tmp/busybox_latest |
|
@umohnani8 PTAL |
|
I think it was created by Docker in the client originally, and then copied from the cache... I have kubeadm running OK with crio/crictl, so trying to replace the |
|
The busybox was just an example, the main containers are all cached from kubernetes |
|
@rhatdan : minikube 0.26 still had kpod |
|
|
|
Looks like Most probably something with the |
|
Here is the image, had to gzip it to attach it: busybox_latest.gz |
|
Here is the image caching code: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/machine/cache_images.go#L291 I think it is using this library to do it: |
|
@afbjorklund Thanks, that helps a lot |
|
Here is the entire log, with debugging enabled: _ _
_ _ ( ) ( )
___ ___ (_) ___ (_)| |/') _ _ | |_ __
/' _ ` _ `\| |/' _ `\| || , < ( ) ( )| '_`\ /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )( ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)
$ sudo podman --log-level=debug load -i /tmp/busybox_latest
DEBU[0000] overlay test mount with multiple lowers succeeded
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true
INFO[0000] [graphdriver] using prior storage driver: overlay
INFO[0000] CNI network rkt.kubernetes.io (type=bridge) is used from /etc/cni/net.d/k8s.conf
INFO[0000] Initial CNI setting succeeded
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/library/busybox:latest"
DEBU[0000] IsRunningImageAllowed for image docker-archive:
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0000] ... will first try using the original manifest unmodified
Copying blob sha256:4febd3792a1fb2153108b4fa50161c6ee5e3d16aa483a63215f936a113a88e9a
DEBU[0000] Detected compression format gzip
DEBU[0000] No compression detected
0 B / 706.12 KB [-------------------------------------------------------------]DEBU[0000] Using original blob without modification
1.30 MB / 706.12 KB [======================================================] 0s
Failed
DEBU[0000] parsed reference to refname into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]docker.io/tmp/busybox_latest:latest"
Failed
ERRO[0000] error pulling "dir:/tmp/busybox_latest": unable to pull dir:/tmp/busybox_latest
$ |
|
Here is the actual list of containers, from the minikube cache: These are all copied over to All of them seem to fail silently, and then it pulls them all again... |
|
@umohnani8 Can you give this another look now that we have an image to test with? |
|
@mheon working on it. |
|
Thanks! |
|
The actual error is "Error writing blob: blob size mismatch", but it only writes "Failed". Seems like it is not expecting a mix of uncompressed and compressed blobs perhaps ? Also seen by the progressbar overflowing while copying, |
|
This sounds like it could be a c/image bug |
|
@mtrmac PTAL |
|
For starters, https://github.com/projectatomic/libpod/blob/27107fdac1d75f97caab47cd13efb1d9900cf350/libpod/image/image.go#L141 is swallowing the underlying error report. The error is, per |
|
So that's where our debug was going... Nice catch @mtrmac |
|
First of all, please rebase the minikube version of c/image to something fresh, maybe (Note: You may want to add |
|
… the flip side, though, is that That can’t work, and is mostly fixed (a bit stalled?) in containers/image#427 . |
Hmm, this is starting to have more far-reaching implications than a simple |
:) IIRC fixing either the minikube or the podman side would be sufficient—but the podman-side fix is not quite available right now. |
|
Seems like minikube is aiming for switching to another library instead: kubernetes/minikube#2730 https://github.com/containers/image --> |
|
@mheon What is the current state of this? |
|
The c/image fix upstream seems to be stalled. |
|
And minikube still has kpod-b85d0fa (Dec 2017) |
|
Still blocked , but I have pinged @cyphar to see if the PR can get some movement. |
|
Seems to be working now (v0.7.4), thanks. |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
Description
Not being able to load images, not getting any error reports (only "failed, and dir:")
Steps to reproduce the issue:
CRI: try to use "sudo podman load" instead of "docker load" kubernetes/minikube#2757
make out/minikube out/minikube.iso./out/minikube --container-runtime=cri-o --iso-url=file://$PWD/out/minikube.isoDescribe the results you received:
Describe the results you expected:
Sucessfully being able to load all the images from cache, just like
docker loadAdditional information you deem important (e.g. issue happens only occasionally):
Seems to pull the images correctly from the network, just not from the load disk.
Output of
podman version:Output of
podman info:Additional environment details (AWS, VirtualBox, physical, etc.):
VirtualBox
The text was updated successfully, but these errors were encountered: