Building images with podman-remote seems to be broken

[afbjorklund]

Running into various issue when trying to use "minikube podman-env", with podman-remote (version 1.9.3)

Not sure if it was working previously, but seems to have some out-of-the-box issues now. Workaround is ssh.

Error with minikube:

DEBU[0011] successfully received /var/tmp/varlink_send722062984 
DEBU[0011] created new context dir at /var/tmp/buildTarball973636679 
DEBU[0011] untar of /var/tmp/varlink_send722062984 successful 
Error: error reading info about "/var/tmp/buildTarball973636679/var/tmp/buildTarball973636679/Dockerfile": stat /var/tmp/buildTarball973636679/var/tmp/buildTarball973636679/Dockerfile: no such file or directory

But runs fine locally:

DEBU[0005] successfully received /var/tmp/varlink_send247430795 
DEBU[0005] created new context dir at /var/tmp/buildTarball962538094 
DEBU[0005] untar of /var/tmp/varlink_send247430795 successful 
DEBU[0005] reading local Dockerfile "/var/tmp/buildTarball962538094/Dockerfile" 

Also seen some new kind of network issues, with the CNI networking:

error running container: error configuring network list if1 for [/bin/sh -c true]: missing network name:

It also seems that the official binaries for darwin are broken (not remote).

https://github.com/containers/podman/releases/tag/v1.9.3

Reproducer:

minikube start --container-runtime=cri-o
eval $(minikube podman-env)
podman1-remote build

https://minikube.sigs.k8s.io/docs/handbook/pushing/#3-pushing-directly-to-in-cluster-cri-o-podman-env

[gbraad]

Can you verify this on a Fedora install?

…

On Sun, Dec 6, 2020, 21:47 Anders Björklund ***@***.***> wrote: Running into various issue when trying to use "minikube podman-env", with podman-remote (version 1.9.3) Not sure if it was working previously, but seems to have some out-of-the-box issues now. Workaround is ssh. Error with minikube: DEBU[0011] successfully received /var/tmp/varlink_send722062984 DEBU[0011] created new context dir at /var/tmp/buildTarball973636679 DEBU[0011] untar of /var/tmp/varlink_send722062984 successful Error: error reading info about "/var/tmp/buildTarball973636679/var/tmp/buildTarball973636679/Dockerfile": stat /var/tmp/buildTarball973636679/var/tmp/buildTarball973636679/Dockerfile: no such file or directory But runs fine locally: DEBU[0005] successfully received /var/tmp/varlink_send247430795 DEBU[0005] created new context dir at /var/tmp/buildTarball962538094 DEBU[0005] untar of /var/tmp/varlink_send247430795 successful DEBU[0005] reading local Dockerfile "/var/tmp/buildTarball962538094/Dockerfile" Also seen some new kind of network issues, with the CNI networking: error running container: error configuring network list if1 for [/bin/sh -c true]: missing network name: It also seems that the official binaries for darwin are broken (not remote). https://github.com/containers/podman/releases/tag/v1.9.3 Reproducer: minikube start --container-runtime=cri-o eval $(minikube podman-env) podman1-remote build — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9870>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAOZSTH7PITWDZALDLDYDSTODNXANCNFSM4UPKR6FA> .

[afbjorklund]

Can you verify this on a Fedora install?

Not sure what a Fedora install is, there are no client binaries for Fedora and there is no support for it in the ISO or the KIC...

So you would have to use the "static" (linux) binary for podman-remote, from the releases:

https://github.com/containers/podman/releases/download/v1.9.3/podman-remote-static.tar.gz

But the problem itself should be reproducable (reports appreciated), about trying to use podman-env and podman-remote ?

Can also use the local method, but that was the one with some issues about reproducing...

export PODMAN_VARLINK_BRIDGE="sudo varlink -A 'podman varlink \$VARLINK_ADDRESS' bridge"

---

We will see next year, but probably have to wait for Podman v3 - as there's still issues with Podman v2.

Using the ssh makes it slightly inconvenient with the build context, but should work for the time being...

tar cf - Dockerfile | minikube ssh --native-ssh=false sudo podman build -

When working with a remote Fedora VM (with the "generic" driver), it should work OK - haven't tested.

See #9638

[gbraad]

I mean, try this out on a Fedora 32 or 33 release to see if this issue is reproducible outside of the Minikube environment (preferably with the same versions). I do not remember seeing this. Since this information is helpful to determine where the issue is. For instance, our CRC images use Podman 1.9.3 and I do not recall seeing this, which makes this Minikube specific. Though, this has been a while back since we ran this.

…

On Mon, Dec 7, 2020 at 2:46 PM Anders Björklund ***@***.***> wrote: Can you verify this on a Fedora install? Not sure what a Fedora install *is*, there are no client binaries for Fedora and there is no support for it in the ISO or the KIC... https://github.com/containers/podman/releases/download/v1.9.3/podman-remote-static.tar.gz But the problem itself should be reproducable (reports appreciated), about trying to use podman-env and podman-remote ? Can also use the local method, but that was the one with some issues about reproducing... ------------------------------ export PODMAN_VARLINK_BRIDGE="sudo varlink -A 'podman varlink $VARLINK_ADDRESS' bridge" We will see next year, but probably have to wait for Podman v3 - as there's *still* issues with Podman v2. Using the ssh makes it slightly inconvenient with the build context, but should work for the time being... tar cf - Dockerfile | minikube ssh --native-ssh=false sudo podman build - — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9870 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAOZXL65E7WQWV7TUZ6ADSTR24RANCNFSM4UPKR6FA> .

-- Gerard Braad | http://gbraad.nl STEM is the source, but BUILD is how you develop! [ Better Understanding Involves Learning and Doing ]

[afbjorklund]

Would need your help with testing on Fedora (or CentOS, or CRC), don't have such an environment (#3552)
I have been using Vagrant for doing some local testing, but it's not really wrking for doing nested virtualization

Supposedly it was working for previous releases, but I don't think we have too much regression testing on it...
But there is no maintenance track for podman v1 (outside of RHEL), so we would have to move to podman v2.

We are going to move to containerd and buildkitd as the primary CRI (replacing the current default of docker),
but it "would be nice" to have a working cri-o and podman alternative - even though it is very expensive to do.

As soon as it works reasonably, I think we will upgrade from 1.9.3 to 2.2.1 (or 2.2.2) - mostly for the software.
Don't really want to maintain our own podman packages and supply our own podman clients, just for varlink.

• ISO: Upgrade podman to version 2.2.0 and remove varlink #9635
• KIC: Upgrade podman to version 2.2.0 and remove varlink #9636

Basically the requirement is:

• podman load

• podman build

And to ssh as user (not root)

[afbjorklund]

For instance, our CRC images use Podman 1.9.3 and I do not recall seeing this, which makes this Minikube specific. Though, this has been a while back since we ran this.

I thought that CRC didn't support podman-remote (and the podman service) yet, but maybe I am mistaken.

This issue is specially about orchestrating the build from the remote laptop, not running podman locally on VM.

We are also trying to provide the minikube VM as an alternative to docker-machine and podman-machine.
But it's a secondary priority (kubernetes is more important), and it seems to silently break from time-to-time too.

Some users prefer being able to use a single VM for both Docker/Podman and for Kubernetes, instead of two.
Then again other users prefer to run their minikube inside the Docker VM, using docker-in-docker for the nodes.

[gbraad]

I thought that CRC didn't support podman-remote (and the podman service

We had (have). We had an issue with the client on Windows. They weren't available in time to continue the trouble ...

so we would have to move to podman v2.

... as we were also waiting for this to align. Currently available on FCOS, so we have started testing for exposing this... just waiting for the GA as part of our image base. Perhaps we can assist with this? So, the question if it is worth spending time on V1 now.

…

On Mon, Dec 7, 2020 at 6:04 PM Anders Björklund ***@***.***> wrote: For instance, our CRC images use Podman 1.9.3 and I do not recall seeing this, which makes this Minikube specific. Though, this has been a while back since we ran this. I thought that CRC didn't support podman-remote (and the podman service) yet, but maybe I am mistaken. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#9870 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAOZQBM7SOZ4KXQWZXPDTSTSSDZANCNFSM4UPKR6FA> .

-- Gerard Braad | http://gbraad.nl STEM is the source, but BUILD is how you develop! [ Better Understanding Involves Learning and Doing ]

[afbjorklund]

Perhaps we can assist with this? So, the question if it is worth spending time on V1 now.

We buried varlink thoroughly (6') on the previous Podman Community meeting.
And then there was a shared sign of relief about this, again on the last meeting.

https://podman.io/community/meeting/

Supporting Mac users (podman-machine) or Kubernetes users (minikube) is not a priority.

So generally they are moving back to Vagrant, when it comes to providing Podman VMs.

---

All the code has been deleted from the master podman branch (3.0.0-dev) already.
And none of the podman packages (ubuntu/fedora) have it enabled, even for 2.2

So we will remove it, and use SSH: #9230

I think only RHEL still uses podman v1 ?

[afbjorklund]

Minikube now uses podman v2, and it seems like image building is working again.

$ podman-remote version
Client:
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Server:
Version:      2.2.1
API Version:  2.0.0
Go Version:   go1.13.15
Git Commit:   a0d478edea7f775b7ce32f8eb1a01e75374486cb
Built:        Fri Dec 11 10:17:24 2020
OS/Arch:      linux/amd64

Now using CONTAINER_HOST, rather than the old PODMAN_VARLINK_BRIDGE

export CONTAINER_HOST="ssh://docker@127.0.0.1:43215/run/podman/podman.sock"
export CONTAINER_SSHKEY="/home/anders/.minikube/machines/minikube/id_rsa"