Podman Backend problem: executable file bash not found in $PATH

[JoshOrndorff]

I've been using zombienet pretty successfully with the native provider for a few weeks now.

I would now like to publish a zombienet file that works with the podman backend as well. This should make it easier for newcomers to try out my code without having to compile it locally. So I've created a Tuxedo Parachain docker image. And the docker image works fine when I use it manually.

However, when I try to use it with zombienet, the process crashes pretty early on. I believe the relevant error is executable file bash not found in $PATH. So I guess zombienet is assuming some things about the docker image that I am not satisfying? But I'm not clear what the assumptions are.

Here is my parachain dockerfile in case that helps.

And here is the complete output of the process.

$ zombienet -p podman spawn zombienet.toml 
╔════════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════╗
║ 🧟 Zombienet 🧟    │ Initiation                                                                                         ║
╟────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────╢
║ Provider           │ podman                                                                                             ║
╟────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────╢
║ Namespace          │ zombie-3efdc54f                                                                                    ║
╟────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────╢
║ Temp Dir           │ /tmp/zombie-3efdc54f_-126445-MIfnLEn1h0Jv                                                          ║
╚════════════════════╧════════════════════════════════════════════════════════════════════════════════════════════════════╝

	 Monitor: prometheus - url: http://127.0.0.1:38909

	 Monitor: tempo - url: http://127.0.0.1:43741

	 Monitor: grafana - url: http://127.0.0.1:39275
┌─────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Pod                     │ temp                                                                                               │
├─────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Status                  │ Launching                                                                                          │
├─────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Command                 │ bash -c ../polkadot-sdk/target/release/polkadot --help | grep ws-port                              │
└─────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Pod                     │ temp-1                                                                                             │
├─────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Status                  │ Launching                                                                                          │
├─────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Command                 │ bash -c ./target/release/parachain-template-node --help | grep ws-port                             │
└─────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘

 Error:  	 Error: Command failed with exit code 125: podman play kube --network zombie-3efdc54f /tmp/zombie-3efdc54f_-126445-MIfnLEn1h0Jv/temp-1.yaml
Trying to pull ghcr.io/off-narrative-labs/tuxedo-parachain:latest...
Getting image source signatures
Copying blob sha256:046f4b1aa876e438dc59296e8254bfef9db2010c7b5f7af380b019bd759149a1
Copying blob sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a
Copying blob sha256:5b0247f2b2519c67af3318b95572c2b0fe68e3b46c5a946c92e3d5339aaa0eaa
Copying config sha256:49e68c2ade5d360c32cf507bda70f136b7bda658873665a5b09ee531a4e10fc5
Writing manifest to image destination
Storing signatures
a container exists with the same name ("temp-1") as the pod in your YAML file; changing pod name to temp-1_pod
error starting container e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: failed to start 1 containers
[error starting container e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found]
Pod:
867c71897b1d7b77275300c99d3356dc634c50123a84830aa6d164dbead3a8f1
Container:
e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892



Error:  	 Error: Command failed with exit code 125: podman play kube --network zombie-3efdc54f /tmp/zombie-3efdc54f_-126445-MIfnLEn1h0Jv/temp-1.yaml
Trying to pull ghcr.io/off-narrative-labs/tuxedo-parachain:latest...
Getting image source signatures
Copying blob sha256:046f4b1aa876e438dc59296e8254bfef9db2010c7b5f7af380b019bd759149a1
Copying blob sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a
Copying blob sha256:5b0247f2b2519c67af3318b95572c2b0fe68e3b46c5a946c92e3d5339aaa0eaa
Copying config sha256:49e68c2ade5d360c32cf507bda70f136b7bda658873665a5b09ee531a4e10fc5
Writing manifest to image destination
Storing signatures
a container exists with the same name ("temp-1") as the pod in your YAML file; changing pod name to temp-1_pod
error starting container e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found
Error: failed to start 1 containers
[error starting container e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892: executable file `bash` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found]
Pod:
867c71897b1d7b77275300c99d3356dc634c50123a84830aa6d164dbead3a8f1
Container:
e830ca27769476d68ca9793ae707005ae7ce62f3d0395e52c5c9046e7ce94892



undefined
┌────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Node'              │ /tmp/zombie-3efdc54f_-126445-MIfnLEn1h0Jv/logs                                                     │
│ s logs:            │                                                                                                    │
│ m                  │                                                                                                    │
└────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘

[pepoviola]

Hey @JoshOrndorff, thanks for the feedback! We made the assumption of bash is present and in the $PATH.
I just tested with the latest version published and looks like your last commit fix it the image.

 podman run -it --entrypoint /bin/sh  ghcr.io/off-narrative-labs/tuxedo-parachain:para-docker-6
Trying to pull ghcr.io/off-narrative-labs/tuxedo-parachain:para-docker-6...
Getting image source signatures
Copying blob cb6328cb7f20 done
Copying blob 7a2c55901189 done
Copying blob b6fc82b25c1b done
Copying config aabc48693e done
Writing manifest to image destination
Storing signatures
$ bash
node-dev@7c490839c68a:/$ exit
exit
$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Let me know if this doesn't fix it and I can check again.

Thanks!!

[JoshOrndorff]

Indeed, bash was the assumption. I wanted to track it down a little further before posting.

That line I removed is based on this line in the example Substrate dockerfile. This line also used to appear in the node template repo, but it is no longer there.

Anyway, it is fixed if we just leave some standard utilities in the image. I'm happy to leave those utilities in my docker image, and this issue documents the cause if anyone else faces it.

Sorry to scribble all over your repo. I'm finding zombienet pretty useful.

[JoshOrndorff]

One more note for posterity. I had some trouble with leftover state after some earlier ran crashed. To get back to a fresh starting point, I ran docker system prune.