Latest version of Podman available for RHEL 9 fails when trying to run container on disconnected host #24614
Comments
|
please define what a disconnected environment means ... |
|
Hello. Disconnected means no Internet connectivity.
…On Tue, Nov 19, 2024, 3:08 PM Brent Baude ***@***.***> wrote:
please define what a disconnected environment means ...
—
Reply to this email directly, view it on GitHub
<#24614 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOEEC2PLX6VW6YKPTGNVF6L2BOSEFAVCNFSM6AAAAABSDBRNPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBWG42TKOBVGA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
sbrivio-rh
added
network
|
It depends on what you mean by "disconnected", as @baude pointed out. If you have a network interface on the host that doesn't actually route traffic to the public internet, things should work (see also https://bugzilla.redhat.com/show_bug.cgi?id=2277954). If you don't have a network interface at all, or the interface doesn't have addresses, pasta(1) needs (by default) to source addresses and routes from an interface to provide network connectivity to the container, so it will refuse to start. At that point, the kind of network connectivity we should provide to the container is not clearly defined. It probably makes sense to require In any case, we're looking into options to improve usability on this topic, see also https://pad.passt.top/p/PastaWithoutNetworkConnectivity |
|
@sbrivio-rh Thank you for the reply! I want to second this bug as it is an issue for me too. My team develops an image that is designed to be run in air-gapped environments with podman. For the last few years running images with podman while not connected to the internet has been no issue. Now when we started getting the Here is my most recent test, I just re-installed Fedora 41 on a bare-metal laptop and ran with wifi connected to the internet, works fine: With wifi network disconnected but interface enabled, pasta error: with wifi disconnected from network and using So @sbrivio-rh what appears to be happening is the case you described where if no network interface is available the pasta error is expected unless you add However I certainly do have a network interface, and it works fine, In fact I'm submitting posting this comment using the same laptop from my tests above just with my wifi connected to the internet so I know for a fact that my network available, and enabled, and interface is working. This looks like a bug in podman to me, the expected behavior should be the first case that you described "If you have a network interface on the host that doesn't actually route traffic to the public internet, things should work ", that expected behavior is not happening for me, as you can see from my tests above. Simply disconnecting from a wifi network causes the Podman should work fine without an internet connection without having to add a special option This should be very easy to reproduce as for anyone as well, on Fedora 41 with the latest updates, simply disconnect from wifi, or unplug your network cable and try running a container. |
Yes, but as far as I understand, it has no address and routes. From my earlier comment (emphasis added in quote):
It's not an issue in Podman. It can be considered an issue in pasta (well, the behaviour is intended, but I see your problem). It's a separate component, just like, before Podman 5.x, you would use slirp4netns (also a different component) for rootless networking.
The problem is that pasta doesn't know which address it should use. By default, it copies one from the interface, but if there's no address, what should it do? You can also configure addresses and routes with pasta's options Until now, almost all the reports we got of "purposely" disconnected environments were from setups with an isolated network, but an interface with address and route, so we took care of that case. If this (different) case is relevant, then we should probably think of introducing some arbitrary default address and routes. We didn't do it so far because pasta doesn't have a way to know that a network interface is currently missing an address, and if you bring up a container before the address is there, we don't want to silently continue with a behaviour that differs from the case where the interface already has an address assigned. On the other hand, the non-determinism involved here was mostly problematic because of #22197, which has been solved now, so we might reasonably revisit it. I would like to wait for some inputs from Podman developers before "fixing" this. |
|
@Luap99, long story short: we have at least two reports of users who would expect pasta to start even if they have a network interface that's not connected at all (no address, no route) and, at least in one case, it will never get connected. This is different from cases with "isolated networks", intermittent connectivity, or quadlets at boot. What do you think? We could handle that in pasta (with a link-local address?) I suppose: now that #22197 is solved, the "timing at boot" part is not critical anymore. |
Do we need an address at all? If the host doesn't even have one there would be now way to connect to the host. (Well I guess the --map-host-loopback --map-gw would allow you to reach localhost so maybe we need one?) I guess having a link local address inside should work fine if then also keeps working once the host interface gets a proper routable external address? I guess that would still need the monitoring part to update routes in the container? https://pad.passt.top/p/PastaWithoutNetworkConnectivity Overall I am not sure if I would draw a difference between this case and the no connectivity early at boot that should be updated later (because well we can never know when connectivity will happen if at all). @bfields3 If you have an issue in RHEL you should report that though the Red Hat support channels to ensure things get fixed in RHEL because anything we do upstream will only be fixed in a future version and then RHEL may or may not rebase to a new version but that would only happen with something like RHEL 9.6 never an older release. So to ensure fixes are backported when needed bugs must be reported through the Red Hat support channel. Linking upstream reports there is totally fine. |
Right, we don't actually need an address to fix the two cases from this ticket. And for The only problem with that is that we would force local connections to have loopback addresses (no
Yes, at that point we would (implicitly) switch to NAT. Routes inside the container are not really that important, as long as they are usable and used.
Not necessarily: we could add that later on top (exactly because what route we set up in the container doesn't matter that much).
Right, I think so too. But we can have a fallback address even without that. |
Yes you are right, if it defaults to NAT with a default route then there is no requirement to update the address later so I think this is the best solution. |
|
Just adding my 👍 to this. Last week I was doing some local development involving an image when my WiFi went out. Podman stopped working with the same error @bfields3 mentioned, and I was blocked from making further progress even though my work required no network connection (I work on an offline product). This scenario used to work, so thanks for your attention @sbrivio-rh & @Luap99. Looking forward to a patch! |
|
All,
Thank you so much for your assistance on this. I also would like to
mention that I did the same tests with Windows environment using a Windows
11 host and RHEL and Fedora VMs. Same results as what Jared got as well.
Respectfully,
Benny Fields III
…On Thu, Nov 21, 2024 at 9:51 AM Michael Clayton ***@***.***> wrote:
Just adding my 👍 to this. Last week I was doing some local development
involving an image when my WiFi went out. Podman stopped working with the
same error @bfields3 <https://github.com/bfields3> mentioned, and I was
blocked from making further progress even though my work required no
network connection (I work on an offline product). This scenario used to
work, so thanks for your attention @sbrivio-rh
<https://github.com/sbrivio-rh> & @Luap99 <https://github.com/Luap99>.
Looking forward to a patch!
—
Reply to this email directly, view it on GitHub
<#24614 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOEEC2MGGUJHCYN6S5TZXJ32BX6RHAVCNFSM6AAAAABSDBRNPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJRGYYDIMRQGY>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@sbrivio-rh I think the case that is much more common will be cases like @mwcz described, where he is working on a laptop and his wifi goes down. There are a million reasons why this might happen and I'm sure all of us have experienced some of these: Reasons why you might not be connected to a network:
I feel like in these cases podman run should still work fine without having to remember any special steps to avoid getting a "pasta" error. All of us have container development workflows that we rely on, it will be very annoying if we have to alter these depending on if our network interface is connected to a network with an address or not, and have to swap back and forth depending on the state of our network. |
...I just spent about 10 hours on trains today, it's not that I don't understand, but we had, in the past couple of months, several types of reports with rather different use cases (from "I'm often on trains/busses and I have a database client on the host" to "my container is air gapped") that all fall under "no connectivity". We addressed some of them, and we're trying (we've been trying for a while, actually) to understand what's the best solution for some other ones. It's not that simple. Podman could just happily start your containers with Now that #22197 is (mostly, or at least conceptually) solved, things are a bit simpler, and there are a couple of solutions we could adopt (essentially in the categories of 1. no address/routes in the container or 2. link-local/hardcoded addresses routes in the container). I'm playing with those. 2. is probably the most compatible, albeit conceptually wrong. But perhaps the conceptually wrong isn't that problematic after all. |
Issue Description
Using RHEL 9 hosts and the latest version of Podman available for RHEL 9, once a container image is downloaded/pulled, when trying to create/start the container in a disconnected environment, it fails with the error shown below:
Steps to reproduce the issue
Steps to reproduce the issue
Each test used a container image pulled from quay.io. Each test performed the podman run command to start/run the container in a disconnected environment and connected as shown in attachment. Troubleshooting for Podman.pdf
Describe the results you received
Please see attached for detailed test results:
Troubleshooting for Podman.pdf
Describe the results you expected
Expected results would include the successful output of the container being created/run after running the podman run command.
podman info output
Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Additional environment details:
Test 1 and 2 were RHEl 9 in a VM on a Fedora 41 bare-metal host. Test 3 was bare-metal Fedora 41 host.
Additional information
Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting
The text was updated successfully, but these errors were encountered: