Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Running a container using system mode networking with exposed port failed in linux #3515

Open
praveenkumar opened this issue Feb 14, 2023 · 5 comments
Open

[BUG] Running a container using system mode networking with exposed port failed in linux #3515

praveenkumar opened this issue Feb 14, 2023 · 5 comments
Labels
kind/bug Something isn't working

Comments

@praveenkumar
Copy link
Member

In linux default network-mode is system so if a user try to run the podman preset and then try to create/run a container which expose the port it fails with following error.

$ podman-remote run -d -p 8080:80 docker.io/httpd:2.4 --log-level debug
Trying to pull docker.io/library/httpd:2.4...
Getting image source signatures
Copying blob sha256:9e8776e4b876795dc00ce93ee6409bc492f17894f9e3b61aeaf9a00c610c703c
Copying blob sha256:b7f64f2f874701178f37872cfc13dbf12adbcfda23ed90d73b8df12421a851e4
Copying blob sha256:f506d7aab6524182207e901fa77ff8ed100bf402fe4c7a8ad6fe178b8d8aa311
Copying blob sha256:05289ee4f2842c404631e433b1bbb9251cc3262bd0942b0c4be974061c5fa12e
Copying blob sha256:bb263680fed18eecdc67f885094df6f589bafc19004839d7fdf141df236a61aa
Copying config sha256:3a4ea134cf8e081516a776ce184dedc28986f941ed214b9012dc888049480f5a
Writing manifest to image destination
Storing signatures
Error: Post "http://gateway.containers.internal/services/forwarder/expose": dial tcp: lookup gateway.containers.internal: no such host

because podman bundle expect that gateway.containers.internal should be resolve able since it is podman machine requirement when exposing the port and for system mode networking we don't have it ( no gvsior stack running ). So either we need to check we can make it work for system mode networking or switch to user-mode networking for podman preset.
@praveenkumar praveenkumar added kind/bug Something isn't working status/need triage labels Feb 14, 2023
@cfergeau
Copy link
Contributor

The use of gateway.containers.internal does not seem to be possible to disable from podman-machine without code changes:
https://github.com/containers/podman/blob/main/libpod/networking_machine.go#L117-L132
https://github.com/containers/common/blob/main/pkg/machine/machine.go#L66-L70

praveenkumar added a commit to praveenkumar/crc that referenced this issue Mar 30, 2023
@praveenkumar
Remove container.conf file if system mode networking in use
1611a41 
The container.conf file has a setting called machine_enabled in the engine section,
which lets the podman client know that a command is running on an instance created
with the podman machine command. This allows the use of gvisor-tap-vsock when a
container is created with an exposed port. However, this setting should be disabled
for system mode networking, so that it doesn't prevent the creation of containers that
need to expose a port.

With this patch user can able to use microshift/podman preset with
system mode networking and start containers with exposed port but
not able to access that service from the host.

workaround for  crc-org#3515
@praveenkumar praveenkumar mentioned this issue Mar 30, 2023
Merged
@praveenkumar
Copy link
Member Author

crc-org/snc#675 and crc-org/snc#676 adds a marker file (podman/microshift bundle) and as workaround we do need to delete it during system mode networking.

praveenkumar added a commit to praveenkumar/crc that referenced this issue Mar 31, 2023
@praveenkumar
Remove container.conf file if system mode networking in use
dc803f9 
The container.conf file has a setting called machine_enabled in the engine section,
which lets the podman client know that a command is running on an instance created
with the podman machine command. This allows the use of gvisor-tap-vsock when a
container is created with an exposed port. However, this setting should be disabled
for system mode networking, so that it doesn't prevent the creation of containers that
need to expose a port.

With this patch user can able to use microshift/podman preset with
system mode networking and start containers with exposed port but
not able to access that service from the host.

workaround for  crc-org#3515
praveenkumar added a commit that referenced this issue Mar 31, 2023
@praveenkumar
Remove container.conf file if system mode networking in use
08c1923 
The container.conf file has a setting called machine_enabled in the engine section,
which lets the podman client know that a command is running on an instance created
with the podman machine command. This allows the use of gvisor-tap-vsock when a
container is created with an exposed port. However, this setting should be disabled
for system mode networking, so that it doesn't prevent the creation of containers that
need to expose a port.

With this patch user can able to use microshift/podman preset with
system mode networking and start containers with exposed port but
not able to access that service from the host.

workaround for  #3515
@nichjones1 nichjones1 moved this to Backlog in Project planning: crc Aug 8, 2023
@praveenkumar
Copy link
Member Author

Try to make usermode networking as default for linux

@jsliacan
Copy link
Contributor

jsliacan commented Oct 18, 2023 •
edited
Loading

I am not reproducing this locally (on f38), using default network-mode:

$ podman-remote --log-level debug run -d -p 8080:80 docker.io/httpd:2.4
INFO[0000] podman-remote filtering at log level debug   
DEBU[0000] Called run.PersistentPreRunE(podman-remote --log-level debug run -d -p 8080:80 docker.io/httpd:2.4) 
DEBU[0000] SSH Ident Key "/home/jsliacan/.crc/machines/crc/id_ecdsa" SHA256:qlqqoG+4lv5jeEBS72elsxK9HDh7u2S66/yjt+EU7P4 ecdsa-sha2-nistp521 
INFO[0000] key SHA256:SIpCXnqYGezkPLeMkoR93eN9mrkVHcS4cmNkXmkwtLo added to /home/jsliacan/.ssh/known_hosts 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.4.4/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.4.4/libpod/networks/pasta/exists 
DEBU[0000] Adding port mapping from 8080 to 80 length 1 protocol "" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Found credentials for quay.io in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v4.4.4/libpod/images/pull 
Trying to pull docker.io/library/httpd:2.4...
Getting image source signatures
Copying blob sha256:e94c45cb708a06cd1826d92621df930cba22fbd733fb20867b72c1871b605ca3
Copying blob sha256:a378f10b321842c3042cdeff4f6997f34f4cb21f2eff27704b7f6193ab7b5fea
Copying blob sha256:c20157372e943d84bb5a0624e80395697de1f41ecd54b3bcead2b03bb6b13fe8
Copying blob sha256:073cbcfef6634b5131786873a7a92a3b3bda43672e5830126dfa94352649358d
Copying blob sha256:c36006acf55e9d0c608ef6466d254a69faa43f7be8fe065d1b2a340d9002054b
Copying config sha256:ca77aadc3cbc20bfe57686d7d715acb10be6904ce303d9632b73c6fe4c5f6b85
Writing manifest to image destination
Storing signatures
DEBU[0011] DoRequest Method: GET URI: http://d/v4.4.4/libpod/images/docker.io%2Fhttpd:2.4/json 
DEBU[0011] DoRequest Method: POST URI: http://d/v4.4.4/libpod/containers/create 
DEBU[0011] DoRequest Method: POST URI: http://d/v4.4.4/libpod/containers/6d81d293f07f270332199b2ae0e339c9e0449f30a25f74e62ea6513053c2c512/start 
6d81d293f07f270332199b2ae0e339c9e0449f30a25f74e62ea6513053c2c512
DEBU[0012] Called run.PersistentPostRunE(podman-remote --log-level debug run -d -p 8080:80 docker.io/httpd:2.4) 
DEBU[0012] Shutting down engines

For comparison, this is what I get if I switch to user network-mode:

INFO[0000] podman-remote filtering at log level debug   
DEBU[0000] Called run.PersistentPreRunE(podman-remote --log-level debug run -d -p 8080:80 docker.io/httpd:2.4) 
DEBU[0000] SSH Ident Key "/home/jsliacan/.crc/machines/crc/id_ecdsa" SHA256:l2tmAlvq8Oe5z4sf2ATMExUtA6NNoKegpuCsFyG3F/o ecdsa-sha2-nistp521 
INFO[0000] key SHA256:SIpCXnqYGezkPLeMkoR93eN9mrkVHcS4cmNkXmkwtLo added to /home/jsliacan/.ssh/known_hosts 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.4.4/libpod/_ping 
DEBU[0000] DoRequest Method: GET URI: http://d/v4.4.4/libpod/networks/pasta/exists 
DEBU[0000] Adding port mapping from 8080 to 80 length 1 protocol "" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Found credentials for quay.io in credential helper containers-auth.json in file /run/user/1000/containers/auth.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v4.4.4/libpod/images/pull 
Trying to pull docker.io/library/httpd:2.4...
Getting image source signatures
Copying blob sha256:c20157372e943d84bb5a0624e80395697de1f41ecd54b3bcead2b03bb6b13fe8
Copying blob sha256:e94c45cb708a06cd1826d92621df930cba22fbd733fb20867b72c1871b605ca3
Copying blob sha256:073cbcfef6634b5131786873a7a92a3b3bda43672e5830126dfa94352649358d
Copying blob sha256:c36006acf55e9d0c608ef6466d254a69faa43f7be8fe065d1b2a340d9002054b
Copying blob sha256:a378f10b321842c3042cdeff4f6997f34f4cb21f2eff27704b7f6193ab7b5fea
Copying config sha256:ca77aadc3cbc20bfe57686d7d715acb10be6904ce303d9632b73c6fe4c5f6b85
Writing manifest to image destination
Storing signatures
DEBU[0011] DoRequest Method: GET URI: http://d/v4.4.4/libpod/images/docker.io%2Fhttpd:2.4/json 
DEBU[0011] DoRequest Method: POST URI: http://d/v4.4.4/libpod/containers/create 
DEBU[0011] DoRequest Method: POST URI: http://d/v4.4.4/libpod/containers/66ec19aed3fa73aa926efea1c7b72df4676bcc766bc73e7d05618099e56648c2/start 
66ec19aed3fa73aa926efea1c7b72df4676bcc766bc73e7d05618099e56648c2
DEBU[0012] Called run.PersistentPostRunE(podman-remote --log-level debug run -d -p 8080:80 docker.io/httpd:2.4) 
DEBU[0012] Shutting down engines

@praveenkumar
Copy link
Member Author

@jsliacan This is using openshift bundle or microshift one? if it is openshift then try using microshift one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
Project planning: crc
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@praveenkumar @cfergeau @jsliacan