podman exec -i truncates the STDIN input #5046

goneri · 2020-01-31T23:32:05Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Hi,

I tried to get Ansible working with pipelining. Pipelining makes use of STDIN. The following command works fine with 1.8.0-dev:

echo 'print("a\n");'| podman exec -i 'ubi8' '/bin/sh' -c '/usr/bin/python3.6 && sleep 0'

But I get a failure if I try to inject something bigger (an Ansiblz). I managed to reproduce the problem with dd and an ubi8 container:

[goneri@t580-gone c]$ cat /usr/bin/vim | /usr/bin/dd of=/tmp/a
6336+1 records in
6336+1 records out
3244296 bytes (3.2 MB, 3.1 MiB) copied, 0.012102 s, 268 MB/s
[goneri@t580-gone c]$ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
904+1 records in
904+1 records out
463112 bytes (463 kB, 452 KiB) copied, 0.0150833 s, 30.7 MB/s
[goneri@t580-gone c]$ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
904+1 records in
904+1 records out
463112 bytes (463 kB, 452 KiB) copied, 0.0147833 s, 31.3 MB/s
[goneri@t580-gone c]$ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
904+1 records in
904+1 records out
463112 bytes (463 kB, 452 KiB) copied, 0.0139259 s, 33.3 MB/s
[goneri@t580-gone c]$ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
912+1 records in
912+1 records out
467208 bytes (467 kB, 456 KiB) copied, 0.0139809 s, 33.4 MB/s

Pass a content to podman exec -i.

The content is truncated.

podman-1.8.0-1580495541.git4699d5e0.fc31.x86_64

The text was updated successfully, but these errors were encountered:

mheon · 2020-02-03T14:48:32Z

Can you try this as root, and see if that resolves the issue? Want to see if this is definitely Podman or potentially fuse-overlayfs

goneri · 2020-02-03T15:51:24Z

Hi @mheon,

I don't see any difference:

root@t580-gone /h/g/g/a/c# podman ps
CONTAINER ID  IMAGE                                       COMMAND    CREATED        STATUS            PORTS  NAMES
70f6d55ff428  registry.access.redhat.com/ubi8/ubi:latest  /bin/bash  8 seconds ago  Up 8 seconds ago         agitated_ellis
root@t580-gone /h/g/g/a/c# cat /usr/bin/vim | podman exec -i agitated_ellis '/usr/bin/dd' of=/tmp/a
1648+0 records in
1648+0 records out
843776 bytes (844 kB, 824 KiB) copied, 0.00835746 s, 101 MB/s
root@t580-gone /h/g/g/a/c# cat /usr/bin/vim | podman exec -i agitated_ellis '/usr/bin/dd' of=/tmp/a
1368+0 records in
1368+0 records out
700416 bytes (700 kB, 684 KiB) copied, 0.00526816 s, 133 MB/s
root@t580-gone /h/g/g/a/c# cat /usr/bin/vim | podman exec -i agitated_ellis '/usr/bin/dd' of=/tmp/a
1344+0 records in
1344+0 records out
688128 bytes (688 kB, 672 KiB) copied, 0.00612946 s, 112 MB/s
root@t580-gone /h/g/g/a/c# cat /usr/bin/vim | podman exec -i agitated_ellis '/usr/bin/dd' of=/tmp/a
1336+0 records in
1336+0 records out
684032 bytes (684 kB, 668 KiB) copied, 0.00609849 s, 112 MB/s

mheon · 2020-02-03T15:53:37Z

@haircommander Could we be losing information along the Unix socket? Conmon not reading it fast enough, maybe?

TristanCacqueray · 2020-02-04T15:54:53Z

I couldn't reproduce with run:

localhost ~ $ cat /usr/bin/vim | /usr/bin/dd of=/tmp/a
6344+1 records in
6344+1 records out
3248224 bytes (3.2 MB, 3.1 MiB) copied, 0.0433427 s, 74.9 MB/s

localhost ~ $ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
Error: no container with name or ID ubi8 found: no such container

localhost ~ $ cat /usr/bin/vim | podman run -i 'ubi8' '/usr/bin/dd' of=/tmp/a
Trying to pull docker.io/library/ubi8...
  denied: requested access to the resource is denied
Trying to pull registry.fedoraproject.org/ubi8...
  manifest unknown: manifest unknown
Trying to pull registry.access.redhat.com/ubi8...
Getting image source signatures
Copying blob fd8daf2668d1 done
Copying blob cb3c77f9bdd8 done
Copying config 096cae65a2 done
Writing manifest to image destination
Storing signatures
6344+1 records in
6344+1 records out
3248224 bytes (3.2 MB, 3.1 MiB) copied, 0.0708013 s, 45.9 MB/s

localhost ~ $ cat /usr/bin/vim | podman run -i 'ubi8' '/usr/bin/dd' of=/tmp/a
6344+1 records in
6344+1 records out
3248224 bytes (3.2 MB, 3.1 MiB) copied, 2.71181 s, 1.2 MB/s

localhost ~ $ rpm -q podman
podman-1.6.2-2.fc31.x86_64

However, with this podman version, the podman exec stdin is ignored:

localhost ~ $ podman run --name ubi8 ubi8 sleep infinity

localhost ~ $ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
0+0 records in
0+0 records out
0 bytes copied, 0.000112077 s, 0.0 kB/s

mheon · 2020-02-04T16:04:43Z

Ah, that's the pipe issue - we fixed that in 1.7.0

mheon · 2020-02-04T16:05:57Z

The fact that run works means it's probably not a direct problem with attach, but something related to exec. Interesting...

Also, are there objections to using podman cp to use this?

lfarkas · 2020-02-06T10:40:05Z

is there any newer version for el7? it's getting more and more annoying...even on fedora and on rhel/centos-7 it not possible to load a mysql/mariadb dump into container. isn't it a basic feature?

mheon · 2020-02-06T14:38:08Z

Fedora definitely has a version with a fix for the pipes issue with run, though we're still not sure what's going on with exec here.

I believe there is an updated repo for EL7 that has more up-to-date versions of Podman; @lsm5 poke

goneri · 2020-02-06T15:24:23Z

@mheon, Podman 1.7.0-2 from Fedora is also broken:

[goneri@t580-gone ~]$ podman --version
podman version 1.7.0
[goneri@t580-gone ~]$ cat /usr/bin/vim | podman exec -i 'ubi8' '/usr/bin/dd' of=/tmp/a
0+0 records in
0+0 records out
0 bytes copied, 4.4775e-05 s, 0.0 kB/s

lfarkas · 2020-02-12T13:20:55Z

it's still not working in 1.8.0

chrmarti · 2020-02-17T15:21:31Z

I see a similar issue with 1.8.0 when piping a tar.gz. It gets part of the file across, but then breaks:

$ podman --version
podman version 1.8.0
$ uname -a
Linux chrmarti-ubuntu 5.3.0-29-generic #31-Ubuntu SMP Fri Jan 17 17:27:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ cat libpod-1.8.0.tar.gz | podman exec -i 41d03453f tar tvz
drwxrwxr-x root/root         0 2020-02-06 22:22 libpod-1.8.0/
-rw-rw-r-- root/root     24746 2020-02-06 22:22 libpod-1.8.0/.cirrus.yml
drwxrwxr-x root/root         0 2020-02-06 22:22 libpod-1.8.0/.copr/
-rw-rw-r-- root/root       676 2020-02-06 22:22 libpod-1.8.0/.copr/Makefile
-rw-rw-r-- root/root      1084 2020-02-06 22:22 libpod-1.8.0/.copr/prepare.sh
drwxrwxr-x root/root         0 2020-02-06 22:22 libpod-1.8.0/.github/
-rw-rw-r-- root/root      1738 2020-02-06 22:22 libpod-1.8.0/.github/ISSUE_TEMPLATE.md
drwxrwxr-x root/root         0 2020-02-06 22:22 libpod-1.8.0/.github/workflows/
-rw-rw-r-- root/root       671 2020-02-06 22:22 libpod-1.8.0/.github/workflows/stale.yml
-rw-rw-r-- root/root       457 2020-02-06 22:22 libpod-1.8.0/.gitignore
-rw-rw-r-- root/root       479 2020-02-06 22:22 libpod-1.8.0/.golangci.yml
-rw-rw-r-- root/root       322 2020-02-06 22:22 libpod-1.8.0/.pre-commit-config.yaml
-rw-rw-r-- root/root      1766 2020-02-06 22:22 libpod-1.8.0/.ubuntu_prepare.sh
-rw-rw-r-- root/root     87135 2020-02-06 22:22 libpod-1.8.0/API.md
-rw-rw-r-- root/root     15798 2020-02-06 22:22 libpod-1.8.0/CONTRIBUTING.md
-rw-rw-r-- root/root      3401 2020-02-06 22:22 libpod-1.8.0/Dockerfile
-rw-rw-r-- root/root      2528 2020-02-06 22:22 libpod-1.8.0/Dockerfile.centos
-rw-rw-r-- root/root      2271 2020-02-06 22:22 libpod-1.8.0/Dockerfile.fedora
-rw-rw-r-- root/root     11357 2020-02-06 22:22 libpod-1.8.0/LICENSE
-rw-rw-r-- root/root     24980 2020-02-06 22:22 libpod-1.8.0/Makefile
-rw-rw-r-- root/root       258 2020-02-06 22:22 libpod-1.8.0/OWNERS
-rw-rw-r-- root/root     10461 2020-02-06 22:22 libpod-1.8.0/README.md
-rw-rw-r-- root/root     99783 2020-02-06 22:22 libpod-1.8.0/RELEASE_NOTES.md

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now
Error: non zero exit code: 2: OCI runtime error

edsantiago · 2020-02-19T18:44:04Z

Just to clarify: pipes have nothing to do with this, at least not between host and container (possibly between conmon and podman though). Same thing happens with simple redirection:

# podman exec -i alpine_container dd of=/tmp/aaa </bin/vi
1808+0 records in         [ expected: 2747]
1808+0 records out
# !!
2696+0 records in
2696+0 records out

Curiously, it does not seem to happen (despite dozens of retries) when of=/dev/null:

# podman exec -i alpine_container dd of=/dev/null </bin/vi
2747+1 records in
2747+1 records out

(I say "curiously" because conmon has no way of knowing that the container's output is /dev/null; so interprocess communication should be the same in either case). (But I may be wrong).

jonasbb · 2020-03-09T20:17:51Z

This issue now blocks compatibility with the VSCode Remote extension (microsoft/vscode-remote-release#116 (comment)).

jan-hudec · 2020-03-09T21:07:15Z

@edsantiago,

I say "curiously" because conmon has no way of knowing that the container's output is /dev/null

Could it be a timing issue then? Writing /dev/null is faster, because the kernel does not even bother copying the data. And the first test where it wrote less data the first time suggests the same—first run is slower, because the executable code is not cached yet.

mheon · 2020-03-16T17:54:32Z

Exact amount copied seems to vary, but on my system, it seems like ~60kb is the threshold for a successful, non-truncating copy.

mheon · 2020-03-16T18:03:07Z

Fragment of strace, around where the copy stopped:

[pid 31397] read(0, "\31\245K\30pR\334\242\2\220iHb\334\246\36\240ZTB@\348\247\16*\r\10\240 b\334"..., 512) = 512
[pid 31397] write(1, "\31\245K\30pR\334\242\2\220iHb\334\246\36\240ZTB@\348\247\16*\r\10\240 b\334"..., 512) = 512
[pid 31397] read(0, "", 512)            = 0
[pid 31397] close(0)                    = 0
[pid 31397] close(1)                    = 0

The file I sent wasn't evenly divisible by 512, which makes me think this might be client-side - we're reading along as normal, but then we get a read of 0 for EOF.

The closes are pretty interesting, because I don't know where or why we're doing that, and why we're closing STDOUT after EOF on STDIN (and not STDERR, which remains open).

mheon · 2020-03-16T18:34:28Z

Block size has an effect

msekletar · 2020-03-17T22:22:04Z

I was able to reproduce the issue fairly reliably (on Fedora 32 with 1.8.1).

In one terminal I ran following command,

while true; do podman exec -i container dd of=/tmp/test < test 2>&1 | grep -q 500000 || echo fail ; done

Btw my test file had 500 kB. While the loop was running it wasn’t printing anything, no truncation occurred. And then in the second terminal I spawned and exited another container (plain bash and just ran exit). After that string fail got printed immediately, dd didn’t write entire file. I don’t know what exactly is going on but it seems that running containers in parallel to podman exec is somehow related.

mheon · 2020-03-17T22:27:51Z

We've traced this further and believe that it's a problem somewhere in Conmon. For now, in use-cases related to dd, setting a large block size (I tested with bs=32k) should alleviate the issue.

ttys3 · 2020-03-18T03:35:10Z

and also don't forget mysql
#5419

redbaron · 2020-03-29T10:05:54Z

It was resolved in containers/conmon#131 ,fix is in conmon >= 2.0.14

- exec: add test for containers#5046, in which conmon swallowed chars on a large byte transfer - pod: add 'pod exists' tests, both positive and negative; consolidate tests; add '--label', and check in 'pod inspect' add 'pod ps' tests - networking: add test for containers#5466, in which detached run with --userns=keep-id would not forward a port Signed-off-by: Ed Santiago <santiago@redhat.com>

Problem happens after all tests pass: Run command with stdout: podman exec --user pytest -i ansible-test-controller-hLEbtZTx sh -c 'tar cf - -C /root/ansible_collections/scale_computing/hypercore/tests --exclude .tmp output | gzip' Run command with stdin: tar oxzf - -C /home/runner/work/HyperCoreAnsibleCollection/HyperCoreAnsibleCollection/ansible_collections/scale_computing/hypercore/tests WARNING: Command "tar oxzf - -C /home/runner/work/HyperCoreAnsibleCollection/HyperCoreAnsibleCollection/ansible_collections/scale_computing/hypercore/tests" returned exit status 2. >>> Standard Error gzip: stdin: unexpected end of file tar: Unexpected EOF in archive Maybe this will help. See also containers/podman#5046 Signed-off-by: Justin Cinkelj <justin.cinkelj@xlab.si>

ansible-test-gh-action is used for ansible-test sanity and units. Some sanity and units test are update, as errors were reported. Gitlab CI config is removed, we do not need it any more. Podman is not used - we had some tar and pipe problems. See also containers/podman#5046 Problem happens after all tests pass: Run command with stdout: podman exec --user pytest -i ansible-test-controller-hLEbtZTx sh -c 'tar cf - -C /root/ansible_collections/scale_computing/hypercore/tests --exclude .tmp output | gzip' Run command with stdin: tar oxzf - -C /home/runner/work/HyperCoreAnsibleCollection/HyperCoreAnsibleCollection/ansible_collections/scale_computing/hypercore/tests WARNING: Command "tar oxzf - -C /home/runner/work/HyperCoreAnsibleCollection/HyperCoreAnsibleCollection/ansible_collections/scale_computing/hypercore/tests" returned exit status 2. >>> Standard Error gzip: stdin: unexpected end of file tar: Unexpected EOF in archive Signed-off-by: Justin Cinkelj <justin.cinkelj@xlab.si>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Jan 31, 2020

goneri mentioned this issue Jan 31, 2020

podman connection broken with pipelining==True ansible/ansible#67004

Closed

chrmarti mentioned this issue Feb 21, 2020

Podman support microsoft/vscode-remote-release#116

Closed

mheon mentioned this issue Mar 7, 2020

podman exec -i import large .SQL file got stuck #5419

Closed

tyll mentioned this issue Mar 23, 2020

Use podman for the molecule test linux-system-roles/network#165

Closed

4 tasks

rhatdan closed this as completed Mar 29, 2020

chrmarti mentioned this issue Apr 27, 2020

Test: Podman support microsoft/vscode-remote-release#2854

Closed

2 tasks

edsantiago mentioned this issue Jun 1, 2020

system tests : more tests #6456

Merged

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman exec -i truncates the STDIN input #5046

podman exec -i truncates the STDIN input #5046

goneri commented Jan 31, 2020

mheon commented Feb 3, 2020

goneri commented Feb 3, 2020

mheon commented Feb 3, 2020

TristanCacqueray commented Feb 4, 2020

mheon commented Feb 4, 2020

mheon commented Feb 4, 2020

lfarkas commented Feb 6, 2020

mheon commented Feb 6, 2020

goneri commented Feb 6, 2020

lfarkas commented Feb 12, 2020

chrmarti commented Feb 17, 2020

edsantiago commented Feb 19, 2020

jonasbb commented Mar 9, 2020

jan-hudec commented Mar 9, 2020

mheon commented Mar 16, 2020

mheon commented Mar 16, 2020

mheon commented Mar 16, 2020

msekletar commented Mar 17, 2020

mheon commented Mar 17, 2020

ttys3 commented Mar 18, 2020

redbaron commented Mar 29, 2020 •

edited

Loading

podman exec -i truncates the STDIN input #5046

podman exec -i truncates the STDIN input #5046

Comments

goneri commented Jan 31, 2020

mheon commented Feb 3, 2020

goneri commented Feb 3, 2020

mheon commented Feb 3, 2020

TristanCacqueray commented Feb 4, 2020

mheon commented Feb 4, 2020

mheon commented Feb 4, 2020

lfarkas commented Feb 6, 2020

mheon commented Feb 6, 2020

goneri commented Feb 6, 2020

lfarkas commented Feb 12, 2020

chrmarti commented Feb 17, 2020

edsantiago commented Feb 19, 2020

jonasbb commented Mar 9, 2020

jan-hudec commented Mar 9, 2020

mheon commented Mar 16, 2020

mheon commented Mar 16, 2020

mheon commented Mar 16, 2020

msekletar commented Mar 17, 2020

mheon commented Mar 17, 2020

ttys3 commented Mar 18, 2020

redbaron commented Mar 29, 2020 • edited Loading

redbaron commented Mar 29, 2020 •

edited

Loading