Standard output truncated when using ssh with clitest

[rudymatela]

The standard output gets truncated when using ssh with clitest.

Consider a host with an user user with passwordless authentication set up (ssh-keygen and ssh-copy-id user@localhost).

Consider also the following clitest file (ssh.clitest):

$ echo 'first'
first
$ ssh user@localhost echo 'second'
second
$ echo 'third'
third

The above tests should pass, but they do not, see:

$ clitest ssh.clitest
#1  echo 'first'
#2  ssh user@localhost echo 'second'
#3  echo 'third'
---------------------------------------------------------
[FAILED #3, line 5] echo 'third'
@@ -0,0 +1 @@
+third
---------------------------------------------------------

FAIL: 1 of 3 tests failed

The output of the third command gets truncated.

I am on an Arch Linux with:

* clitest 0.4.0
* bash 5.0.16
* glibc 2.31
* linux 5.5.13

What about running ssh inside a script?

What about the following script?

#!/bin/bash
echo 'first'
ssh user@localhost echo 'second'
echo 'third'

When it is tested with the following clitest file (ssh2.clitest):

$ ./script.sh
first
second
third
$ ./script.sh
first
second
third

The first test works, but the second does not:

$ clitest ssh2.clitest
#1  ./script.sh
#2  ./script.sh
-----------------------------------------
[FAILED #2, line 5] ./script.sh
@@ -0,0 +1,3 @@
+first
+second
+third
-----------------------------------------

FAIL: 1 of 2 tests failed

Unfortunately, this means that clitest will not work for any script that uses ssh at some point.

[rudymatela]

Is this related to issue #27?

Through no fault of clitest, when writing to /dev/stderr directly, standard output gets truncated. (cf. Issue #27)

Althrough the fault here is the same (standard output getting trunctated), I don't think this issue is related to #27, see my investigation below. (It still could be of course, but this is my "hunch" after investigating a bit.)

Does ssh write to /dev/stdout or /dev/stderr?

The ssh program does not seem to write to /dev/stdout or /dev/stderr directly:

$ strace ssh user@localhost echo 'second' 2>&1 | grep /dev
openat(AT_FDCWD, "/dev/tty", O_RDWR|O_NONBLOCK) = 3
openat(AT_FDCWD, "/dev/null", O_RDWR)   = 3
openat(AT_FDCWD, "/dev/null", O_WRONLY) = 7

Putting clitest aside

Like @aureliojargas did in #27, let's put clitest aside and test just plain bash commands.

$ cat ./script.sh
#!/bin/bash
echo 'first'
ssh user@localhost echo 'second'
echo 'third'

If we run it, we get first, second and third:

$ ./script.sh
first
second
third

Now, let's try redirecting both stdout and stderr:

$ ./script.sh >stdout 2>stderr
$ cat stdout
first
second
third
$ cat stderr
$

So far so good. I tried a few different combinations and all seems to be working with just plain bash commands.

[rudymatela]

This is a weird behaviour that happens without clitest:

$ cat script.sh 
#!/bin/bash
echo 'first'
bash -c "echo 'second'"
ssh user@localhost bash -c "echo 'third'"
echo 'end'
$ ./script.sh 
first
second

end

Maybe it's related? 🤔

[rudymatela]

I found a workaround.

When making an ssh call, just make sure we redirect its stdin:

$ echo 'first'
first
$ ssh user@localhost echo 'second' </dev/null
second
$ echo 'third'
third

Now it suddenly works:

$ clitest fix.txt 
#1	echo 'first'
#2	nice ssh udge-run@localhost echo 'second' </dev/null
#3	echo 'third'
OK: 3 of 3 tests passed

This also applies to scripts that call ssh. If we make sure every call to ssh has its stdin being taken from a file, clitest works.

Weird... 🤔 💭

[rudymatela]

@aureliojargas Btw, thanks for making and maintaining this awesome tool. I have been using clitest for some years now, it is very handy.

[vmmello]

I agree that this is a bug in clitest. And it's indeed related to #27, on the same execution line.

A different test file that causes a similar bug (without using ssh) is the following:

$ echo a
a
$ echo b ; read
b
$ echo c
c

Running it will return:
$ ./clitest test-issue-42-2.clitest

#1	echo a
#2	echo b ; read
#3	echo c
--------------------------------------------------------------------------------
[FAILED #3, line 5] echo c
@@ -0,0 +1 @@
+c
--------------------------------------------------------------------------------

FAIL: 1 of 3 tests failed

The problem is that when clitest runs tests, all test commands are connected to the same standard input of clitest (that is $tt_temp_file from tt_process_test_file() ). When a test reads from stdin (the second case above, and ssh in the other example) it alters clitest's expected result of the next test.

So the solution is probably to change the execution line for it to read from /dev/null when running a command, like:

tt_run_test() {
...
  eval "$tt_test_command" > "$tt_test_output_file" 2>&1 < /dev/null
...
}

I tested the above change in the current master branch (a58e8aa) and it successfully ran the 272 tests of test.md.

NOTE: ssh does at least 1 read from stdin on every execution. When running it inside a shell script or inside a loop it'll always eat at least one line from the running shell (what @rudymatela pointed in the examples above), so in such cases it's better to pass -n to it ( or < /dev/null) to avoid this kind of problem.

[aureliojargas]

Thank you both @rudymatela and @vmmello for the extensive investigation and solution! I'm really impressed and thankful for that.

@vmmello could you open a pull request for the fix, to get it properly credited to you in the git history?

[aureliojargas]

Thank you both again, @vmmello's PR got merged and this issue should be fixed now. ❤️

[vmmello]

Great @aureliojargas! Thanks for the fast evaluation and merge!