capsys fails for subprocess calls

[neiljp]

On attempting to test external (subprocess) calls, I noticed unexpected failures, even though the output is shown in the 'captured' stdout output area. I have reduced my example code to something like:

import pytest, subprocess

# These fail, for a local file containing "foo", using cat & echo commands
@pytest.mark.xfail
def test_cat(capsys):
    subprocess.call(["cat","foo.txt"])
    assert capsys.readouterr().out == "foo"
@pytest.mark.xfail
def test_echo(capsys):
    subprocess.call(["echo","foo"])
    assert capsys.readouterr().out == "foo"

# Given this...
def some_func():
    print("foo")
#...this passes as expected
def test_internal_print(capsys):
    some_func()
    assert capsys.readouterr().out == "foo\n"

My understanding is that pytest should have the marked xfail cases above pass (without the marks).

attrs (17.4.0)
pip (9.0.1)
pkg-resources (0.0.0)
pluggy (0.6.0)
py (1.5.2)
pytest (3.4.2)
setuptools (38.6.0)
six (1.11.0)
smmap2 (2.0.3)
wheel (0.30.0)

pytest 3.4.2, Debian stretch.

[pytestbot]

GitMate.io thinks possibly related issues are #623 (capsys: Output is not reported after failing test that called capsys.readouterr()), #2314 (capsys fails to capture the output by stdout), #1164 (test_something fails), #2686 (tmpdir and subprocess fail in CI environment, work locally), and #74 (test_cmdline_python_package() fails).

[RonnyPfannschmidt]

capsys unlike capfd does not capture usage of the underlying FD, this is per design

[neiljp]

capfd does not appear to do so either. I ended up using a subprocess.Popen and capturing the stdout using PIPE.

[RonnyPfannschmidt]

if capfd doesnt do it then thats a bug

[offbyone]

I can definitely repro this with capfd:

def test_script_runs(capfd):
    with pytest.raises(subprocess.CalledProcessError):
        subprocess.check_call("some shell comand", shell=True)

    out, err = capfd.readouterr()
    assert "usage:" in out

        out, err = capfd.readouterr()
>       assert "usage:" in out
E       AssertionError: assert 'usage:' in ''

test/test_interface.py:18: AssertionError

If I remove capfd then I see the usage: statement in pytest's captured stdout.

[RonnyPfannschmidt]

@offbyone which would be a different bug, please report it separately