maturin fails if sys.stdout is an io.StringIO object

[zlind0]

Bug Description

I am a package maintainer of a fedora-like linux distribution. In our case of using rpmbuild to build a python package by maturin, it throws an exception. This happens in the stage that some rpmbuild-related python scripts are called to generate the dependency of package.

Handling maturin>=1.0,<2.0 from build-system.requires
Requirement satisfied: maturin>=1.0,<2.0
   (installed: maturin 1.3.0)
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /usr/bin/python3
📡 Using build options features, bindings from pyproject.toml
Traceback (most recent call last):
  File "/usr/lib/rpm/anolis/pyproject_buildrequires.py", line 428, in main
    generate_requires(
  File "/usr/lib/rpm/anolis/pyproject_buildrequires.py", line 360, in generate_requires
    generate_run_requirements(backend, requirements)
  File "/usr/lib/rpm/anolis/pyproject_buildrequires.py", line 265, in generate_run_requirements
    dir_basename = prepare_metadata('.')
  File "/usr/lib64/python3.10/site-packages/maturin/__init__.py", line 210, in prepare_metadata_for_build_wheel
    sys.stdout.buffer.write(_output)
AttributeError: '_io.StringIO' object has no attribute 'buffer'
error: Bad exit status from /var/tmp/rpm-tmp.SoHxqG (%generate_buildrequires)

This exception turns out to be a known risk of using sys.stdout.buffer while the context of standard streams is unknown. By using sys.stdout.buffer, it is supposed that sys.stdout is io.TextIOWrapper. This is true for most of the common usages. However, in some cases, such as a library usage, sys.stdout can be io.StringIO, which does not have buffer, and here comes the exception.

It's been officially documented at here.

There is also similar issues for flake, at [1] [2].

Your maturin version (maturin --version)

1.3.0

Your Python version (python -V)

3.10.13

Your pip version (pip -V)

23.1

What bindings you're using

None

Does cargo build work?

• Yes, it works

If on windows, have you checked that you aren't accidentally using unix path (those with the forward slash /)?

• Yes

Steps to Reproduce

I have packed all the resources into a tarball. By downloading and extracting it you can reproduce this problem.

wget https://a.lind0.space/safetensors-0.4.0-bug.tar.gz
tar -xzf safetensors-0.4.0-bug.tar.gz
cd safetensors-0.4.0/
python3 -s anolis/pyproject_buildrequires.py --generate-extras --python3_pkgversion 3

Then, you will get this output:

Handling maturin>=1.0,<2.0 from build-system.requires
Requirement satisfied: maturin>=1.0,<2.0
   (installed: maturin 1.3.0)
(python3dist(maturin) < 2~~ with python3dist(maturin) >= 1)
🍹 Building a mixed python/rust project
🔗 Found pyo3 bindings
🐍 Found CPython 3.10 at /usr/bin/python3
📡 Using build options features, bindings from pyproject.toml
Traceback (most recent call last):
  File "/root/python-safetensors/safetensors-0.4.0/anolis/pyproject_buildrequires.py", line 428, in main
    generate_requires(
  File "/root/python-safetensors/safetensors-0.4.0/anolis/pyproject_buildrequires.py", line 360, in generate_requires
    generate_run_requirements(backend, requirements)
  File "/root/python-safetensors/safetensors-0.4.0/anolis/pyproject_buildrequires.py", line 265, in generate_run_requirements
    dir_basename = prepare_metadata('.')
  File "/usr/lib64/python3.10/site-packages/maturin/__init__.py", line 210, in prepare_metadata_for_build_wheel
    sys.stdout.buffer.write(_output)
AttributeError: '_io.StringIO' object has no attribute 'buffer'

[messense]

What's your proposed solution here? From reading the issues you linked, I'm leaning towards that if you are patching sys.stdout it's your responsibility to use io.TextIOWrapper.

@konstin What do you think?

[konstin]

Normally, i'd say maturin should support mocked sys.stdout, but for the case of PEP 517 specifically, the spec says:

Frontends should call each hook in a fresh subprocess, so that backends are free to change process global state (such as environment variables or the working directory). A Python library will be provided which frontends can use to easily call hooks this way.

I would argue that a linux distribution should follow the PEP 517 recommendation. You can use the build library which provides an easy integration.

[zlind0]

What's your proposed solution here? From reading the issues you linked, I'm leaning towards that if you are patching sys.stdout it's your responsibility to use io.TextIOWrapper.

@konstin What do you think?

Sorry I am not that familiar with python so I am searching for solution as well. If I find an elegant solution, I will comment.

Actually in the case I am not patching the sys.stdout, the type of sys.stdout differs because the standard output is redirected.

[messense]

Actually in the case I am not patching the sys.stdout, the type of sys.stdout differs because the standard output is redirected.

How exactly was the stdout redirected? I'm not sure why redirect stdout (like using > /dev/null in shell) would change it to io.StringIO.

[messense]

Actually in the case I am not patching the sys.stdout, the type of sys.stdout differs because the standard output is redirected.

How exactly was the stdout redirected? I'm not sure why redirect stdout (like using > /dev/null in shell) would change it to io.StringIO.

Ok, found it in your code

@contextlib.contextmanager
def hook_call():
    captured_out = StringIO()
    with contextlib.redirect_stdout(captured_out):
        yield
    for line in captured_out.getvalue().splitlines():
        print_err('HOOK STDOUT:', line)

so you are effectively patching sys.stdout with io.StringIO().

I'm going to close this as wontfix.