Cannot read from STDIN with read_to_string if reading before piping Rust program writes to STDOUT in another Rust program

[kiseitai3]

Using std::io::read_to_string, stdin().read_to_string, or stdin().read_to_end() blocks on nonexisting buffer and ignores incoming buffer. This happens if I pipe a program compiled with Rust to itself. For example, ./target/debug/rumtk-v2-interface --port 55556 --local | ./target/debug/rumtk-v2-interface --daemon --outbound --port 55555 --local. Reading using the buffer methods work well assuming I am able to piece together the buffer before converting to UTF-8 String. Flushing STDOUT from the piping program does not force the STDIN read to succeed.

I tried this code. Only read_some_stdin works:

pub fn read_stdin_() -> RUMResult<RUMString> {
        let mut stdin_handle = stdin().lock();
        let has_data = match stdin_handle.has_data_left() {
            Ok(flag) => flag,
            Err(e) => return Err(format_compact!("No data in STDIN: {}", e)),
        };
        println!("{}", &has_data);
        if has_data {
            match std::io::read_to_string(stdin_handle) {
                Ok(s) => {
                    println!("{}", &s);
                    Ok(s.into())
                }
                Err(e) => Err(format_compact!("Error reading from STDIN: {}", e)),
            }
        } else {
            Ok(RUMString::default())
        }
    }

    pub fn read_stdin() -> RUMResult<RUMString> {
        let mut buf: Vec<u8> = Vec::new();
        match stdin().read_to_end(&mut buf) {
            Ok(s) => {
                println!("{}", &s);
                Ok(buf.to_rumstring())
            }
            Err(e) => Err(format_compact!("Error reading from STDIN: {}", e)),
        }
    }

    pub fn read_some_stdin(input: &mut StdinLock) -> RUMResult<RUMString> {
        let mut buf: [u8; 1024] = [0; 1024];
        match input.read_exact(&mut buf) {
            Ok(_) => Ok(buf.as_slice().to_rumstring()),
            Err(e) => Err(format_compact!(
                "Error reading 1024 bytes from STDIN: {}",
                e
            )),
        }
    }

I expected to see this happen: Have the stdout buffer pass to the stdin buffer of a Rust program piped to itself via the terminal. I expected to receive the buffer when reading or at least get an error I could handle/ignore if a read is empty.

Instead, this happened: Nothing happens even though program can read from stdin and write to stdout in isolated tests.

This is similar to forum? issue: [How to handle Stdin hanging when there is no input](https://users.rust-lang.org/t/how-to-handle-stdin-hanging-when-there-is-no-input/93512).

I am currently not sure why the issue happened after looking at the std library's STDIN source code, but I am not experienced in the codebase so it could be a me issue.

Thanks for your work on Rust and thank you for your help!

Meta

rustc --version --verbose:

rustc 1.86.0-nightly (4a4309466 2025-02-02)
binary: rustc
commit-hash: 4a43094662727040d8522163f295b19d1aed0e49
commit-date: 2025-02-02
host: x86_64-unknown-linux-gnu
release: 1.86.0-nightly
LLVM version: 19.1.7

Backtrace

<backtrace>

[RalfJung]

Does the sending part of the pipe ever close their end of it (e.g. by exiting)? If not, this is entirely expected behavior: read_to_end reads until the end, as the name says -- and as long as the sending program keeps the pipe open, the end has not been reached yet, more data could come in any time.

Can you share the code for the sending part?

Generally, read_to_end and read_to_string should only be used for IO streams that have a clear end, like files. They make no sense on an IO stream that is expected to stay around and keep streaming data.

[kiseitai3]

Hmm. I agree with your observation.

I had a similar idea last night while writing the bug report. To test that hypothesis, I tried adding a break. Same issue as reported. Tonight, I introduced a panic! on the sender to confirm that nothing gets picked up on the receiving end despite the pipe technically closing.

Hello, the blank line, and the panic printout comes from the receiver. The JSON bit is actually printed from the receiving end. Nothing was sent to the other endpoint I was using in the test, but that might be a separate issue. The null hypothesis seems to be correct here. The pipe closure does allow the receiver to capture the input.

I believe you are right.

I did check the repository and confirmed that trait Read is fully defined for Stdin and StdinLock.
I think that the docs should warn against usage of these methods with stdin if the sender is not expected to exit or close the stream.
I clearly misunderstood what was meant by reading to end. I assumed it meant reading to end of message and overlooked the fact that std{in,out,err} are present for the lifetime of the program and have no concept of message. I need to read using the buffer methods until the first empty read.

Depending on how you think it is best to proceed (and if you do not mind), this could be a great opportunity for me to open a PR and contribute.

Thank you!

[RalfJung]

Yeah, pointing this out in the docs seems like a good idea. I won't have the time to oversee a doc fix here, but if you just make a PR, a suitable reviewer will be assigned to that. :)

[hkBst]

@rustbot label A-docs

[kiseitai3]

Added a note to the two methods making clear that EOF is never reached/ expected to be reached if reading from an stdin pipe that does not get closed.

[kiseitai3]

Just to add more documentation to the record for future others that may fail to read the documentation, Once data is picked up from stdin, it looks like we never get back to 0 bytes.

In this test, I broke reads into 512 bytes max using ...

pub fn read_stdin() -> RUMResult<RUMString> {
        let mut stdin_lock = stdin().lock();
        let mut stdin_buffer: Vec<u8> = Vec::with_capacity(BUFFER_SIZE);
        let (mut size, mut buf) = read_some_stdin(&mut stdin_lock)?;
        println!("Reading STDIN!");
        while size > 0 {
            for itm in buf.iter() {
                stdin_buffer.push(*itm);
            }
            let result = read_some_stdin(&mut stdin_lock)?;
            size = result.0;
            buf = result.1;
            println!("Read {} bytes", &size);
        }
        print!("{}", &stdin_buffer.to_rumstring());
        Ok(stdin_buffer.to_rumstring())
    }

It seems that the minimum resultant buffer is all zeroes. I am not sure if this is a side effect from the kernel or writing to stdout in Rust.

This Go Reddit confirms our suspicions about reading and writing to std. Someone does recommend forcing a flush of buffers on writer side but my writer function already flushes after every write and the receiving end never triggers.

At any rate, I could not figure out the reason for the returned 61 bytes upon completing the buffer reads from the code review alone. I will see if I return to this in the future to have a more complete account of events, but for now, I can just check if the first byte of a returned buffer is zero. That should be good enough for my purpose.

Hopefully, this report helps others in the future.