Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libnative: Convert non-blocking fds to blocking when necessary #13355

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions src/liblibc/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2378,6 +2378,15 @@ pub mod consts {
pub static CLOCK_MONOTONIC: c_int = 1;

pub static WNOHANG: c_int = 1;

pub static F_GETFL: c_int = 3;
pub static F_SETFL: c_int = 4;
#[cfg(target_arch = "mips")]
pub static O_NONBLOCK: c_int = 0x0080;
#[cfg(target_arch = "arm")]
#[cfg(target_arch = "x86")]
#[cfg(target_arch = "x86_64")]
pub static O_NONBLOCK: c_int = 0o04000;
}
pub mod posix08 {
}
Expand Down Expand Up @@ -2828,6 +2837,10 @@ pub mod consts {
pub static CLOCK_MONOTONIC: c_int = 4;

pub static WNOHANG: c_int = 1;

pub static F_GETFL: c_int = 3;
pub static F_SETFL: c_int = 4;
pub static O_NONBLOCK: c_int = 0x0004;
}
pub mod posix08 {
}
Expand Down Expand Up @@ -3217,6 +3230,10 @@ pub mod consts {
pub static PTHREAD_STACK_MIN: size_t = 8192;

pub static WNOHANG: c_int = 1;

pub static F_GETFL: c_int = 3;
pub static F_SETFL: c_int = 4;
pub static O_NONBLOCK: c_int = 0x0004;
}
pub mod posix08 {
}
Expand Down
57 changes: 52 additions & 5 deletions src/libnative/io/file_unix.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ use std::io;
use libc::{c_int, c_void};
use libc;
use std::mem;
use std::os;
use std::rt::rtio;
use std::slice;

Expand Down Expand Up @@ -54,9 +55,11 @@ impl FileDesc {
// rtio traits in scope
pub fn inner_read(&mut self, buf: &mut [u8]) -> Result<uint, IoError> {
let ret = retry(|| unsafe {
libc::read(self.fd(),
buf.as_mut_ptr() as *mut libc::c_void,
buf.len() as libc::size_t) as libc::c_int
blocking(self.fd(), |fd| {
libc::read(fd,
buf.as_mut_ptr() as *mut libc::c_void,
buf.len() as libc::size_t) as i64
}) as libc::c_int
});
if ret == 0 {
Err(io::standard_error(io::EndOfFile))
Expand All @@ -69,8 +72,10 @@ impl FileDesc {
pub fn inner_write(&mut self, buf: &[u8]) -> Result<(), IoError> {
let ret = keep_going(buf, |buf, len| {
unsafe {
libc::write(self.fd(), buf as *libc::c_void,
len as libc::size_t) as i64
blocking(self.fd(), |fd| {
libc::write(fd, buf as *libc::c_void,
len as libc::size_t) as i64
})
}
});
if ret < 0 {
Expand All @@ -87,6 +92,48 @@ impl FileDesc {
}
}

// libc read/write can return EAGAIN or EWOULDBLOCK if the fd has O_NONBLOCK set.
// However, we're expecting blocking reads/writes. So if we get either of those errors
// we need to unset O_NONBLOCK and try again.
unsafe fn blocking(fd: fd_t, f: |fd_t| -> i64) -> i64 {
loop {
match f(fd) {
-1 => {
let e = os::errno() as int;
if e == libc::EAGAIN as int || e == libc::EWOULDBLOCK as int {
// the fd is marked as nonblock. Turn it off.
let mut flags = libc::fcntl(fd, libc::F_GETFL);
if flags == -1 { return -1 }
if flags & libc::O_NONBLOCK == 0 {
// O_NONBLOCK doesn't seem to be set. Did something else turn it off?
// Or does this system provide an O_NDELAY (which isn't POSIX) that differs
// from O_NONBLOCK? Try again without the loop.
return f(fd);
// Alternatively, something else could have unset the O_NONBLOCK flag in
// between our read/write call and the fcntl. But for that to happen, and
// then for something to subsequently turn it back on again before our
// single retry above, would be quite bizarre and should never happen in
// practice.
}
flags &= !libc::O_NONBLOCK;
if libc::fcntl(fd, libc::F_SETFL, flags) == -1 {
// fcntl() failed. POSIX says this should only happen due to EBADF, which
// would be the same error any subsequent read/write returns. But Linux
// seems to define EPERM as well. Just to be safe, lets re-issue the
// read/write and return. In the EPERM case it's plausible that the
// read/write would then return EAGAIN, but I don't believe we can actually
// hit the EPERM error code so that's a moot point.
return f(fd);
}
} else {
return -1
}
}
n => return n
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I said earlier, I'm a little uncomfortable blanket applying this to all file descriptors. We are only aware of this problem in connection with the stdout file descriptors.

Additionally, I think the interface may need to be tweaked slightly. The errno of the read/write syscall is currently overwritten when fcntl fails. I also think that this shouldn't cause an infinite loop, I would expect this to not retry if O_NONBLOCK wasn't set before, and I would only expect it to retry once afterwards. Is there a reason to continually try?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stdin and Stderr too definitively have this problem. And I fully expect it's possible to get the same problem with other file descriptors, e.g. by explicitly opening a terminal device. And of course you can dup2() stdout and get an fd that != 1 and yet I believe it will share the O_NONBLOCK flag.

Also, if we don't have this problem with other fds, then they'll never trigger this code.

The errno of read/write being overwritten is unfortunate. However, according to the man pages, the only errors that fcntl() can return for F_GETFL and F_SETFL are EAGAIN/EWOULDBLOCK and EBADF (a linux manpage indicates EPERM is possible, but only when clearing O_APPEND and we're not doing that). And of course EBADF is a legitimate error from read/write as well, so I'm not worried about it.

I don't believe this will cause an infinite loop, unless there's some way the system can return EAGAIN/EWOULDBLOCK when the O_NONBLOCK flag is not set. I checked and the only other way I found for that to happen is with O_NDELAY, which a) isn't POSIX, and b) seems to be equal to O_NONBLOCK on everything I looked at except for Linux on sparc, which we don't support. I could certainly add in code to ensure that the flags do have O_NONBLOCK set, but I don't expect that code to ever catch anything.

Continually retrying is necessary. Fds are a shared resource (certainly among threads, and as this bug indicates, certain file descriptors share their flags among processes). So libnative could remove O_NONBLOCK only to have something else immediately set it again before libnative has a chance to re-enter read()/write(). Therefore, doing this in an infinite loop is necessary. If it makes you feel any better, it makes progress on every pass through the loop, and it will eventually end (and of course it should only ever need a single retry in pretty much every single case. I will be pretty surprised if anyone actually manages to need multiple passes through the loop).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about it some more, bailing if O_NONBLOCK isn't set is technically dangerous, because some other process could have unset it before we called fcntl(), and then could set it again afterwards. This of course is only a problem if we check for the presence of O_NONBLOCK first.

O_NONBLOCK sucks. POSIX really should have just declared a read_nonblock() and write_nonblock().

Granted, this particular case should never be hit in practice. So I guess I can live with this edge case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The errno of read/write being overwritten is unfortunate ... so I'm not worried about it.

Just this morning I was sure that libnative would never see EAGAIN. Just a week ago I was sure that kill would fail if a process had died. Please handle errno(), defensive programming I've found is far better than assuming everything will go right.

I don't believe this will cause an infinite loop, unless there's some way the system can return

In the spirit of defensive programming, please remove the loop. The code is no less readable, and failing in an infinite loop (albeit rarely) is arguably much worse than returning an error from print. This is not an iron-clad solution against failing prints, this is simply trying to alleviate a bug which has been sporadically seen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alexcrichton This isn't about assuming it will go right. This is about the documented errors that fcntl() can return. According to POSIX the only valid error it can return here matches an error that read and write would return (for the same reason). And Linux documents a second error that I don't believe we can ever hit.

The problem with trying to handle errno correctly is that we can't simply save errno and restore it to what it was before the fcntl. And trying to change things such that blocking() returns the IoResult directly is awkward. The one thing I could do is re-issue the call to read/write, and I'm willing to do that if it makes you happy, but I just don't think it's necessary.

In the spirit of defensive programming, please remove the loop.

No. That's absolutely incorrect. The loop makes progress on every single pass through it (either by succeeding, or by removing the O_NONBLOCK flag). Removing the loop makes this code wrong. It in fact exists because of the spirit of defensive programming. It's defending against a rare race condition where some other thread/process just happens to flip O_NONBLOCK back on at the wrong time.

Removing the loop here would be like removing the loop in an atomic.fetch_and_add() implementation because you don't want that to spin forever.


impl io::Reader for FileDesc {
fn read(&mut self, buf: &mut [u8]) -> io::IoResult<uint> {
self.inner_read(buf)
Expand Down
8 changes: 8 additions & 0 deletions src/test/run-make/libnative-nonblock/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
-include ../tools.mk

all:
$(RUSTC) write.rs
$(CC) -o $(TMPDIR)/mknblock mknblock.c
mkfifo $(TMPDIR)/fifo
/bin/sh -c "(sleep 1; cat) < $(TMPDIR)/fifo > /dev/null" &
/bin/sh -c "($(TMPDIR)/mknblock; $(TMPDIR)/write) > $(TMPDIR)/fifo"
20 changes: 20 additions & 0 deletions src/test/run-make/libnative-nonblock/mknblock.c
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Copyright 2014 The Rust Project Developers. See the COPYRIGHT
// file at the top-level directory of this distribution and at
// http://rust-lang.org/COPYRIGHT.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.

#include <fcntl.h>
#include <stdio.h>

int main() {
if (fcntl(1, F_SETFL, O_NONBLOCK) == -1) {
perror("fcntl");
return 1;
}
return 0;
}
33 changes: 33 additions & 0 deletions src/test/run-make/libnative-nonblock/write.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
// Copyright 2014 The Rust Project Developers. See the COPYRIGHT
// file at the top-level directory of this distribution and at
// http://rust-lang.org/COPYRIGHT.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.

use std::io;
use std::io::IoResult;
use std::os;

fn run() -> IoResult<()> {
let mut out = io::stdio::stdout_raw();
for _ in range(0u, 1024) {
let mut buf = ['x' as u8, ..1024];
buf[1023] = '\n' as u8;
try!(out.write(buf));
}
Ok(())
}

fn main() {
match run() {
Err(e) => {
(writeln!(&mut io::stderr(), "Error: {}", e)).unwrap();
os::set_exit_status(1);
}
Ok(()) => ()
}
}