64kiB I/O limit after calling redirect_stdout

[stevengj]

Via JuliaLang/IJulia.jl#243, I observed this:

julia> (rd, wr) = redirect_stdout()
(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting))

julia> ccall(:puts, Cint, (Ptr{Uint8},), string([1:2^16]))
-1

julia> nb_available(rd)
65536

julia> readavailable(rd)
"[1,2,3,4,5, .............[output omitted]...............12773,127"

Note that the ccall returned -1 (EOF error) and the output was truncated to 64kiB.

This doesn't seem like desirable behavior. cc: @vtjnash, @Keno, @phlpn

[stevengj]

If I use print rather than libc's puts, the following code just hangs for me:

(rd, wr) = redirect_stdout()
print(string([1:2^16]))

I guess it is just waiting for the buffer to be read (with readavailable or whatever) until it writes more, because this works fine in IJulia (which reads the stdout pipe from a separate task).

[stevengj]

It seems like 64KiB is some default pipe buffer size in libuv?

I would be surprised if it can't be made to grow its buffers dynamically as needed; this seems like a severe limitation of its pipes otherwise.

Or are external C programs supposed to keep trying to write until space is available in the buffer? This creates a bit of a difficulty (especially as the IJulia task that is supposed to read from the buffer is blocked while external non-threaded C code executes).

[vtjnash]

Libuv doesn't have internal buffers, so I assume this is hitting the kernel limit, and the kernel is trying to push back against being requested to copy potentially unlimited amounts of data without an intervening read?

[stevengj]

Why shouldn't the kernel attempt to copy as much information as the user asks it to, until it runs out of memory?

[vtjnash]

Is errno set to EAGAIN? We may have requested the kernel to let us know when we are writing data faster than the kernel can get rid of it (by enabling nonblocking io)

[Keno]

Yes @vtjnash is correct. The libuv buffer size is 64K because that's the default kernel buffer size. The kernel may have given you EAGAIN.

[stevengj]

Yup, errno() is 35 (EAGAIN).

Is this desirable behavior? This means that external code in IJulia cannot write more than 64k to stdout.

[Keno]

It's kind of a catch-22. Since you're reading on the same thread as writing, it's either that or deadlocking your process. Though @vtjnash thinks (and I agree) that this is indeed a bug in our libuv code.

[vtjnash]

it's a bug in the libuv code, the process should have deadlocked there instead ("fix" coming shortly)

[elextr]

See also #8762

Note the pipe limit is only 4k on windows.

[vtjnash]

relatedly (somewhat), ye old SIGPIPE bug (#4219) seems to be back again

julia> r=Base.Pipe(C_NULL)
Pipe(null, 0 bytes waiting)

julia> w=Base.Pipe(C_NULL)
Pipe(null, 0 bytes waiting)

julia> Base.link_pipe(r,true,w,true)

julia> close(r)

julia> write(w,1)

signal (13): Broken pipe
write at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line)
unknown function (ip: -512753308)
uv_write2 at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
jl_write_copy at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
jl_putc_copy at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
write at stream.jl:732
write at io.jl:47
jlcall_write;62085 at  (unknown line)
unknown function (ip: -513619130)
jl_trampoline at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
unknown function (ip: -513662444)
jl_apply_generic at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
unknown function (ip: -513097098)
unknown function (ip: -513095799)
unknown function (ip: -513092849)
unknown function (ip: -513096751)
unknown function (ip: -513002420)
jl_toplevel_eval at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
jl_toplevel_eval_in at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
jl_f_top_eval at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
eval_user_input at REPL.jl:55
jlcall_eval_user_input;62037 at  (unknown line)
unknown function (ip: -513662444)
jl_apply_generic at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
anonymous at task.jl:97
unknown function (ip: -513619130)
jl_trampoline at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
unknown function (ip: -513051537)
unknown function (ip: -513048724)
unknown function (ip: -513050067)
unknown function (ip: -513049969)
jl_handle_stack_switch at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
julia_trampoline at /home/jameson/julia/usr/bin/../lib/libjulia-debug.so (unknown line)
unknown function (ip: 4203654)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 4199593)
unknown function (ip: 0)

(this is ubuntu, so signal(13) is SIGPIPE)

[vtjnash]

fixed in 0c48dd8

now it'll block in that first write call, like it was supposed to :/

[stevengj]

So the only way to do this is to have a second thread for reading from stdout? (If that's even possible with libuv...)

[vtjnash]

yep. or we can go back to being non-blocking, and then you have to deal with partial writes (libuv does this for us in nonblocking mode)

[stevengj]

I don't think it's reasonable to expect external C libraries to deal with partial writes.

[elextr]

Note also that there are no guarantees of atomicity for writes over the pipe buffer size, if another thread writes to stdout its output may be interleaved between the pieces of any write greater than the buffer size.

[vtjnash]

I don't think it's reasonable to expect external C libraries to deal with partial writes.

agreed, but that leaves us where we are now

some nice reading: http://linux.die.net/man/7/pipe

[staticfloat]

Is this actually ready for backporting? The discussion in 0c48dd8 makes it unclear.

[ivarne]

I'm not sure. Both the bug and the fix is above my head. @stevengj or @vtjnash will have to answer.

[vtjnash]

it should be backported

[tkelman]

can we fix the test failures in libuv first? JuliaLang/libuv#28 JuliaLang/libuv#29

Okay it looks like the tests in our fork of libuv weren't updated to account for our changes elsewhere in the library, I think, so maybe it's alright.

[tkelman]

"fix" backported in abaa08e

[ivarne]

Can this be closed now?

[stevengj]

I suppose; it seems like there is not much more we can do about it.