readall on STDIN has 10MB limit

[canadaduane]

Here's a simple script:

readwrite.jl:

text = readall(STDIN)
print(text)

If I pipe 10,485,760 bytes or fewer to it, it will succeed:

ruby -e "10_485_760.times{ print 'x' }" | time julia readwrite.jl >/dev/null
        2.87 real         0.51 user         0.20 sys

If I pipe more than 10,485,760 bytes to it, it will freeze (note ctrl-C):

ruby -e "10_485_761.times{ print 'x' }" | time julia clean.jl >/dev/null
^C       14.96 real         0.48 user         0.20 sys

10,485,760 / 1024 is 10,240 which seems like there's some kind of 10MB limit going on here.

Just as a sanity check, I confirmed that an equivalent script in Ruby succeeds:

ruby -e "10_485_761.times{ print 'x' }" | time ruby -e 'puts STDIN.read' >/dev/null
        2.63 real         0.06 user         0.05 sys

I'm using Mac OS Mavericks. Julia 0.3.6.

[stevengj]

Could this be some kind of kernel limit, similar to #8789? In this case, however, there shouldn't be a fundamental problem, since the producer and the consumer of the pipe are different processes.

[nbaum]

_uv_hook_readcb (in stream.jl) limits the bytes read into an unlimited-size buffer to READ_BUFFER_SZ, currently 10485760.

Although it stops the async reading when the buffer is "full", it doesn't close the stream, which is what readall() is waiting for.

Possibly things will break if the callback closes a stream even though there is still data in it, and that doesn't really fix readall, anyway. It looks like readall wants to wait for the stream to stop being active and then start_reading() over again if it's still open.