Problem writing more than 65536 bytes to a pipe

[davidlevner]

ipc-run-writing-to-pipe-issue.zip

I am trying to use IPC::Run to run a program and pipe data to the program's standard input. Everything works fine if the data is less than 65536 bytes long. However, data that is longer than 65536 bytes is truncated.

The zip file I attached contains code I've written to demonstrate the problem. The contents of the zip file are the following files:

count-bytes.pl: Reads STDIN and prints the number of bytes and lines read.
ipc-run-test.pl: A scaffold for testing IPC::Run.
ipc-run-test-data.txt: 10000 lines of test data. Each line is 8 bytes long.
ipc-pipe-test.pl: Runs count-bytes.pl and pipes to it 80000 bytes using perl's native 'open'.

Here is a session that demonstrates the problem (what I typed is in italics):

$ perl ipc-run-test.pl
Command: perl count-bytes.pl
Input line (blank line to end the input): @ipc-run-test-data.txt
Input line (blank line to end the input):
Output line 1: '8192 lines, 65536 bytes'
The child sent nothing sent to STDERR.
Command 'perl count-bytes.pl' returned 0
$

ipc-run-test.pl used IPC::Run::start to run 'perl count-bytes.pl' and sent it all the data in ipc-run-test-data.txt. However, count-bytes.pl only saw the first 8192 lines (65536 bytes).

For completeness, I wrote ipc-pipe-test.pl which uses perl's native 'open' function to run count-bytes.pl (open($fh, "|perl count-bytes.pl")).

$ perl ipc-pipe-test.pl
10000 lines, 80000 bytes
$

I'm running Fedora 28, perl5 (revision 5 version 26 subversion 2), IPC::Run says $VERSION = '0.99'.

Have I made some mistake or am I misunderstanding the IPC::Run documentation?

[ntyni]

Hi, I believe we've hit this issue too (though the reporter seems to have forgot to attach the actual ipc-run-test.pl file relevant to the testcase.

My test case would be

#!/usr/bin/perl
use IPC::Run;
use IO::Handle;
use strict;

my $count = shift || 10000;
my $file  = shift || '/usr/share/dict/words';
my $fh = IO::Handle->new;

IPC::Run::run(["dd", "if=$file", "bs=1", "count=$count"], ">pipe", \*$fh);
my $got = 0;
while (<$fh>) {
    $got += length;
}
print "Got: $got bytes\n";

which takes the number of bytes to read from a largish file (/usr/share/dict/words by default) and passes them through a pipe, then reads them out. When given an argument of >65536 bytes, this hangs for me.

The more or less equivalent version without IPC::Run works fine:

#!/usr/bin/perl
use strict;
use IO::Handle;

my $count = shift || 10000;
my $file  = shift || '/usr/share/dict/words';
my $fh = IO::Handle->new;

open($fh, '-|') || exec "dd", "if=$file", "bs=1", "count=$count";
my $got = 0;
while (<$fh>) {
    $got += length;
}
print "Got: $got bytes\n";

It looks to me like IPC::Run::_write() should write in smaller chunks to avoid flooding the kernel pipe buffer and blocking on the write. Sorry, no patch at the moment.

[ntyni]

The case where I ran into this is https://bugs.debian.org/961368