Skip to content

Commit

Permalink
Added capability for "Direct Piping" - that is, transferring data bet…
Browse files Browse the repository at this point in the history
…ween

2 processes directly using OS pipes and connecting them with dup2, instead
of buffering the output of process 1 into python memory before sending to
process 2.

This is in response to issue amoffat#119 amoffat#119

The effect can be enabled by setting call arg _piping="direct" on the inner process.
I had issues with getting 'Input/Output error reading stdin' from dd, until
I set _tty_out=False. In the code I have therefore set _tty_out to False when
_piping == "direct". This has caused 3 of the unit tests to fail.

test2.py on my PC shows 13s vs 0.5s performance difference
  • Loading branch information
mcclymont committed Oct 9, 2013
1 parent 80af572 commit 7279f1d
Showing 1 changed file with 28 additions and 0 deletions.
28 changes: 28 additions & 0 deletions test2.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/python
import sh
import time
import sys

class Timer:
def __enter__(self):
self.start = time.time()
return self

def __exit__(self, *args):
self.end = time.time()
print "Time taken: %s" % (self.end - self.start)

def debug(line):
sys.stdout.write(line)

print "Original piping method:"
with Timer() as t:
sh.dd(sh.dd("if=/dev/zero", "bs=1M", 'count=512', _piped="out", _err=debug), "of=/dev/null")

print "Direct piping method:"
with Timer() as t:
sh.dd(sh.dd("if=/dev/zero", "bs=1M", 'count=512', _piped="direct", _err=debug), "of=/dev/null")


print 'done'
sys.exit()

0 comments on commit 7279f1d

Please sign in to comment.