-
Notifications
You must be signed in to change notification settings - Fork 14
Writing into a stream
Multiple protobuf messages can be written into a file or any output stream by
calling dump
function. In this example, messages
is a list of protobuf
messages which are written into the specified output stream/file:
import stream
# dump by file name
stream.dump('test.gam', *messages)
# dump by file object
stream.dump(f, *messages)
NOTE for vg users
The new version of stream library, now as a part of
libvgio, writes a header at the start of
the stream depending on the output format. For example, headers like b'GAM'
or
b'VG'
can be found before the actual protobuf messages in GAM and VG files
repectively. In order to keep the generated files by pyStream compatible with
the new version of stream, you should write the expected header value to the
file using header
keyword argument; e.g.
stream.dump('test.gam', *messages, header=b'GAM')
for GAM files and
header=b'VG'
for VG files (introduced in v1.6.2).
NOTE
After dumping a set of messages, the stream gets closed. Another call for dump
will rewrite the output file unless the underlying file is not opened by the
stream; i.e. the stream is created by providing a file object rather than a file
name.
This method opens an output stream and return iterable Stream
object. It gets
more control over the output stream as it does not get closed after each
writing. So, the output file can be appended. This example show how to append
two list of messages to an output stream:
import stream
# open by file name in append ('ab') mode
with stream.open('test.gam', 'ab') as ostream:
ostream.write(*messages)
ostream.write(*another_messages)
# open by file object `f` in append ('ab') mode
# NOTE that file-like object `f` is passed as a keyword argument `fileobj`
with stream.open(fileobj=f, 'ab') as ostream:
ostream.write(*messages)
ostream.write(*another_messages)
NOTE
A stream can be explicitly closed by calling close
method. Particularly, a
stream must be closed when it is opened without using context manager (with
statement).
Since v1.6.3, not only protobuf messages but also any serializable objects can
be written to output streams. You can provide a custom serialize function using
serialize
keyword argument to any API methods or Stream class. The given
method should take an object and return its serialized byte string.
As a special case, byte arrays can be written to the stream using
serialize=lambda x:x
. For example:
with stream.open('file.out', 'wb', serialize=lambda x:x) as ostream:
ostream.write(b'hello')
ostream.write(b'world!')
The resulting file would contain two objects b'hello'
, and b'world!'
of
bytes
type.
Since v1.6.2, a header can be written at the start of output streams by
providing the header
argument to any API functions (both dump
and write
functions or when a Stream
is instantiated). Headers are byte strings which
can be helpful for detecting the type of upcoming messages in the stream.
Since v1.6.4, a stream can write the header tag before each message group if
option persistent_header
is set. It can be useful for indicating what type
of messages is upcoming.
The streams encoded by Stream library is
GZip compressed. The compression can be disabled by passing gzip=False
when
opening an stream (works for both dump
and open
methods).
By default, the buffer size is zero meaning no buffering. In this case, all
protobuf messages provided on each write
call are written in a
group of messages (see Stream library for
encoding details). The messages can be buffered and written into a stream in a
fixed-size group by setting buffer size to a non-zero value when opening an
stream. The size of the buffer can be set by keyword argument buffer_size
passed to open
or dump
methods.
Messages can be grouped in varied size when writing to a stream by setting
buffer size sufficiently large or infinity (-1) and calling flush
method
of Stream
class whenever desired.