Writing into a stream

High-level method dump

Multiple protobuf messages can be written into a file or any output stream by calling dump function. In this example, messages is a list of protobuf messages which are written into the specified output stream/file:

import stream

# dump by file name
stream.dump('test.gam', *messages)

# dump by file object
stream.dump(f, *messages)

---

NOTE for vg users

The new version of stream library, now as a part of libvgio, writes a header at the start of the stream depending on the output format. For example, headers like b'GAM' or b'VG' can be found before the actual protobuf messages in GAM and VG files repectively. In order to keep the generated files by pyStream compatible with the new version of stream, you should write the expected header value to the file using header keyword argument; e.g. stream.dump('test.gam', *messages, header=b'GAM') for GAM files and header=b'VG' for VG files (introduced in v1.6.2).

---

---

NOTE

After dumping a set of messages, the stream gets closed. Another call for dump will rewrite the output file unless the underlying file is not opened by the stream; i.e. the stream is created by providing a file object rather than a file name.

---

Low-level method open

This method opens an output stream and return iterable Stream object. It gets more control over the output stream as it does not get closed after each writing. So, the output file can be appended. This example show how to append two list of messages to an output stream:

import stream

# open by file name in append ('ab') mode
with stream.open('test.gam', 'ab') as ostream:
    ostream.write(*messages)
    ostream.write(*another_messages)

# open by file object `f` in append ('ab') mode
# NOTE that file-like object `f` is passed as a keyword argument `fileobj`
with stream.open(fileobj=f, 'ab') as ostream:
    ostream.write(*messages)
    ostream.write(*another_messages)

---

NOTE

A stream can be explicitly closed by calling close method. Particularly, a stream must be closed when it is opened without using context manager (with statement).

---

Writing non-protobuf serializable objects

Since v1.6.3, not only protobuf messages but also any serializable objects can be written to output streams. You can provide a custom serialize function using serialize keyword argument to any API methods or Stream class. The given method should take an object and return its serialized byte string.

As a special case, byte arrays can be written to the stream using serialize=lambda x:x. For example:

with stream.open('file.out', 'wb', serialize=lambda x:x) as ostream:
    ostream.write(b'hello')
    ostream.write(b'world!')

The resulting file would contain two objects b'hello', and b'world!' of bytes type.

Header

Since v1.6.2, a header can be written at the start of output streams by providing the header argument to any API functions (both dump and write functions or when a Stream is instantiated). Headers are byte strings which can be helpful for detecting the type of upcoming messages in the stream.

Since v1.6.4, a stream can write the header tag before each message group if option persistent_header is set. It can be useful for indicating what type of messages is upcoming.

Optional GZip compression

The streams encoded by Stream library is GZip compressed. The compression can be disabled by passing gzip=False when opening an stream (works for both dump and open methods).

Buffered write

By default, the buffer size is zero meaning no buffering. In this case, all protobuf messages provided on each write call are written in a group of messages (see Stream library for encoding details). The messages can be buffered and written into a stream in a fixed-size group by setting buffer size to a non-zero value when opening an stream. The size of the buffer can be set by keyword argument buffer_size passed to open or dump methods.

Grouping message

Messages can be grouped in varied size when writing to a stream by setting buffer size sufficiently large or infinity (-1) and calling flush method of Stream class whenever desired.