Support Storm's custom serialization support for multi-lang. [msgpack]

[amontalenti]

A recently merged pull request 84 on the Storm project implements custom serialization support for the multi-lang protocol. This means that you can technically register other faster serializers than JSON for Storm's tuple data, e.g. we could use msgpack.

In addition, thanks to Storm's built-in support for serialization hooks with Kyro, we could actually switch Tuple serialization from Kyro's built-in JVM-based serializer to something like msgpack. This would potentially allow us to avoid the encode/decode cost associated with using ShellBolts and ShellSpouts.

It would be great for someone from the community to do a prototype of this using a streamparse project, and prove its viability. Then we could merge the support into the core library, build a support library in Java for the custom serializers, and offer documentation for how to set up Storm to leverage them. This will likely fit in very well with our lein-based Storm project management.

[jsgilmore]

Hi, good to see someone uses my serialisation pull request. I've already implemented a protocol buffer serialization http://www.google.com/url?q=https://github.com/jsgilmore/protoshell&sa=U&ei=jDlvU4DGKsbx0gX9vYGQDQ&ved=0CAsQFjAA&usg=AFQjCNEo9x4r3QZJPuDSzv4PD-gSz0VNMA that you could also use for improved throughput.

[amontalenti]

@jsgilmore Thank you. That's pretty interesting. Looks like you're using this with GoStorm?

[amontalenti]

Also here are links to protoshell and gostorm for reference.

[dan-blanchard]

Now that Pyleus is out and has support for msgpack serialization, it seems like we could pretty easily adapt their code for this (since it's Apache licensed, and that should be compatible with BSD).

Here are the relevant links:

• Java MessagePackSerializer
• serializers Python sub-package

[amontalenti]

Indeed, this is a good idea. Thanks for the links, @dan-blanchard.

[rduplain]

Related to #174.

[dan-blanchard]

pystorm/pystorm#1 addresses this.

[dan-blanchard]

This issue was moved to pystorm/pystorm#5