Consider using flatbuffers or protobufs to send things over the FFI (Bookmarks, in particular)

[thomcc]

I'm wary of using JSON to send anything big over the FFI for perf reasons. It's also extremely tedious to parse out the JSON on the Java/Swift side.

This would let us generate the code for all three. The downside is that it's less seamless in Rust than using Serde.

[rfk]

I feel like @bbangert will have some useful comments on this one...

[bbangert]

There's some serde options still....
https://github.com/dflemstr/serde-protobuf

[thomcc]

Serialization is not yet implemented in this version.

We'd need this.

That said, I don't think the process for compiling the protobuf to rust in a build.rs would be a problem, that sort of thing is an intended use case for build.rs.

[st3fan]

@thomcc @rfk @csadilek crazy idea - even with JNI/JNA there is a good amount of overhead right? what if we replaced FFI with a (UNIX) socket? Or use any of the other high perf Linux IPC methods? Maybe that combined with protobufs / serde?

[thomcc]

That’s certainly an option, but I don’t think it would make many of the things we currently have difficulty with any easier, and we do take advantage of sharing an address space/direct calls in several places, which would have to be worked around (doable, but awkward).

I also don’t think it would reduce the overhead, which is mostly serialization and string encoding overhead in two directions (something like protobufs, hopefully, would combine encoding snd serialization, but thats true regardless of a socket being involved)

[bbangert]

@st3fan I had discussed something like that with @thomcc a week or so ago as well. And there's been some jokes about effectively doing local gRPC (I've been looking at gRPC for our service API's) as well.

One thing that looks like a more ideal solution would be something like Djinni:
https://github.com/dropbox/djinni

Which was created by DropBox to ease their cross-platform toolkit in C++ to bind to Java/Objective-C. We would need one that is for Rust of course, so its not a perfect fit, but the solution feels like it fits in the problem-space here. And for bonus points, our ideal version would also tackle ownership issues. ;)

Barring a resource investment into getting a solution like that for ourselves, I do think protobuf is a decent choice here. It's not without its pain points, but its probably one of the better supported tools out there.

[linabutler]

Protobuf seems like a good choice. Servo uses Bincode, but AFAIK there aren't any Rust or Swift libraries for it... CBOR looks interesting, too, and has implementations for Swift and Kotlin...

[eoger]

Did some exploration this afternoon, here's what I found out:

• I Want (some) Safety: if I add a field to a struct that's passed over the FFI, I want our build process to guarantee that the Kotlin/Swift side is aware the struct shape has changed. This is where schemas and build-time code generation shine in my opinion. It seems like CBOR/Bincode/MessagePack don't provide schema compilers as they are focused on simplicity.
• Djinni, as Ben mentioned, doesn't have Rust bindings and I'm not quite sure we want to invest time in that.
• Flatbuffers/Cap'n Proto seemed very promising, from benches they both seems fast. However, most probably because they try hard to work around memory allocation, their Rust APIs could be more ergonomic:
  • capnproto-rust
  • flatbuffers
• Protobuffs seem like a good trade-off:
  • Kotlin/Swift/Rust code generators.
  • Simple Rust API: classic structs + encode/decode to/from byte buffers.
  • It's fast enough.

I have a strong preference for Protobuffs version 2 over 3, simply because version 3 doesn't have optional types (you can somewhat emulate them) and instead serializes default values ("", 0, false etc.).
proto2 should be supported "pretty much forever" so I'm not worried about that either.

Feel free to correct my statement as this is the result of two hours of exploration of the problem space :)

Thoughts?

[rfk]

what if we replaced FFI with a (UNIX) socket?

One of the ideas that came out of the Tofino project was to run a kind of local "user-data agent" in a separate process, that would manage your places db and other things and provide access over a service API. This would have some interesting advantages (remove FFI ownership/safety issues, force asynchronicity, privilege isolation) as well as disadvantages (managing lifecycle of the agent process, round-trip efficiency) but could be worth exploring.

But this would also feel like Conway's Law writ large.

One thing that looks like a more ideal solution would be something like Djinni:
https://github.com/dropbox/djinni

Agreed; I feel like our wild-blue-sky-ambition would be wind up with something that looks like a cross between wasm-bindgen and Djinni, using annotations on rust code to generate strongly-typed and fast bindings for our target languages. Moving from JSON to protobuf seems like a good step in that general direction, so 👍 from my perspective.

[eoger]

Closed by #626

[vladikoff]

Great work!! 👍

[OpenMIS]

使用Cap'n proto、FlatBuffers等数据类型作为链接库的函数的参数，达到自动生成代码的，高性能、自动化解决方案。

设计想法
1> 使用Cap'n proto、FlatBuffers等无序列化性能消耗的，不建议选择Protobufs、Json等序列化会消耗性能的格式。
优先采用Cap'n proto，它的性能最好。
2> 使用指针，可以达到零拷贝。
3> 提供两个通用函数
网关函数(char*)
释放内存函数(char*)

Using Cap'n proto, flatbuffers and other data types as the parameters of the function of the link library to achieve a high-performance and automatic solution for automatically generating code.
Design idea
1> For disordered serialization using cap'n proto, flatbuffers and other formats, it is not recommended to select formats that will consume performance such as protobuffs and JSON serialization.
Cap'n proto is preferred, and its performance is the best.
2> Using pointers, zero copies can be achieved.
3> Two general functions are provided
Gateway function (char *)
Free memory function (char *)