🙋 Android bindings

[bltavares]

Hi there (not sure if the appropriate repo to do this)

Thanks for the project, it seems to be moving quite well. The idea of a dat component in Rust is quite nice as it could reach many devices.

One platform that interests me is Android. I've been looking at staltz/dat-installer and this is quite a nice example of possible app on Android that would benefit a lot of having datrs available as a lib. I would like to help with providing such capabilities for Android, but I'm not sure how to help.

I have no experience on bindings and Android NDK's FFI, but I'm interested to be able to use dat on mobile applications. A bonus point would be to provide a Flutter plugin which would allow cross-platform development as well.

What would be the next steps for this?

[yoshuawuyts]

@bltavares heya! -- thanks for the question! I've been thinking about this as well, and I think there's more or less two takes on this:

The Short Haul

In the short run we could probabably get something working on Android, by creating bindings using cbindgen (tutorial), and then manually binding that to Android.

Probably a first goal would be to get an in-memory version of Hypercore working first. But then as we build out the networking layer we can build that part out too.

This can probably already be experimented with -- probably by adding C support through cbbindgen first (PRs would be super welcome!), and later adding more android-specific bits too.

The Long Haul

The manual binding part of the step above is probably going to be the biggest bottleneck for a good workflow.

Ideally we could create a similar experience to cbindgen / wasm-bindgen, where all that's needed to create Android bindings is to slap an attribute on the thing we want to export.

This would require a good amount of work, but would be massively beneficial to the Rust ecosystem in general. Perhaps something like this exists already, but I haven't seen it yet.

Conclusion

Hopefully this sketches an idea of how to approach this. The story here is by no means smooth, but I think that it can be pulled off, and improved as progress is made. I'm super encouraging if you want to dive into Hypercore, and try and create C/Android bindings. It'd be super cool to see how far you could get! (I think quite far, actually!)

Is this something you would want to work on?

Further Reading

• Rust-Kotlin FFI Example
• Example Service for the above bindings
• android-rs-glue - seems discontinued, but might be a useful reference

[bltavares]

I would like to help with this, certainly! But it is a type of work which I'm not used to, so I'm not sure how to take the first steps towards it.

The long haul approach would be quite powerful for the community as well, and I wonder if there would be more interest around this.

I'll read more about NDK and FFI on Rust, and see how much I can go forward, but I would love if other s could help as well, as this is quite new to me :)

[bltavares]

(To centralize information on the issue)

After a quick search, there are more projects related to Java/JNI that could help:

• https://github.com/sfackler/rust-jni-sys
• https://github.com/Geal/rust_on_mobile
• https://github.com/Dushistov/rust_swig
• https://github.com/Monnoroch/rust-jni

[luizirber]

Hi,

I have a bit of experience using cbindgen to generate C headers for FFI and using then in Python, and I've been paying attention to this repo to check what would be the right time to have an FFI available for other languages (I really want dat working in Python!).

I don't have any Android/JNI experience, but maybe it would be viable to share some code with cbindgen/FFI?

[yoshuawuyts]

@luizirber having a C API would be a fantastic first step. It'd be so good to allow Dat work with Python to allow people to run it in their data pipeline.

I'd be happy to help land any FFI patches needed -- if you'd like to start experimenting with this that would be fantastic! Any contributions to move the C API along would be great! 🎉

[bltavares]

Maybe we could make a meta-issue to track the implementation of the C-API.

If we could reimplement the nodejs API using the bindings, I think we would be able to use it for Python, Java and so on.

I was thinking more of the organization of it as I'm reading more about FFI, but please correct if I'm wrong:

• Create a hypercore::ffi module to expose the C interface without messing with the Rust-itc code
• Create two new cargo proviles ffi.release and ffi.debug to produce the cdylibs to be used
• Expose the header files on the project as well

And for the projects:

• Create a new Python project that wraps around the .so (should this live in a new repo?)
• Create a new Java project that wraps around the .so (should this live in a new repo?)
  • Create a Flutter plugin that consumes the Java project (this would be my end-goal)

Looking at the project, this seems to expose the main part of the Dat protocol, but not the dat cli. Should we also expose a C API for what would be the datrs CLI, or each platform would also implement it's dat API on top of hypercore's C API?

[bltavares]

Reading more about cargo we wouldn't be able to override the build section using profiles, so probably we would need to create a new FFI project to integrate with cbindgen and build.rs.

Maybe using a workspace would be better if we want to keep the FFI part on the same project, but I'm not sure what is the best strategy for such kind of projects

[luizirber]

for sourmash-rust I've been keeping everything in the same crate, with an sourmash::ffi module (and a sourmash::wasm too). It is better to avoid putting FFI-specific things (like the #[no_mangle] attribute) in the Rust code, and it was even recommended during RustConf

I think the FFI code should go in this repo, and then have separate Python and Java projects that use the FFI code. In the Python side you can use milksnake to make it easy to deal with setuptools and CFFI.

[bltavares]

Considering this presentation it might not be possible to reuse the same C FFI for JNI integration for the Java part.

[yoshuawuyts]

@bltavares It looks like it should be possible (working production example of a C FFI targeting Android), but the JNI crate linked in the presentation looks really good too!

Perhaps it would be useful to try using jni for the android bindings directly, and expose methods with ffi. @bltavares would you like to experiment with using jni?

---

@luizirber I like that approach a lot; thanks for linking all the resources too! -- a PR to setup an FFI structure would be an amazing start!

[bltavares]

Thanks for pointing out to the example. Looking further it seems to use JNA instead of JNI to avoid the complexity of making the functions match what the JVM expects as symbol names.

As I mentioned before, I don't have much experience with those and I had to search, coming to understand that JNA is slower than JNI, but I think it would be ok for this moment to be slow if it means sharing the C interface to kick of the FFI efforts.

I would like to experiment with either jni or ffi as a Java lib, and I'll probably put some time this week to get a hello world running.

(Sorry to be over-posting. I'm trying to coordinate efforts and get feedback as early on as possible to move to the right direction)

[luizirber]

@bltavares: I looked into the JNA example and it seems pretty similar to what happens in Python and CFFI.

I keep pushing for FFI because the same interface can be used for any language that support FFI. For my use case there are solutions that are closer to Python (like PyO3), but I would like to give access to my lib to other languages too (like R or Julia) without having to write R-specific Rust bindings.

(BTW, sorry for hijacking the issue...)

[bltavares]

Hi, just to keep everyone in the loop.

I was able to spend some time today fighting setting up a Gradle based project that uses a export "C" fn from Rust.

It is very rough, but I'm already able to:

• Build a Java lib wrapping a .so
  • Use the Java lib from another Java CLI project
• Build an Android lib wrapping the Java lib
  • This is mainly to be able to add Android specific dependencies and configs
• Compile the APK, but it crashes because it is missing the correct x86_64-linux-android/libhypercore.so, as I'm only packaging a win-x86-64/hypercore.dll

I'm currently developing on Windows, and cross-compilation is kind of a pain.
Next week I'm planning to use trust to build the desired .so that will be wrapped.

Thinking about this, maybe the next steps for hypercore could be:

• Create a hypercore::ffi module (as already suggest)
• Setup trust to build and publish each .so for many platforms (Windows/Linux/Mac/Android/iOS)
• Create a new project for each language binding (already proposed also), which uses the published bindings from GitHub Releases

Having the published .so for all desired architectures would help a lot creating the bindings, specially as the tooling for cross-compilation is not easy to get right. What do you think?

[yoshuawuyts]

Setup trust to build and publish each .so for many platforms (Windows/Linux/Mac/Android/iOS)

You can use crossgen for this. Opened up an issue to target libraries specifically too yoshuawuyts/crossgen#11 ✨

[bltavares]

That is neat. I'll probably give it a try to use crossgen and send a PR. First (next week probably), I'll fork the project and mess with the trust template to see what would be an ideal setup and come back to this issue.

[yoshuawuyts]

Found another, recently updated topic on internals: https://internals.rust-lang.org/t/idea-first-class-android-ndk-integration/5057