Porting to C

[OxMarco]

A question more than an issue.
Can Reticulum be ported to C? It would benefit all those uC like the ESP32 or the NRF52840 that power many low power LoRa transceivers. Micropython is quite expensive CPU-wise and increases power consumption.
What are the main problems to be faced when porting it to C? I can do it myself but I'll need support throughout the process.

[markqvist]

Yes, it can, and it will, given that I have the resource to do it. It has always been my intention to create a version in C. When I have added the last few features to the Python reference implementation, and I lock in the API, I will start work on the C implementation.

[markqvist]

Also, I would very much appreciate help on this endeavour, so if you want to join in, you are very welcome.

I do however think it’s necessary to wait until the Python implementation is done, the API has been locked in, and the documentation has been completed.

[OxMarco]

Okay ping me once you're ready to create the C version. I've listed some interesting ideas on the Meshtastic forum.

[OxMarco]

Are you still maintaining it?

[blademaster98]

Are you still maintaining it?

hard to tell , no updating since last August. Let's find out is there anything we can do?

[OxMarco]

Well first of all check if the current version is reliable enough to be used, then start porting it to C using a proper HAL in order to support as many embedded platforms as possible.

[blademaster98]

Well first of all check if the current version is reliable enough to be used, then start porting it to C using a proper HAL in order to support as many embedded platforms as possible.

sounds like a plan

[markqvist]

I am still very much maintaining Reticulum. Life has not been too generous with time lately though, and I've had to focus on things more related to survival than passions. Although I had a goal of having Reticulum in an API-stable beta release by the first day of 2021, it did not pan out that way, and I was severely delayed. Things have calmed down for me now though, and for the next few months, I should be able to complete that work.

I'd really strongly suggest against starting a C-port right now. There's still a few crucial details that need locking in, and doing this without the API documentation I'm currently writing would be a frustrating experience.

I would very much appreciate the help once it is time, but please have just a little more patience :) I know it's been a long wait, but it will be worth it very soon.

[markqvist]

And in the meantime I am open for any discussion on the matter in this thread. I will be available to answer questions and discuss it here.

[blademaster98]

And in the meantime I am open for any discussion on the matter in this thread. I will be available to answer questions and discuss it here.

How nice, happy to know you were back. It's ok, we will wait for your docs and lock in the few crucial details

[markqvist]

Hello everyone who is still listening here. Today, I released Reticulum 0.2.0. This is the first release marked as being in beta state. This also means the API documentation and manual has been published at: https://markqvist.github.io/Reticulum/manual/.

In a sense, one could at this point start writing a C port. In the service of being as transparent as possible, I will have to say that I am planning a very big change to the internals of Reticulum though, which will greatly impact any implementations.

I am 95% sure that I will change Reticulum to completely use Elliptic Curve Cryptography on Curve25519 for all assymetric cryptography operations. This will mean RSA is no longer needed, which has been a very limiting factor in the project so far. By using Ed25519 signatures and X25519 ECDH key derivation instead of RSA, protocol overhead will be greatly reduced, and security will be staggeringly enhanced. It really seems like a win-win, but when I worked on the theory behind Reticulum 5 years ago, I didn't know enough about elliptic curves to utilize them confidently, and as such chose to go with RSA-1024 instead.

I'm currently researching the last 5% of doubt I have about this decision, and one of the topics I'm looking into is the feasibility of running X25519/Ed25519 on low-power MCUs and what C implementations exist.

I am very interested to hear anyones thoughts on this.

[OxMarco]

Hey, first of all nice to have you back and to see Reticulum is under development.
Elliptic curves are perfectly supported by most embedded platforms, there are tons of libraries for Arduino, ESPxxx and other popular uC with limited computing resources.
I'm a bit confused by the retransmission parameter m, why don't you append it to the message itself so that other nodes can check how many times it has been rebroadcasted?

Going to the actual task of porting it to C, we need to define some common guidelines beforehand, such as using size-constrained integers (uint16_t, int32_t, ...) and possibly avoid float and double values that may lead to rounding and endianness problems.
Another part of our preliminary analysis would be to check how many dynamically allocated vectors have been used in the Python code, generally for embedded systems there should always be a well-defined max value to preserve system stability.
Finally, as we want to make it as general as possible, I'd recommend using a RTOS (ChibiOS, FreeRTOS, ...) to handle threading and parallel tasks.

[markqvist]

The m parameter is not appended into the message, because it is already there in the header field ;) Byte two of the header field is the "hops" byte, which tells how many hops a packet has taken, which is also true for announces. I updated the "wire format" section in the manual to make it more clear.

Ok, I just finished moving everything to elliptic curves. Was actually a lot less work than I had anticipated. And as you say, it seems there is good support on microcontrollers and in C, so this is going to be the way forward. Protocol overhead has also been greatly reduced now. It's all very, very nice.

What you're saying regarding locking in the typing definitely makes sense, as does having a way to control dynamic vector allocation. I'm going to take a break for a couple of days now, cause I'm literally exhausted ;) When I start again, I will start mapping out the outline of the C version.

I'm quite hesitant to use a RTOS, since I've been down that road before a long time ago, and in the end it ended up being much more hassle than help. My current stance is that making the C port as dependency free as possible is the way to go. We'll need crypto libraries, yes, but that should be more or less it. Task scheduling and async io should be possible with an internal job handler, since the internal "transport machine" of Reticulum is actually pretty simple. You're welcome to convince me otherwise though :)

[blademaster98]

Happy to hear that. I am here still listening

[OxMarco]

Ok moving forward, I've created a pull request to act as WIP for us all. Other general things we need to think about before starting are:

• inline logging: Serial print, file log, other?
• classes: do we keep them or use a purely functional C-style?
• string and number manipulation: any good replacement for native Python libraries?

Small OT: have you seen the Helium project (www.helium.com)?

[samuk]

Rweather now has HKDF rweather/arduinolibs@4429613

[markqvist]

That is great news, it's going to speed up things a bit :)

[4c3e]

I am also interested in helping here. My primary use for a C port would be to make it easier to use RNS in C programs targeting Linux on x86 and ARM/RISCV. Unfortunately I don't have much experience with embedded systems, but would like to help make that possible as well.

Unfortunately, there doesnt seem to be an existing Fernet C implementation so I think I will start by trying to implement that.

^https://github.com/4c3e/fernet-c/tree/main

Would love input on choosing the crypto and utility libraries 👍

Currently I'm looking at libsodium and tiny-aes for crypto

edit: I'm probably going to spend time working on a python i2p interface implementation before diving in to a complete library rewrite in C 😄

[OtisByron]

I'm an electronics engineer, not a software engineer, but I'm watching this very closely. RNS and associated software and hardware make up a brilliant system that I'm very excited about.

I was hopeful that a C port would allow RNS to run on a microcontroller system like Arduino in order to minimize power consumption for remote repeater systems. Solar powering an RNode indefinitely would be a lot easier than using a Raspberry Pi (I've been dreaming about updated RNode firmware that would allow it to be a self contained RNS repeater, but I'm not sure that there is enough memory/horsepower in the Atmega). Any thoughts about whether something like this is on the C port road map or if it's even feasible?

Edit:
I am aware of the previous conversation in this thread about uC vs Raspberry Pi. My question is specifically about using the RNode hardware to run RNS as a standalone system.

[markqvist]

@4c3e and @OtisByron, thanks for joining here and offering your help and interest. The Reticulum C port is getting closer by the day. Currently, there is only one protocol-level addition that needs to be added to the Python implementation, before I am willing to commit to work on the C version (namely an ordered data mode for links). When that is done, and after a round of testing, I think it will be feasible to start the work.

@4c3e, if you manage to do a portable C implementation of Fernet, that would help the effort along tremendously. Many of the other components exist in various forms already, some don’t. Have a look at the other comments in this thread, if you didn’t already, there is some good resources regarding crypto libs. Either way, the help is welcome.

@OtisByron, regarding the feasibility of Reticulum on very resource-limited hardware like RNode, it is my current position that it is feasible, yes. A device like an RNode will not be able to handle very deep networks with tens of thousands of nodes, due to RAM limitations, but for smaller setups, it should be sufficient as a standalone system. On larger systems with a bit more RAM and CPU, it should also be possible to handle larger networks, but then you also get the higher power consumption, so I really understand where you are coming from with that. One of the design goals of RNode was to use as little power as possible, exactly for purposes like this.

I hope this gives a little more clarification about the current status and intents. If you have any questions, ideas or insights, I’d love to hear them.

Thanks.

[markqvist]

Since this is not an issue per se, but now a pretty long and informative repository on the C-porting effort, I am moving this to a discussion.

[Narashiman]

Hi. I recently came across this cryptoPP library. I believe it would be possible to implement Fernet easily by using this library.
https://github.com/weidai11/cryptopp

[samuk]

Is there a sense of where this fits into the overall roadmap?

I'm interested in building solar networks, doing that in a cost-effective way requires a version that can run on microcontrollers.

[markqvist]

Ok, I will have something more concrete info on this in the pretty near future, but we are getting there soon, I promise. For now there is a much more coherent roadmap of the entire current state of Reticulum available here: https://github.com/markqvist/Reticulum/blob/master/Roadmap.md

The very short version is that to do this properly, and to ensure the future viability of everything related to Reticulum, I need to secure funding for this work. The development of this entire ecosystem is currently more than a full-time job, and believe it or not, so far all development has been completely unsponsored, apart from the (very much appreciated donations) from people like you. These help a great deal in paying for expenses, but I really do need to eat and pay bills at some point too ;)

I'm working on this, and will hopefully have more info to share in the near future. If anyone has ideas on that front in the meantime, I am also very interested in hearing them!

[khimaros]

@markqvist this is really exciting! i'm curious if you've considered a memory safe language as an alternative to C/C++. Rust comes to mind because it has a small VM (and no garbage collector), but there are others such as Elixir or Go depending on what the requirements are. for security critical software, this seems like it would be of benefit.

[khimaros]

i've been exploring this space a bit recently and will share a bit about what i've found. i'm happy to dive into any of this, answer questions, do more research, etc. if that would be helpful.

there has been a lot of progress in rust-embedded and it is now supported in the stable release. the rust-embedded book is a great starting point. https://docs.rust-embedded.org/book/ (the book assumes an STM32 based MCU, but there are also great ESP resources)

there are various RTOS support for Rust, such as in NuttX: https://lupyuen.github.io/articles/rust2#rust-meets-nuttx and FreeRTOS: https://github.com/lobaro/FreeRTOS-rust and RIOT: https://doc.riot-os.org/using-rust.html

it is possible to run "bare metal" without the standard library or any of the Rust virtual machine present. in other words, your code is the very first thing invoked by the bootloader. it should be possible to get as "small as C" in practical terms.

calling into Rust from C/C++ and generating bindings for almost any other language (including JS/WASM) is also well supported.

could you dive in a bit more to what the requirements are, which platforms/peripherals you want to target, etc?

more resources here: https://github.com/rust-embedded/awesome-embedded-rust

[markqvist]

Hmm, that is really interesting. It seems like things are much further along than what I thought. Last time I looked at this, which is admittedly quite some time ago, this was not the case.

The great thing about a portable C implementation is that it can target almost anything that has the resources to run it. But the resources are key here, so we are probably going to be looking at something like devices with a 64MHz clock, 128K of RAM as a baseline minimum for a system that can usefully run a RNS implementation, while also doing a useful amount of other stuff. That points in the direction of ARM Corex M3, M4 and up, ESP devices and possibly RISC-V systems as well.

One thing that is pretty important to me at this point is to ensure being able to link the implementation with firmwares/binaries generated with the Zephyr RTOS. A quick search turned up this as apparently being no problem.

This is very interesting, and an avenue I will have to explore much more extensively, thanks for informing me about the recent developments.

[ellenhp]

Or maybe someone else will take it upon them to lead the development of a Rust implementation :)

We'll see how far I get before I burn out, but I just started poking at a Rust port a few days ago. I'm primarily targeting ESP32s and other platforms with std, so this won't be suitable for most micros, but reticulum requires an RTC and a random number generator and other goodies that aren't available on no_std targets anyway. I have a lot of boilerplate more or less working but the protocol implementation is WIP. Is there a written spec somewhere? I've just been going from the code and the docs. The docs are already pretty good but a formal spec would be amazing. The code is python, so there are no type annotations which is making it hard to read.

https://github.com/ellenhp/reticulum-rs

[markqvist]

Awesome effort @ellenhp!

Wait, you can't just infer what types everything is from my perfectly logical code?! I thought I had made it so clear! :P

Joking aside, at the moment the Python code is the written spec. At least, it's the best we have ;) I agree, it's not the same as a formal written spec, but this havin been more or less a one-man effort until recently, this was the only way for me to make somehting as complex as this entire system happen. So view the Python code as the authority on implementation and protocol for now. I can be on standby to answer any questions or ambiguities there might be.

Now that the reference implementation is more or less complete though, I might actually have time to write that formal spec and get the docs overhauled a bit to make all this stuff more clear.

In regards to the MCU implementation, here's a few suggestions and ideas:

• It's probably useful to get the basic Identities, Destinations, Packets and Announce mechanisms working first. With those running, the implementation can already be tested against the Python reference.
• When that is functional, I suggest implementing Links, then Resources, then Requests/Responses.
• I'd suggest waiting with Channel and Buffer mechanisms until last, since they are the newest additions to protocol suite, and are still prone to go through a few growing pains (I fixed some serious performance issues just two days ago, for example). They are also the least useful on micro platforms, in my opinion (since well-delimited message/request approaches are a better fit for most situations where a MCU is deployed).
• Regarding the Resource implementation, there may be certain complexities we need to deal with when you get there. The Python implementation slices Resources up into chunks of up to 16 megabytes for compression handling and efficient encryption. This is probably too much for most MCUs to handle, even ESP32 and the likes. We may need to add a mechanism to either disable compression, or to signal the need for smaller chunks. But let's solve it when we get there.
• I'd suggest writing the full Transport Instance support (ie., routing) last. Even without the ability to route packets for other nodes, the MCU implementation will be 95% functional. In the real world, I think the amount of situations where you want to use a small MCU as a router are very, very limited. It is extremely useful in a few niche cases though, so I definitely welcome the intention! I just think it's worth prioritising here, to give this the maximum chance of success :)

Again, thanks for your effort!

[ellenhp]

Good to know, thank you! I'm most of the way through the first few things you suggest working on. I think the path response stuff would be grouped under the announce mechanisms though, and that's not complete, and there are probably a few things I haven't seen yet in the codebase related to announces. I'm also expecting to need to refactor the persistence code a few more times before I'm happy with it. It's hard because I can't just throw everything in sqlite or something.

I do have some specific questions:

• For delivering a basic message end to end is it as simple as constructing a packet and pushing it out over the wire? I have the encryption step set up to automatically occur when a WirePacket is constructed, and if it's received by the correct node, it should be able to decrypt the payload. I have the encryption and decryption stuff all built and tested thanks to the fernet and rustcrypto crates.
• Header 2 packets (with a transport field) are used when a packet has been forwarded, to indicate who forwarded it, right?

One thing I want to call out: The ESP32 has about 250k of SRAM I think, but some of the modules come with multiple megabytes of PSRAM that you can access via the MMU and a separate allocator. I know 4MiB is a common amount and they sell WROVER modules with as much as 8MiB. So it's possible that a chunk size of a megabyte or two may be fine.

[rafael2k]

That is really cool! C rocks!

[markqvist]

Yes, you can't imagine how long I have been looking forward to getting this ball rolling :)

[attermann]

Just recently found Reticulum. Very compelling project. Amazing that you've done all of this solo @markqvist. I'd love to be able to help push the µRNS effort along. Nice to see @ellenhp tackling the rust implementation. I assume the plan is to still have a C/C++ implementation as well, is that right? If so and you still don't have the bandwidth yourself I'd be willing to take a stab at it. Just want to be sure I wouldn't be stepping on any toes.

[Erethon]

Also meant to ask; is there a better forum for discussing this than this thread? A discourse or slack channel would be nice (if you don't already have one) to encourage more dialog with the community as things evolve.

There's the matrix channel in #reticulum:matrix.org.

[attermann]

LoRa issue is resolved. After much grief I realized that I had missed setting the preamble length 🤪. I now have a fully autonomous (admittedly incomplete but functioning) MCU LoRa transport node 🎉.

[faragher]

Amazing! There are plenty of people who could find use from a pure repeater even without any other functionality. Well done.

[SebastianObi]

Wow, this is really great. Many people have been waiting for a Reticulm ESP32 transport node for a long time. Now you can operate low-power outdoor LoRa repeaters/nodes. I am waiting for the functional firmware for testing.

[attermann]

OK, if you insist... but remember that you asked for it 😉.

https://github.com/attermann/reticulum-mcu

This is super basic experimental firmware that I'm using on a T-Beam currently, but should be able to be configured for other LilyGo LoRa boards as well without a lot of effort. I was able to use this firmware to bridge two UDP nodes over LoRa by enabling the T-Beam UDP Interface (see README) and send a path request and subsequent data between the two nodes.

It goes without saying that this is very experimental software so all of the usual caveats apply. Looking forward to feedback.

[rafael2k]

Yay, this is just wonderful, a C++ reticulum! I'll test soon in a Raspberry Pi 4 running Linux.
: ))

[ion232]

It's great to see the work that's been done on other implementations of Reticulum. Currently I'm working on an implementation of Reticulum in Zig. For those unfamiliar with Zig - here are some reasons why it's a great choice.

• It's intended to be a better C, which allows it to perform the same functions but with better safety, performance and ergonomics.
• Much of the std library can / will work without requiring a specific runtime like libc, etc, which makes targeting bare metal much easier.
• There's a ZigEmbeddedGroup that maintains microzig, which is essentially an abstraction over embedded devices. There is already support for the RP2040, which I'm planning to target and test on.
• The Zig std library currently has a kitchen sink of various functionality, which includes most of the crypto needed for Reticulum, and I have been using this for now, so that currently I don't need any external dependencies.
• Zig has compatability with C, so C code can be used in Zig, and Zig can be transpiled to C. This means I should be able to directly leverage mbedTLS / OpenSSL, without needing bindings.

The more implementations the better in my opinion.
I'll push up my repo once I have identity working and welcome any thoughts or questions people have in the meantime.

[ellenhp]

About TRNG, are you referring to the fact that the ROSC on the RP2040 isn't necessarily secure enough?

Yep. It's fine for testing but I think they specifically recommend not pulling entropy from it for cryptographic purposes because nobody's really looked at how much of its randomness comes from things that an attacker could predict. Right now I'm targeting an STM32L4 with a TRNG because it's a pretty low cost option. And yeah, nothing against Zig but a lot of people, myself included, see the CoC as woefully inadequate because it only covers interactions in official spaces. So like, sexually harassing another contributor via email is explicitly allowed by the CoC because it covers only official spaces. Also it's against the CoC to discuss the CoC which puzzles me. Lots of weird things, I don't mean to make a point of it but I've been sexually harassed by co-maintainers of FOSS projects before so it's a thing I look for when I engage with a new project or technology.

[markqvist]

Fantastic to hear @ion232! The more languages the merrier! I must say I have no experience with Zig, so I will only be able to provide general pointers and help in regards to the implementation, but feel free to ask any questions that pop up here on the forum, or on the Matrix channel!

But now, the pertinent question... Is it possible to move zig?

[faragher]

But now, the pertinent question... Is it possible to move zig?

You know what you doing.

[xtian]

@ion232 Is your Zig work ready to be public yet?

[ion232]

@ion232 Is your Zig work ready to be public yet?

I'm afraid not, I put it on hold while taking care of other priorities. I do intend to finish it though and get it working on the Pico W. Did you have a particular use case, or just checking the status of it?

[rafael2k]

That is great news!! Right now as a user (hopefully will be able to start helping with code at some point), we support whatever decision to put Reticulum in place running native code!
: ))