Endpoint AI/ TinyML for Toit!

[skye0402]

It would be great to have a library that supports ONNX or TFlite. TinyML is fast growing and companies like Edgeimpulse do likewise. On ESP32 you can use TFlite for microcontrollers or ONNX through WebAssembly.
Toit already provides the concepts needed to manage ML models on devices. Happy to work together with a use case.

[kasperl]

This is a great idea, @skye0402! Where would you suggest we start?

[skye0402]

@kasperl Good point 😄

I did some research and here are my findings:

I propose to focus on TensorFlow Lite format. For that, multiple running examples for ESP32 already exist.
Let's assume Toit could handle a TFLite format: It would cover the complete TF world that can be converted to TFLite. Models that can't are not designed for mobile devices or MCUs anyway. Also with the help of ONNX other formats can be cross-converted to TFLite. I was not able to find a framework for ONNX on MCUs itself.

I suggest to look into these recent examples made for Arduino Nano 33 BLE Sense - a device with just 256kB RAM and 1MB flash.

So the main challenge is to either write a tflite format interpreter (meaning a library that can read and run the models) in Toit or make use of the existing C++ code of the TFLite for MCUs and transfer this with a Toit-wrapper into Toit.

Happy to further exchange on that.

[kasperl]

@skye0402 Thanks!

I'd recommend starting with the existing TFLite C++ code and finding a way of conditionally compiling the support into the Toit VM (ifdefs). With that, it should be relatively easy to expose a few primitives to high-level Toit code (primitives can be optional through weak linking) and going with that. Next step could be to add some nice tooling to jag to make it easy to push new models to a device while you iterate on them.

[aditi16690]

It would be great to have support to run ONNX model in a toit environment. Right now my code is in C which unpacks and run inferences using onnx model. If toit provides such a support for onnx then we will be able to take full advantage of the capabilities provided by toit.

[wifwucite]

It would be nice to integrate with Edge Impulse also for collecting data. The easiest way would be through logging over serial, but a full integration via web sockets would be great. :-)