OpenMV, libmpix.org, OpenCV: joint effort

[josuah]

A few years back, there was only one project I could find that aimed implementing various image processing blocks on microcontrollers: OpenMV.

This project is great but the license does not allow it to be used outside of OpenMV hardware, so started https://libmpix.org/ under Apache-2.0.

It uses a streaming API: a 640x480 raw bayer frame (1 byte per pixel, 300 kBytes), can be converted to RGB (3 bytes per pixel, 900 kBytes) and compress the resulting frame to JPEG (variable size) without the need for allocating the 900 kBytes: intermediate buffers are just a few lines (like 3 kByte).

How to combine efforts of libmpix with this project?

• Shared SIMD accelerated implementation (for now design phase) low-levels
• Shared high-level API like tethering an API through another?
• Including a library into another?
• Shared nothing, letting the user select different libraries for different purpose? This still allow libraries to leave challenges solved by another library.

Let me know what you think works best.

[josuah]

Looking further, it seems like OpenMV license is MIT for all the image processing libs. It is just not packaged in a way that is made for use in 3rd-party repo, and did not want to copy-paste it as it felt like the wrong approach.

Maybe it is a good time to start packaging some shared blocks across multiple MCU projects:

• @bitbank2 libraries for all the media conversion utilities (JPEG, PNG, GIF... enc/dec)
• pixel format conversion (I was thinking of using CMSIS-DSP as much as possible for these, to allow making a RISC-V version of it in the future)
• classical ISP operations: debayer, image correction, pixel format conversion, resizing...)
• classical CV operations (edge detection, kernel processing like blur/sharpen/denoise)

All of which with extremely low-level APIs so that other projects can build their high-level API based on their need.

[bitbank2]

Beyond my imaging codecs, I can help with the SIMD optimization. Is there a budget/grant for this work?

[sfe-SparkFro]

Thanks for the ideas here, love it!

Maybe it is a good time to start packaging some shared blocks across multiple MCU projects:

• @bitbank2 libraries for all the media conversion utilities (JPEG, PNG, GIF... enc/dec)
• pixel format conversion (I was thinking of using CMSIS-DSP as much as possible for these, to allow making a RISC-V version of it in the future)
• classical ISP operations: debayer, image correction, pixel format conversion, resizing...)

I believe OpenCV has a HAL for low-level things like these, which we have not yet looked into, but would be great to take advantage of!

• classical CV operations (edge detection, kernel processing like blur/sharpen/denoise)

For high-level algorithms like these, we're going to stick with OpenCV's implementations, since the whole point of this project is to be a port of OpenCV. Changing the implementations of these high-level algorithms is out of scope for us.

Beyond my imaging codecs, I can help with the SIMD optimization. Is there a budget/grant for this work?

I don't think we have any funding for this, sorry.

A few years back, there was only one project I could find that aimed implementing various image processing blocks on microcontrollers: OpenMV.

This project is great but the license does not allow it to be used outside of OpenMV hardware

Yes, we're aware of OpenMV! I actually met with Kwabena (co-founder of OpenMV) in the concept phase of this project about the possibility of running the OpenMV firmware on SparkFun hardware. Long-story short, it could probably be done, but there were some technical challenges identified, and there's a different philosophy between OpenMV and our OpenCV project. OpenMV is focused on achieving maximum performance, which comes from careful hardware selection and highly-optimized algorithms that leverages the hardware (DSP, NPU, etc.). Our project is more focused on accessible vision processing on any hardware (still very early right now, but the structure is there) with a very popular and well known API (OpenCV).

Because of those different approaches, it wouldn't surprise me if MicroPython-OpenCV and OpenMV remain completely separate. I'll let @kwagyeman comment for OpenMV, but I don't think there's going to be much overlap. For high-level algorithms, we'll want to stick with OpenCV's implementations, and I expect OpenMV will want to stick with their implementations. There may be some overlap in low-level DSP functionality, but that may quickly become very hardware dependent. While I like the idea of having a single place for common algorithm implementation, I don't know if it will work out in practice.

[kwagyeman]

Hi all, yes, our focus is on maximum performance and software/hardware integration. Note that we are an OpenCV partner company nowadays. I think https://github.com/sparkfun/micropython-opencv is great! Note that our focus moving forward is on enabling AI applications.