Building FlatBuffers in an embedded environment.

[randyK-HH]

I would like to use FlatBuffers in an embedded system and am a little unsure of what the best way to do this is.

I have built the flatcc compiler on a desktop computer (Mac) with no problem and have been able to compile and test the (sample) monster schema properly there. So far so good! Now I want to build the flatcc libraries for an ARM Cortex M0 processor so that I can use them in an embedded application.

Any general advice on how to do this?

[mikkelfj]

So, flatcc is designed for this use case, but I rely on users to test various embedded devices.
I believe systems similar to the one you mention have used flatcc successfully.

Essentiall you just compile the runtime parts for your target system either as a library, or add the few runtime C files to your project along with your application. I will not go into how you can get CMake to cross compile and you can also just compile with a shell script if so inclined (I often do that for testing small projects). You may want to look at the compiler flags set in the CMake project because gcc can be a bit loud on some warnings, but you do not have to use the CMake system that flatcc builds with.

The main problems typically arise from not using a full standard library and in particular when malloc and friends are not available. FreeRTOS systems have compiled and most of these issues have been ironed out, but the <flatcc/portable> sub library can be updated if you run into specific problems, and allocation functions can be redefined via macros.

Another problem arises for extremely RAM limited use cases. Here the builders internal allocation system may have to be rewritten via a pluggable allocator so it can allocate from a fixed memory block or similar. This is possible but a bit involved.

[randyK-HH]

When I #define PORTABLE_DEBUG_ALIGNED_ALLOC=1 as a compiler argument, error messages I get are:

Building file: ../builder.c
Invoking: Cross ARM C Compiler
In file included from C:\Workspaces\freertos_retarget/flatcc/portable/pstdalign.h:160:0,
                 from ../flatcc/flatcc_flatbuffers.h:24,
                 from ../flatcc/flatcc_builder.h:68,
                 from ../builder.c:19:
C:\Workspaces\freertos_retarget/flatcc/portable/paligned_alloc.h:111:2: error: #error "DEBUG: C11_ALIGNED_ALLOC configured"
 #error "DEBUG: C11_ALIGNED_ALLOC configured"
  ^
In file included from C:\Workspaces\freertos_retarget/flatcc/flatcc_alloc.h:72:0,
                 from ../flatcc/flatcc_flatbuffers.h:26,
                 from ../flatcc/flatcc_builder.h:68,
                 from ../builder.c:19:
C:\Workspaces\freertos_retarget/flatcc/portable/paligned_alloc.h:111:2: error: #error "DEBUG: C11_ALIGNED_ALLOC configured"
 #error "DEBUG: C11_ALIGNED_ALLOC configured"

When I #define PORTABLE_C11_ALIGNED_ALLOC=0 (and remove #define PORTABLE_DEBUG_ALIGNED_ALLOC), the project does appear to compile successfully. That seems good! Next step is to try it out.

[randyK-HH]

One minor snag I am encountering now is from the linker. When I call: flatcc_builder_init(&builder); from my main routine I get:

Building file: ../builder.c
Invoking: Cross ARM C Compiler
Finished building: ../builder.c
 
Building file: ../main.c
Invoking: Cross ARM C Compiler
Finished building: ../main.c
 
Building target: freertos_retarget.elf
Invoking: Cross ARM C Linker
./builder.o: In function `flatcc_builder_init':
C:\Workspaces\<PATH REMOVED FOR BREVITY>/../builder.c:425: undefined reference to `flatcc_emitter'
collect2.exe: error: ld returned 1 exit status
make: *** [freertos_retarget.elf] Error 1

I can see flatcc_emitter is a function declaration (ie: no body) at the bottom of flatcc_emitter.h. That seems suspicious and seems like it might be part of the issue.

I also see there is a struct defined in flatcc_emitter.h with the name flatcc_emitter.

[mikkelfj]

It is because I forgot that you also need the emitter.c file when building buffers. It is pluggable module that controls how the output buffer is allocated and emitter.c provides the default implemenation.

[randyK-HH]

That was indeed the issue! Compiles/links successfully. Thank you so much for the help!

Upon running, I noticed that I had to increase my heap size to prevent a crash which I believe was associated with memory allocation. I boosted it from 0x0600 (1536) to 0x2000 (8192). Nothing magical about the 0x2000, I just arbitrarily picked a bigger value.

Previously, you mentioned 2-4K was sufficient for small projects. Does this new heap value seem consistent with your expectation?

[mikkelfj]

Previously, you mentioned 2-4K was sufficient for small projects. Does this new heap value seem consistent with your expectation?

You can try to reduce the default page size of the emitter. It appears to be 3K.

The change the page size define e.g.: -DFLATCC_EMITTER_PAGE_SIZE=1024
see also
https://github.com/dvidelabs/flatcc/blob/master/include/flatcc/flatcc_emitter.h

And you make sure you only use direct_buffer when reading the buffer to avoid allocating a congtigous buffer for the result: flatcc_builder_get_direct_buffer
However!!! it only works for small buffers about half a page size. Otherwise it will return null, and multiple pages will have been allocated. The emitter starts in the middle of a page in order to extend in both directions for reasons I will not go into here. But you can ask again if you want to optimize this.

The above pertains to the emitter which builds up the buffer in reusable pages. In addition there is a number of small allocations for the builder stack - even if they are initially small, the system allocator may also reserve extra memory for each allocated size, so it is hard to tell exactly what the memory requirements are for a given system.

You can look at flatcc_builder_default_alloc in builder.c - each hint option provides an initial allocation size. You can copy and modify the allocator and give to the builders extended init function. Then you can debug how much allocation is needed and modify the allocator accordingly.