FlexBuffers in flatcc

[randyK-HH]

Is FlexBuffers supported in flatcc? At first glance it appears that the answer is no but perhaps there is something equivalent instead. My goal is to store some large byte vectors as efficiently as possible without the overhead that comes with the standard FlatBuffers tables.

[mikkelfj]

No, there is no flexbuffer support. You can get dynamic typing using a custom flatbuffer scheme with unions but not super space efficient, and not what you are looking for here.

If you have large byte vectors, the table overhead will not matter much relative to the vector size. Expect roughly around 12 bytes of per table overhead for pointer to vtable, actual vtable content, offset to byte vector, and padding. I doubt FlexBuffers will be much smaller since it needs to specify a name and a type for each field, and probably some offsets as well.

[randyK-HH]

Indeed FlexBuffers was not really the solution I was looking for. It does appear that using structs rather than multiple tables in a schema is a way to reduce some overhead. (ie: StructBuffers)

Two questions:

1. Is there any way around having a fixed or hard-coded array size for a array of data? See example below for byte_data entry.
2. Is there any way to suppress the padding bytes that sometimes get inserted into the struct data when the struct buffer is being created? (In the example below, a 0x00 byte will be inserted between var5 and var6.)

struct MyData {
    var1    : uint8;    
    var2    : uint8;
    var3    : uint8;
    var4    : uint8;
    var5    : uint8;

    var6    : uint16;   
    
    //byte_data : [uint8];      // Table Entry (need not be fixed size)

    byte_data : [uint8:50];  // Struct Entry (needs to be fixed size?)
}

[mikkelfj]

No, Flatbuffer structs behave almost exactly like C structs. That is the answer to both of your questions.

You can remove some padding by arranging fields differently. A 16-bit field will always start at an offset that is a multiple of 2 bytes, and padding will be inserted if that is not the case. C compilers do the same thing. But the rules are not hard and fast for C compilers so there can be platform differences. Flatcc StructBuffers guarantees a consistent layout across platforms, and can also handle endian translation between little and big endian systems.

The padding is normally forced to contain zero, but you can mess it up depending on how you call the flatcc builder, since it allows you to provide your own zeroed struct.

Struct buffers only work with flatcc, other languages do not support them. But you can have a small root table containing the struct. That will of course add some overhead.

[mikkelfj]

If you are more sensitive to space than to access speed and processing overhead, protocol buffers would be a better choice.

[randyK-HH]

Thank you for the responses!