Can the number of bytes used in a builder object be estimated/returned before the buffer is closed out?

[randyK-HH]

Can the number of bytes used in a builder object be estimated/returned before the buffer is ended or closed out?

For example, let's say I have a simple table:

table MyData
{
    data_1 : [uint8];
    data_2 : [uint8];
}

Over time I want to append data to both data_1 and data_2.

My C code might start with:

ns(MyData_data_1_start(B));

And then I call this from time to time where I am appending new data onto one (or both) of my table array fields.

ns(MyData_data_1_append(B, arbitrary_data_vec, arbitrary_data_size));

or

ns(MyData_data_2_append(B, arbitrary_data_vec, arbitrary_data_size));

I know that when I "end" the data_1 field I can call "flatcc_builder_get_buffer_size() to get the size of the final serialized buffer. So far so good.

IE:

ns(MyData_data_1_end(B));

final_size = flatcc_builder_get_buffer_size(B);
printf("Final Size = %u\n", (uint32_t) final_size);

However, without counting bytes myself (which is my fallback plan), can I get an estimate of how many bytes will be consumed BEFORE I call:

ns(MyData_data_1_end(B));

I tried calling "flatcc_builder_get_buffer_size()" prior to ending and it returns 0.

And actually, one additional related side question. Is it possible to "start" two separate table fields, append to them both over time, and then "end" them both sometime later? Or does the builder require us to close out one table field before starting another?

[mikkelfj]

To your last side question first. Unlike other languages, flatcc allows for having multiple tables open at once, however, they must be nested. You cannot have an arbitrary number of unrelated tables open at once. This is because all unclosed tables share the same stack.

The stack leads to your first question:

Get buffer size should be valid before the end of the entire buffer, but maybe not that exact function call you refer to - there is an end and start method (or something to that effect - look at the source) used extensively internally. These are the virtual start and end offset that grow up and down and eventally extends to the entire buffer and defines the buffer size. But these only grow when data is emitted:

Flatcc builds buffers in two stages: on the stack via the builder object, and in the emitter that maintains final buffer data (even if not physically assembled into a single buffer yet, it maintains a virtual address space that precisely reflects the buffer size so far).

So, for things not yet emitted, you do not get the size data that you are seeking. Things are generally emitted when ended as you have come to suspect.

The build stage is sometimes skipped if the input data can be directly emitted, such as a native little endian integer array, or a string, but conceptually data flows the same way - it is just an optimization - you are still ending something.

So, if you want more insight, you need to watch the stack space. This is not trivial because there are about 7 or different kinds of stacks for different purposes, but the ds stack is the most relevant. All stacks use their current active size as stack pointer because it is robust to reallocation. You find the stack pointers in struct inside the builder, as I recall. While unlikely to change, direct access to these data is undocumented and subject to change.

You want to be particularly careful with nested buffers, notably not dragging references across boundaries, but by and large nested buffers work as child objects same as always, only alignment is really tricky to get right for the builder. Hence size estimates shouldn't be much different.

[randyK-HH]

Let me try to summarize here. Please tell me if you agree/disagree with anything I wrote.

1. From my earlier example, it does not appear that I can have both simple data structures "data_1" and "data_2" open at the same time. In other words, I must start and end one before appending to the other. I attempted to do this in a simple C program and got an assertion failure.

2. If "data_2" was a nested table then we could write/append to both "data_1" and "data_2" concurrently. I have not tried this specifically but am thinking it might look something like below. Is this what you were getting at? Please correct my example as you see fit.

table SubData
{
    data_2 : [uint8];
}

table MyData
{
    data_1 : [uint8];
    sub_data : [SubData];
}

3. Alternatively, we could create separate builders and write/append to "data_1" and "data_2" independently, then clone/copy the data from one builder over to the second.

4. Or alternative to that, store data in my own memory caches and write it all at the end. (This is what I am trying to avoid.)

5. Buffer size information can be estimated before "ending" by viewing stack offset variables inside the builder object (e.g. "B->ds_offset") but this is not exact because it doesn't account for possible final padding bytes that are sometimes added. I did take a look at this one variable for my particular example and it does look accurate.

[mikkelfj]

EDIT: I misread you text. Point 2. is also correct because you introduced nesting. I was thinking of data_1 and data_2 as fields in the same table. Text below is kept, but keep this correction in mind. EDIT EDIT: see next two comments, it does not help to have a sub-table, but I still misread the point 2. text in the following answer.

All but 2. are correct. Nested buffers operate on the same stack - except - you can be build a seperate non-nested table, finish, at add it atomically as a nested table to another buffer as a binary array. But for flatcc's dynamic nested table support the same limitations remain.

Nit pick: data_1 and data_2 can not be updated concurrently in the sense that you think of, but technically they are fields in a table and all fields can be written concurrently until the table closes (there is a stack to one to track this by allocating a temp max vtable size for each nested table), even rewritten if certain compile time flags are configured AFAIR, but data_1 and data_2 are just offset values (or builder references at build time). The content of those references either have to be defined in advance, or nested, which precludes updating contents of vectors or other sub-structures concurrently (including nested buffers).

Another point of estimation precision is the cost of vtable overhead since vtables are reused if possible (not possible across nested buffer boundaries or when table fields are not utilized consistently across table instances). This can largely be ignored, except if you have a vector of tables where the vtable overhead might be significant, or trivial (e.g. a table with 3 optional fields such as x, y and z coordinates where most vtables would be reusable).

[mikkelfj]

If you read this through mail, please note the import EDIT in previous comment.

Another point: structs can use fixed length arrays and all struct content can be written concurrently until the top level struct is closed. So if you have a case where you might have two sensor readings collected concurrently with up to, say, 10 readings each, you could just fill in the arrays as data arrives and the store a sentinel end value at the last element or similar. This won't save space (necessarily), but can be convenient.

[mikkelfj]

Brain fart:
NO: point 2. is still wrong. You cannot update data_1 and data_2 concurrently even if they are in separate nested tables. This is because you cannot have the sub_data table open at the same time as data_1. Ultimately this follows from the fact that there is only one stack context per open buffer.

[mikkelfj]

Something somewhat different that might have your interest in this context: you can build several buffers concurrently as long as you use separate builder objects. When a buffer is completed, you can use it to clone data into another buffer. While this is extra data copying, it should still be fairly efficient if not abused. You can use this to collect long running data via append that eventually needs to land in another buffer, but you cannot keep appending in a single buffer due to the constraints mentioned. You can cache data in your own memory, or use FlatBuffer to do that management in a separate buffer.

[mikkelfj]

Also, I could be more specific in my previous answer, but I don't recall all the details up front. builder.h is reasonably commented and should be a good starting point.