TSI Proposal

[benbjohnson]

Proposal for implementing #7151.

/cc @jwilder @pauldix @e-dard

[jwilder]

Is the local series ID the index position within the list? For example, if you had block ID 1, and ten series within the block, The global series ID would be (1,0), (1,1), (1,2), ... (1,9)? Then if you had series ID (1,4), you could go directly to block 1 in the file and then use 4 as the index into the KEYPOS list to find the offset in the block for that key?

I'm assuming this is the case, but it would be good to explain how this works in more detail.

[benbjohnson]

Yes, that's correct. I can add more detail.

[benbjohnson]

Detail added in 008dc1e.

[jwilder]

I think this is a good start and we should keep iterating on this. Some diagrams like these would really help to visualize the format of the file IMO.

[toddboom]

@benbjohnson gets to use Monodraw!

[e-dard]

I think we could consider storing the maximum and minimum tag value in the block. We could store them in the header, or we could avoid needing to them if we could easily jump to the first and last values in the block.

I would imagine if the tag values were container IDs for example, we could have millions of them, and a binary search for a value outside of [min, max] would be wasteful.

[benbjohnson]

@jwilder @e-dard @pauldix This proposal has been updated with the hash index and dictionary encoding from #7186.

[rw]

• How are the search structures created? In particular, what is the maximum memory usage when creating the TSI hash tables and/or sorted lists?
• Can we future proof by using uint64 instead of uint32 (to support more than 4 billion series)?