support variable length ZId

[LizardWizzard]

Involves some workarounds for serde, which is currently does not support
arrays with length expressed as a const generic parameter. This
workaround might be useful in other places if or when we'll want to use
arrays with const generics.

[hlinnaka]

What do we need variable length Zids for?

[LizardWizzard]

@hlinnaka see #1228 (comment)

[hlinnaka]

What do we need variable length Zids for?

Ok, I see, you want to use a shorter ID for "ZNodeId". I'd suggest just using an int8 for those. And decimal representation would feel more natural too, if they're assigned sequentially rather than by random.

[LizardWizzard]

Actually I think 128bit ids are not that big, since we use one per whole machine (#1228 (comment))

Regarding int8 I think it is too small. But anyway, IMO it is more convenient to use one style for all of our ids (timeline, tenant). They are meant to be managed externally by console, so there may be not much sense in random values for all of them then

[arssher]

What do we need variable length Zids for?

Ok, I see, you want to use a shorter ID for "ZNodeId". I'd suggest just using an int8 for those. And decimal representation would feel more natural too, if they're assigned sequentially rather than by random.

Hm, ok. I initially wanted to assign randomly, which has some pros -- centralized issuing is not required and they don't intersect between clusters, but I'm slightly bothered by very long (16 bytes) ids. I agree though random values be better 128 bits, so let's just assign int8 sequentially.

[LizardWizzard]

@arssher are we switching for timeline id and tenant id for sequential ones too? Do we really need two flavours of ids? isn't int8 too small for that? If we want to serve thousands of tenants on a single cluster consisting of multiple pageservers/safekeepers that seems to be not enough.

[arssher]

No, only node ids. There is actually some sense in having ids of different flavours -- you can guess the type by looking at it. I'm for that variant mostly due to shortness/readability/being easy to remember. int8 means 8 bytes, not bits, that's postgres terminology, so surely enough.

[LizardWizzard]

@arssher If we need to distinguish between entities that ids belong to, lets do that explicitly. We can reserve some of these id bits to some tag value and use it in Debug/Display impls. So id will look something like that: node/abcd12354 or tenant/1234abcd. IMO it is better than some heuristics. Also if our ids are too long, lets shrink them to minimal possible length. e g to have 8 bytes instead of 16

that's postgres terminology,

Good to know, thanks!