Question : Database with DatabaseOpenFlags.DuplicatesSort how to store data bigger than 512 bytes ?

[valeriob]

Hi,
i'm having hard time to find docs about it, but it looks like that when database is opened with DatabaseOpenFlags.DuplicatesSort there is a limit on the data value that can be stored, at last i reach an exception when the value is more thatn 512 bytes.

Thank you

[AlgorithmsAreCool]

Hmm, typically you should only get this error if the key size if the problem. Not the value size.

How large are your keys? (i'll take a peek at other causes in the mean time)

[AlgorithmsAreCool]

Actually, nvm. I see some code in LMDB that can cause this, i'm trying to interpret it

[AlgorithmsAreCool]

So, I've confirmed that when running under DUPSORT the max size of a value is limited to the max key size (normally 511 bytes). It is an internal LMDB thing 🤷‍♀️.

See also:

https://github.com/LMDB/lmdb/blob/4b6154340c27d03592b8824646a3bc4eb7ab61f5/libraries/liblmdb/mdb.c#L7599
mozilla/rkv#49 (comment)

Recommended workaround is to have a 2nd table/database that is not DUPSORT.

Database A: DUPSORT
Key : KeyA
Value : KeyB

Database 2 : Not DUPSORT
Key : KeyB
Value : BigValue

[AlgorithmsAreCool]

I apparently also made a note about this in my other issue. I guess I forgot about it 🙃.

#130

[valeriob]

Thank you very much for your confirmation, i think i've been burned by this in the past but apparenty i forgot too 😄
I guess i'll need to make my key longer, it's probably for the best since i guess that with dupsort there will be some cpu spent on that data, still for time series data it was usefull.

[AlgorithmsAreCool]

Let me know if you have any other questions

[valeriob]

Thank you very much :D