Don't over-allocate invoice bytes #3494
Conversation
When allocating space for an invoice's bytes, it was assumed that the bytes used for the invoice request signature are the same length as those used for the invoice's signature. However, fuzz testing revealed that this isn't always the case since an invoice request could contain more than one signature TLV. Account for this when determining the number of bytes to allocate for the invoice. This comes at the expense of an additional traversal of all TLVs in the invoice request through the end of SIGNATURE_TYPES (i.e., every TLV except experimental ones).
|
Can we just always allocate a fixed 512/1024/2048/4096 bytes and call it a day? Over-allocating for an object that's not gonna stick around long seems fine (and over-allocating by <2x is generally pretty fine as it could reduce memory fragmentation to offset the additional allocated size. |
|
Sure, though we still need to calculate the size and round to the nearest power of 2, if I understand what you're proposing. |
When allocating Vec<u8> for BOLT12 messages, use allocations that are powers of two starting at 512 instead of using the exact number of bytes provided. This helps reduce the amount of heap fragmentation.
This avoids heap fragmentation as it will avoid re-allocations when using OfferBuilder and RefundBuilder, though it may over-allocate.
|
FYI, I made the adjustment. The first two commits will need to be squashed if this looks good. Did the same for when building offers and refunds. |
|
Oh, no, I was proposing we allocate enough space for any reasonable invreq and if we under-allocate cause someone is doing something nuts that's okay. |
Gotcha. Note that the failing fuzzer was from creating an invoice from an invreq, though. |
|
Err, either way yea. If this code is gonna cause problems I vote we replace with a constant. Sadly for Of course if you prefer to fix it and keep it, that's okay with me, you're the one writing the PR :) |
When allocating space for an invoice's bytes, it was assumed that the bytes used for the invoice request signature are the same length as those used for the invoice's signature. However, fuzz testing revealed that this isn't always the case since an invoice request could contain more than one signature TLV. Account for this when determining the number of bytes to allocate for the invoice. This comes at the expense of an additional traversal of all TLVs in the invoice request through the end of
SIGNATURE_TYPES(i.e., every TLV except experimental ones).