Fix sending compound messages corruption when including messages larger than 64KB #260
Conversation
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
Signed-off-by: Marco Pracucci <marco@pracucci.com>
|
|
|
Sorry for the constant nudge, but just wondering if this PR looks good and can be merged soon? |
|
@markan any chance you would be able to take a look at this PR soon? |
| r, w := 0, 0 | ||
| for r < len(msgs) { | ||
| if len(msgs[r]) <= maxMsgLength { | ||
| // Keep it. | ||
| msgs[w] = msgs[r] | ||
| r++ | ||
| w++ | ||
| continue | ||
| } | ||
|
|
||
| // This message is a large one, so we send it alone. | ||
| bufs = append(bufs, bytes.NewBuffer(msgs[r])) | ||
| r++ | ||
| } | ||
| msgs = msgs[:w] |
There was a problem hiding this comment.
Maybe this instead?
| r, w := 0, 0 | |
| for r < len(msgs) { | |
| if len(msgs[r]) <= maxMsgLength { | |
| // Keep it. | |
| msgs[w] = msgs[r] | |
| r++ | |
| w++ | |
| continue | |
| } | |
| // This message is a large one, so we send it alone. | |
| bufs = append(bufs, bytes.NewBuffer(msgs[r])) | |
| r++ | |
| } | |
| msgs = msgs[:w] | |
| for r, b := range msgs { | |
| if len(b) <= maxMsgLength { | |
| // Keep it. | |
| msgs[w] = b | |
| w++ | |
| } else { | |
| // This message is a large one, so we send it alone. | |
| bufs = append(bufs, bytes.NewBuffer(b)) | |
| } | |
| } | |
| msgs = msgs[:w] |
Should be logically equivalent, but removes the need for the continue and captures that we always advance r each step, so it's instantly obvious that the loop completes and looks at each input exactly once.
Or maybe this: (adapted from https://github.com/golang/go/wiki/SliceTricks#filtering-without-allocating)
| r, w := 0, 0 | |
| for r < len(msgs) { | |
| if len(msgs[r]) <= maxMsgLength { | |
| // Keep it. | |
| msgs[w] = msgs[r] | |
| r++ | |
| w++ | |
| continue | |
| } | |
| // This message is a large one, so we send it alone. | |
| bufs = append(bufs, bytes.NewBuffer(msgs[r])) | |
| r++ | |
| } | |
| msgs = msgs[:w] | |
| w := a[:0] | |
| for _, b := range msgs { | |
| if len(b) <= maxMsgLength { | |
| w = append(w, b) | |
| } else { | |
| // This message is a large one, so we send it alone. | |
| bufs = append(bufs, bytes.NewBuffer(b)) | |
| } | |
| } | |
| msgs = w |
There was a problem hiding this comment.
Hey @alvinlin123 or @pracucci ,
Are you still able to follow up on these requested changes?
There was a problem hiding this comment.
Yes, let me try to see if I can get a hold of @pracucci :-)
|
No worries @markan, thank you so much for taking a look! |
|
@pracucci Do you think you could take a look? |
Cortex uses memberlist with a custom transport implementation, based on TCP protocol. We have a use case to send large messages, sometimes bigger than 64KB.
When memberlist implementation builds a compound message, it doesn't take in account the message length. Since in the compound message the length of an inner message is stored as a
uint16, the compound message gets corrupted if contains an inner message larger than 64KB.To solve this issue, I propose to not include messages larger than 64KB in a compound message.
Notes: