Return from throttler goroutine if context is cancelled to prevent goroutine leaks

[rohit-nayak-ps]

Description

goroutine profile: total 5614
5423 @ 0x43b3c5 0x4059fa 0x405755 0x1346ceb 0x4728e1
#	0x1346cea	vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer.(*vstreamer).parseEvents.func2+0xaa	vitess.io/vitess/go/vt/vttablet/tabletserver/vstreamer/vstreamer.go:279

There are reports of these leaking goroutines causing OOMs. I have not been able to reproduce it. I think this can only happen if vstreamer is failing to parse events resulting in the context being cancelled, with some events still present in the throttledEvents channel.
In the normal course of events, including errors that I can think of, we should not see this since the binlog streamer will close its channel causing the goroutine to return. So presumably there is an issue with this vreplication workflow causing these errors.

I have updated the goroutine to look for context cancellations so these goroutines don't leak, hoping that this will fix this issue. I would be happier if we could have reproduced it to ensure the fix works. For now we can try it in the failing workflow.

Signed-off-by: Rohit Nayak rohit@planetscale.com

Checklist

• Tests were added or are not required
• Documentation was added or is not required

[shlomi-noach]

The changes make sense. I will add another check for ctx.Done() right before continue, so that we don't spend an infinite loop waiting for throttler in case of lag, when the context is cancelled.

[shlomi-noach]

the changes look to be safe, and the code is better than before.

[shlomi-noach]

I've added an extra check for ctx.Done()