TBS: "Value log GC request rejected" error

[carsonip]

Errors observed in the wild: message "tail sampler aborted" with error.message "Value log GC request rejected".

Due to #14337 , it is possible to have 2 TBS GC goroutine running concurrently, which will cause one of them to crash. In a hot reload under EA, if the new server's GC crashed, and the old server's GC later stopped as part of the reload, this will leave the server with no GC and the TBS storage size will keep growing.

APM Server version (apm-server version): observed on 8.14.3

[carsonip]

This should be caused by #14337 . Closing this issue to focus on fixing the root cause there.

[carsonip]

However, there's one open question on under what circumstances will cleanup run before gc, as cleanup should always run after Execute.

The error is not due to cleanup before gc, but 2 concurrent gc.