receive: v0.16.0-rc.0 regression. Memory leak.

[bwplotka]

Fun! 🙃

[bwplotka]

Root cause is visible on profiles and indicates #2783

It's snappy and recv gRPC parser holding memory. Either something is wrong with out manual "yolo" serializing / deserializing or bytes from gRPC serialized are still referenced somewhere and never cleaned during those snappy and parsing operations. ):

File: thanos
Type: inuse_space
Time: Oct 1, 2020 at 11:41am (UTC)
Showing nodes accounting for 8247.98MB, 100% of 8247.98MB total
---------------------------------------------------------+------------
flat flat% sum% cum cum% calls calls% + context
---------------------------------------------------------+------------
3021.97MB 100% | google.golang.org/grpc.recvAndDecompress /go/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:641
3021.97MB 36.64% 36.64% 3021.97MB 36.64% | google.golang.org/grpc.(*parser).recvMsg /go/pkg/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:528
---------------------------------------------------------+------------
2833.95MB 100% | github.com/thanos-io/thanos/pkg/receive.(*Handler).receiveHTTP /go/src/github.com/thanos-io/thanos/pkg/receive/handler.go:291
2833.95MB 34.36% 71.00% 2833.95MB 34.36% | github.com/golang/snappy.Decode /go/pkg/mod/github.com/golang/snappy@v0.0.1/decode.go:63
---------------------------------------------------------+------------

I reverted just this commit and here we are:

[bwplotka]

Wonder @pstibrany and @pracucci if you seen something like this on your yoloString impl, but I most likely overdid something (:

cc @brancz

[bwplotka]

I hoped I can escale without full blown benchmarks on Series gRPC + Serializing, but not really ;p

[pstibrany]

Wonder @pstibrany and @pracucci if you seen something like this on your yoloString impl, but I most likely overdid something (:

I don't think so, but it's something to keep an eye on.

[bwplotka]

Fixed on release branch: #3327

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗