NSMgr leaks memory somewhere (probably in grpcfd)

[Bolodya1997]

Current Behavior

NSMgr leaks memory somewhere (probably in grpcfd) after long time working.

nsmgr-8g8qn.pb.gz

Steps to Reproduce

1. Patch NSMgr with the following lines in main.go:

import _ "net/http/pprof"

go func() {
	_ = http.ListenAndServe("127.0.0.1:8080", nil)
}()

2. Run some example for a long time (12+ hours).
3. Collect heap profile with go tools pprof.

[Bolodya1997]

Tried to run NSM without any grpcfd either in networkservice or in registry chains. Almost no leaks for 3 days running setup.
nsmgr-b6r9b.pb.gz
So it keeps looking like the cause of the leaks is somewhere in grpcfd.

In inuse_objects sample I can see registry server.Find leaks in spanlogger, there is already an issue for this - networkservicemesh/sdk#1085.

In inuse_space sample there is a possible leak in gRPC transport.NewBufWriter (~1Mb), but there are only 17 objects, so it can be a normal behavior, needs to be further monitored.

[edwarnicke]

@Bolodya1997 grpcfd should be cleaning up after open file descriptors on connection close

https://github.com/edwarnicke/grpcfd/blob/aa90c1562fb36d3f274db049354d6a1e76fd325a/connwrap.go#L102-L122

[edwarnicke]

@Bolodya1997 in recvfd we are caching files and closing them when the client Connection closes: https://github.com/networkservicemesh/sdk/blob/204761283a51aa10efcf3d994c36c74b50355e26/pkg/networkservice/common/mechanisms/recvfd/client.go#L88-L94

But we appear to be neglecting to do so on the server side:

https://github.com/networkservicemesh/sdk/blob/204761283a51aa10efcf3d994c36c74b50355e26/pkg/networkservice/common/mechanisms/recvfd/server.go#L80-L109

In principle on the Server side once https://github.com/networkservicemesh/sdk/blob/204761283a51aa10efcf3d994c36c74b50355e26/pkg/networkservice/common/mechanisms/recvfd/server.go#L92 happens the files should eventually get closed due to finalization by the garbage collector... but explicit is better than implicit.

[Bolodya1997]

@edwarnicke
There was an issue with finalizer, fixed it - edwarnicke/grpcfd#19.

[edwarnicke]

@Bolodya1997 And released in https://github.com/edwarnicke/grpcfd/releases/tag/v0.1.1 :) Many thanks :)