Span logger leaks memory in infinite watching server.Find case

[Bolodya1997]

Current Behavior

In infinite watching server.Find case span logger just collects logs and doesn't push them. It leads to unlimited memory growth in registry server.

Issue in code

In infinite case defer never happens:

sdk/pkg/registry/core/trace/nse_registry.go

Lines 87 to 88 in c7b15ba

	ctx, finish := withLog(ctx, operation)
	defer finish()

pprof output

registry-k8s-5b77d84b85-lgxn4.pb.gz

Possible solution

Instead of having single span in such case we can handle such hierarchy:

[root span] - empty
[child span 1] .. [child span n] - with data

So all of [child spans] will refer to the single [root span] and will be pushed with some timeout or with some amount of logs collected. [root span] will always be empty so there will be no memory leak.

[denis-tingaikin]

Could you please attach something from logs, build links, pictures, source code links from your research?

[Bolodya1997]

Could you please attach something from logs, build links, pictures, source code links from your research?

Added link to code and pprof output.

[Mixaster995]

Tried to investigate this problem - can't reproduce leaks.
Probably requires really long-time running setup or some special tweaks imitating leaks.
Since problem don't look critical at this moment, so i postponed investigation
Prepared draft PR with possible solution that should fix this problem - #1108