Goroutine leaks in zstd initialization

[pkwarren]

Versions

Please specify real version numbers or git SHAs, not just "Latest" since that changes fairly regularly.

Sarama	Kafka	Go
1.29.1	n/a	1.17.1

Configuration

What configuration values are you using for Sarama and Kafka?

n/a

Logs

When filing an issue please provide logs from Sarama and Kafka if at all
possible. You can set sarama.Logger to a log.Logger to capture Sarama debug
output.

logs: CLICK ME

 Goroutine 29 in state chan receive, with github.com/klauspost/compress/zstd.(*blockDec).startDecoder on top of the stack:
goroutine 29 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00020fa00)
        /Users/pkw/go/pkg/mod/github.com/klauspost/compress@v1.13.6/zstd/blockdec.go:212 +0xec
created by github.com/klauspost/compress/zstd.newBlockDec
        /Users/pkw/go/pkg/mod/github.com/klauspost/compress@v1.13.6/zstd/blockdec.go:118 +0x1b1

 Goroutine 30 in state chan receive, with github.com/klauspost/compress/zstd.(*blockDec).startDecoder on top of the stack:
goroutine 30 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc00020fad0)
        /Users/pkw/go/pkg/mod/github.com/klauspost/compress@v1.13.6/zstd/blockdec.go:212 +0xec
created by github.com/klauspost/compress/zstd.newBlockDec
        /Users/pkw/go/pkg/mod/github.com/klauspost/compress@v1.13.6/zstd/blockdec.go:118 +0x1b1

Problem Description

The fix for #1869 changed from initializing the zstd variables on demand (with sync.Once) to in init. Behind the scenes, this causes goroutines to be started (and calling .Close() on Sarama resources doesn't propagate through to calling .Close() on the zstd instances).

This causes tests that verify no leaked goroutines (i.e. using https://github.com/uber-go/goleak) to start to fail if they use Sarama.

[pkwarren]

With the default configuration, this causes a leak of runtime.NumCPU() goroutines.

[subbu05]

I am also seeing this issue, please check.

Logs
Goroutine 14 in state chan receive, with github.com/klauspost/compress/zstd.(*blockDec).startDecoder on top of the stack:
goroutine 14 [chan receive]:
github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0000e6340)
/Users/s/go/pkg/mod/github.com/klauspost/compress@v1.12.2/zstd/blockdec.go:215 +0x149
created by github.com/klauspost/compress/zstd.newBlockDec
/Users/s/go/pkg/mod/github.com/klauspost/compress@v1.12.2/zstd/blockdec.go:118 +0x173
]
FAIL gecgithub01.walmart.com/pulse/loglibrary 0.757s
FAIL

[dnwe]

Yes, this was sort of a known problem flagged up under #1937 and is something that we'd like to fix (by having the encoders/decoders owned on a per client basis).

However, the "leaked" goroutines aren't so much of a problem by themselves, they're not increasing in number and they are only created once at package initialization. We use github.com/fortytw2/leaktest ourselves and its per-test snapshot-based approach isn't concerned by them.

[github-actions]

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur.
Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.