Remove cgo zstd package

[klauspost]

Switch from cgo package to pure Go implementation. This will (when fixed) work even if no cgo is present.

https://github.com/klauspost/compress/tree/master/zstd#zstd

Compression level is removed. It could be made so the first request determines the compression level. But since there is currently only two levels (fast and default) I am not sure it is worth it.

This does NOT fix #1252 - only updates the code used for compression/decompression.

[klauspost]

CLA signed...

[bai]

Good timing, I just started to experiment with your lib yesterday to do exactly what this PR does. Thank you so much 👍

[d1egoaz]

this fixes:

• Building fails on windows with race flag #1197
• Link to DataDog/zstd breaks Cygwin #1215

[d1egoaz]

@bai might want to include your change to the build files, to disable explicitly cgo

[danielnelson]

We are running into issues with the new dependencies in influxdata/telegraf#6344 and this would solve that issue.

I notice there is a relatively large init function though: https://github.com/klauspost/compress/blob/master/zstd/fse_predefined.go#L72

If this could be ran on first use or manually that would be ideal for us, since a very low number of Telegraf users would use this functionality but everyone would need to run the init code on startup. I recognize this is mostly an issue with Telegraf but if these compression methods could be optional that would be very helpful for us.

[varun06]

--- FAIL: TestMessageEncoding (0.04s)
12041    request_test.go:34: Encoding empty zstd failed
12042        got  [53 28 155 77 1 4 0 0 1 88 141 205 89 56 255 255 255 255 255 255 255 255] 
12043        want [180 172 84 179 1 4 0 0 1 88 141 205 89 56 255 255 255 255 0 0 0 9 40 181 47 253 32 0 1 0 0]

looks like a valid failure.

[klauspost]

@varun06 Yeah, I guess it wants zero byte encoding to be have the complete wrapping. This just returns a 0 byte slice.

I will add an option to the encoder. Added PR: klauspost/compress#155

That said, expecting specific encoding in compression is perhaps a bit iffy. I will update it to the new values.

@danielnelson Actually not an unreasonable thing 👍 I will take a look. PR: klauspost/compress#156

[bai]

Looks like the only thing that prevents this PR from being 🍏 is this newline that makes go fmt check fail: 68a97de. Proof on Travis: https://travis-ci.org/Shopify/sarama/builds/581646021.

Do you think you could remote that extra newline or cherry-pick my commit so we could get this merged? 🙏

[klauspost]

@bai Sorry. I had forgot to set up go fmt. Fixed now.

[bai]

Fantastic, thank you!