Use zstd instead of gzip for embedded tarball

[brandond]

Proposed Changes

Use zstd instead of gzip to compress the self-extracting binary image

Types of Changes

CI, self-extracting binary compression

Verification

• Check size of binary reported by CI:
  k3s binary dist/artifacts/k3s size 44716032 is less than max acceptable size of 61000000 bytes
• Check size of CI artifacts

Linked Issues

#2904

Further Comments

I evaluated bzip2, xz, and sztd. bzip2 did not give significant improvements in compression. xz and zstd are comparable, but there are significant criticisms of xz as a format, and the go libraries don't seem to be as mature as those available for zstd.

zstd is optimized for fast decompression with relatively low memory overhead, and is used to compress RPMs in Fedora since F31. There are cgo bindings for zstd, but I opted for a pure-go implementation since we build the self-extracting binary with CGO_ENABLED=0.

-rw-rw-r-- 1 brandond brandond 15503360 Feb  5 22:55 k3s-root-amd64.tar
-rw-rw-r-- 1 brandond brandond  6777664 Feb  5 22:55 k3s-root-amd64.tar.gz
-rw-rw-r-- 1 brandond brandond  6488804 Feb  5 22:55 k3s-root-amd64.tar.bz2
-rw-rw-r-- 1 brandond brandond  4819000 Feb  5 22:55 k3s-root-amd64.tar.xz
-rw-rw-r-- 1 brandond brandond  4778451 Feb  5 22:55 k3s-root-amd64.tar.zst

Pi3b timing

Before:

sysadm@pi03:~$ time /usr/local/bin/k3s kubectl version -o yaml --client
INFO[0000] Acquiring lock file /home/sysadm/.rancher/k3s/data/.lock
INFO[0000] Preparing data dir /home/sysadm/.rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd
clientVersion:
  buildDate: "2021-01-14T23:53:08Z"
  compiler: gc
  gitCommit: 1d4adb0301b9a63ceec8cabb11b309e061f43d5f
  gitTreeState: clean
  gitVersion: v1.20.2+k3s1
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: linux/arm

real	0m18.656s
user	0m15.910s
sys	0m1.111s

After:

sysadm@pi03:~$ time /usr/local/bin/k3s kubectl version -o yaml --client
INFO[0000] Acquiring lock file /home/sysadm/.rancher/k3s/data/.lock
INFO[0000] Preparing data dir /home/sysadm/.rancher/k3s/data/0d033359c1a686e15e2723b87fdff1556968c148c47cbec056510927800d9ff0
clientVersion:
  buildDate: "2021-02-06T08:31:36Z"
  compiler: gc
  gitCommit: 74cc9fbf0fa815b6aea5bce2b3c6d159161ee2cf
  gitTreeState: clean
  gitVersion: v1.20.2+k3s-74cc9fbf
  goVersion: go1.15.5
  major: "1"
  minor: "20"
  platform: linux/arm

real	0m13.590s
user	0m13.436s
sys	0m1.776s

With github.com.DataDog/zstd - the binary is 2mb bigger, and slightly slower to decompress:

sysadm@pi03:~$ time /usr/local/bin/k3s kubectl version -o yaml --client
INFO[0000] Acquiring lock file /home/sysadm/.rancher/k3s/data/.lock
INFO[0000] Preparing data dir /home/sysadm/.rancher/k3s/data/34a97157661b8a8f482956a44565b29d7eb72e2011d23a49a4cec471a2b8d76a
clientVersion:
  buildDate: "2021-02-08T19:43:26Z"
  compiler: gc
  gitCommit: f6384db6480c7feef8d1aeab032c1ed7263e46ae
  gitTreeState: clean
  gitVersion: v1.20.2+k3s-f6384db6
  goVersion: go1.15.8
  major: "1"
  minor: "20"
  platform: linux/arm


real	0m14.297s
user	0m10.586s
sys	0m1.612s

[brandond]

@erikwilson I am going to stick with the pure-go version - there doesn't seem to be much if any improvement from the cgo implementation when it comes to decompression speed, and it makes the binary a fair bit larger.