[Build] use pigz to speed up a build

[kv-y]

Why I did it

It's possible to speed up some parts of a build using parallel compression/decompression.
This is especially important for build_debian.sh.

How I did it

pigz is a parallel implementation of gzip: https://zlib.net/pigz/
Some programs like docker and mkinitramfs can automatically detect and use it instead of gzip.
For tar we need to select it directly.

To enable this feature you need to set GZ_COMPRESS_PROGRAM=pigz

[kv-y]

Comparison.
Old:

01:45:17 [ building ] [ target/sonic-broadcom.bin ] 
02:17:10 [ finished ] [ target/sonic-broadcom.bin ]
32 min 53 sec

02:47:54 [ building ] [ target/sonic-mellanox.bin ] 
03:02:48 [ finished ] [ target/sonic-mellanox.bin ] 
14 min 54 sec

01:12:03 [ building ] [ target/sonic-marvell-armhf.bin ] 
01:31:48 [ finished ] [ target/sonic-marvell-armhf.bin ] 
19 min 45 sec

01:01:55 [ building ] [ target/sonic-vs.img.gz ] 
01:34:20 [ finished ] [ target/sonic-vs.img.gz ] 
32 min 25 sec

new

02:20:48 [ building ] [ target/sonic-broadcom.bin ] 
02:44:59 [ finished ] [ target/sonic-broadcom.bin ] 
24 min 11 sec

01:52:04 [ building ] [ target/sonic-mellanox.bin ] 
02:04:06 [ finished ] [ target/sonic-mellanox.bin ]
12 min 2 sec

01:25:14 [ building ] [ target/sonic-marvell-armhf.bin ] 
01:42:54 [ finished ] [ target/sonic-marvell-armhf.bin ] 
17 min 40 sec


01:31:40 [ building ] [ target/sonic-vs.img.gz ] 
01:55:14 [ finished ] [ target/sonic-vs.img.gz ]
23 min 34 sec

It may depends from different factors, but I got 15-25% improvements in build time of final debian image (build_debian.sh) with pigz. Most noticeable changes here: tar -I pigz -cf $OLDPWD/$FILESYSTEM_DOCKERFS -C ${DOCKERFS_PATH}var/lib/docker .

[lguohan]

can you make it optional so that people can try it out before we make it default?

[kv-y]

can you make it optional so that people can try it out before we make it default?

I think it's possible. I'll update this PR later then.

[Kalimuthu-Velappan]

Comparison. Old:

01:45:17 [ building ] [ target/sonic-broadcom.bin ] 
02:17:10 [ finished ] [ target/sonic-broadcom.bin ]
32 min 53 sec

02:47:54 [ building ] [ target/sonic-mellanox.bin ] 
03:02:48 [ finished ] [ target/sonic-mellanox.bin ] 
14 min 54 sec

01:12:03 [ building ] [ target/sonic-marvell-armhf.bin ] 
01:31:48 [ finished ] [ target/sonic-marvell-armhf.bin ] 
19 min 45 sec

01:01:55 [ building ] [ target/sonic-vs.img.gz ] 
01:34:20 [ finished ] [ target/sonic-vs.img.gz ] 
32 min 25 sec

new

02:20:48 [ building ] [ target/sonic-broadcom.bin ] 
02:44:59 [ finished ] [ target/sonic-broadcom.bin ] 
24 min 11 sec

01:52:04 [ building ] [ target/sonic-mellanox.bin ] 
02:04:06 [ finished ] [ target/sonic-mellanox.bin ]
12 min 2 sec

01:25:14 [ building ] [ target/sonic-marvell-armhf.bin ] 
01:42:54 [ finished ] [ target/sonic-marvell-armhf.bin ] 
17 min 40 sec


01:31:40 [ building ] [ target/sonic-vs.img.gz ] 
01:55:14 [ finished ] [ target/sonic-vs.img.gz ]
23 min 34 sec

It may depends from different factors, but I got 15-25% improvements in build time of final debian image (build_debian.sh) with pigz. Most noticeable changes here: tar -I pigz -cf $OLDPWD/$FILESYSTEM_DOCKERFS -C ${DOCKERFS_PATH}var/lib/docker .

Thanks for your suggestion. As you said, I also used this in the binary optimization PR.
#10718
I will sync up my changes based on your PR ENV flags.

[kv-y]

I will sync up my changes based on your PR ENV flags.

In fact I'm not sure that we really need this flag because it adds additional overhead to pass this flag to all files.
My suggestion is to directly use pigz instead of gzip whenever possible.
But as I understand, @lguohan wants to test pigz some time before use it by default.

[kv-y]

BTW, there are some places we can also use pigz:

sonic-buildimage/scripts/build_debian_base_system.sh

Line 103 in 9988ff8

(cd $BASEIMAGE_TARBALLPATH && fakeroot tar -zcf $BASEIMAGE_TARBALL .)

But need to check is it really a bottleneck like compressing dockerfs or not.
I'll probably do a separate PR if found that it's useful.

Also one more interisting fact:
Though we don't install pigz directly to sonic-slave we already use it indirectly in sonic-slave (e.g. docker load) because it's installed together with docker as suggested package.

Step 69/76 : RUN apt-get install -y docker-ce=5:20.10.21~3-0~debian-buster docker-ce-cli=5:20.10.21~3-0~debian-buster
...
The following NEW packages will be installed:
apparmor containerd.io docker-ce docker-ce-cli docker-ce-rootless-extras
docker-scan-plugin iptables libnftables0 nftables pigz
...
http://debian-archive.trafficmanager.net/debian buster/main amd64 pigz amd64 2.4-1 [57.8 kB]

https://dev.azure.com/mssonic/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/185360/logs/64

[lguohan]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).