[Reproducible Build] Installing packages during docker-sonic-vs.gz build is rejected by versions control framework

[baxia-lan]

Description

Background information for reproducible framework is described in sonic-net/SONiC#684.

The versions files are dumped after building the docker image, but during build, the package versions to install may vary.

As an example for docker-sonic-vs image build, the python3 package netifaces was installed first with version 0.10.9 by this line, and then when installing sonic-utilities whl, the package version is downgraded to 0.10.7. So the final netifaces version in docker-sonic-vs image is 0.10.7, and the versions-py3 file contains netifaces==0.10.7.

When versions control is enabled, building docker-sonic-vs image throws ERROR: Could not find a version that satisfies the requirement netifaces==0.10.9 error when running pip3 install netifaces==0.10.9 as an intermediate step.

Steps to reproduce the issue:

1. Generate versions files for platform VirtualSwitch

cd sonic-buildimage
make init
rm -rf files/build/versions
export NOJESSIE=1
export NOSTRETCH=1
make SONIC_VERSION_CONTROL_COMPONENTS=none configure PLATFORM=vs
make SONIC_VERSION_CONTROL_COMPONENTS=none target/docker-sonic-vs.gz
make freeze FREEZE_VERSION_OPTIONS=-r 

2. Build docker-sonic-vs.gz with SONIC_VERSION_CONTROL_COMPONENTS=all

make target/docker-sonic-vs.gz-clean
make SONIC_VERSION_CONTROL_COMPONENTS=all target/docker-sonic-vs.gz

Describe the results you received:

Step 31/74 : RUN pip3 install          scapy==2.4.4          pyroute2==0.5.14          netifaces==0.10.9
 ---> Running in 740f16ef2d66
Collecting scapy==2.4.4
  Downloading scapy-2.4.4.tar.gz (1.0 MB)
Collecting pyroute2==0.5.14
  Downloading pyroute2-0.5.14.tar.gz (873 kB)
ERROR: Could not find a version that satisfies the requirement netifaces==0.10.9
ERROR: No matching distribution found for netifaces==0.10.9
The command '/bin/sh -c pip3 install          scapy==2.4.4          pyroute2==0.5.14          netifaces==0.10.9' returned a non-zero code: 1
[  FAIL LOG END  ] [ target/docker-sonic-vs.gz ]
make: *** [slave.mk:717: target/docker-sonic-vs.gz] Error 1
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
Makefile.work:264: recipe for target 'target/docker-sonic-vs.gz' failed
make[1]: *** [target/docker-sonic-vs.gz] Error 2
make[1]: Leaving directory '/tmpfs/src/git/sonic-buildimage'
Makefile:7: recipe for target 'target/docker-sonic-vs.gz' failed
make: *** [target/docker-sonic-vs.gz] Error 2

Describe the results you expected:

The solution can be just update one line in docker-sonic-vs/Dockerfile.j2 to be pip3 install netifaces==0.10.9, but wondering if this is a good solution for similar issues: whether strict the versions defined in Dockerfiles to be consistent, or multiple versions of same package are allowed in versions control framework.

[baxia-lan]

@xumia @qiluo-msft , do you have any thoughts on this issue? Thank you.

[baxia-lan]

Similar to the python packages mentioned in the example, some debian packages installation are also not "reproducible" (eg. python3-pip) installed during building but uninstalled later are not captured in versions-deb-* files. So their versions are not verified when version control is enabled.