From 9cc375fad5c6cad87fb88695c8f34904a462edf7 Mon Sep 17 00:00:00 2001 From: xumia Date: Mon, 12 Oct 2020 12:38:33 +0000 Subject: [PATCH 1/7] Add SONiC Reproduceable Build design doc --- .../SONiC-Reproduceable-Build.md | 457 ++++++++++++++++++ .../images/azure-debian-mirror.png | Bin 0 -> 19428 bytes .../pypi-version-upgrade-automation.png | Bin 0 -> 38968 bytes .../images/upload-package-automation.png | Bin 0 -> 13209 bytes 4 files changed, 457 insertions(+) create mode 100644 doc/sonic-build-system/SONiC-Reproduceable-Build.md create mode 100644 doc/sonic-build-system/images/azure-debian-mirror.png create mode 100644 doc/sonic-build-system/images/pypi-version-upgrade-automation.png create mode 100644 doc/sonic-build-system/images/upload-package-automation.png diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md new file mode 100644 index 0000000000..f7b5d8cf35 --- /dev/null +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -0,0 +1,457 @@ + + +# SONiC Reproduceable Build + + + +# Table of Contents + +- [Revision History](#revision-history) +- [Overview](#overview) +- [Pypi packages](#pypi-packages) + * [Pypi Version Configuration](#pypi-version-configuration) + * [Pypi Version Control and Check](#pypi-version-control-and-check) + * [Pypi Version Upgrade Automation](#pypi-version-upgrade-automation) + * [Work Items](#work-items) +- [Debian Packages](#debian-packages) + * [Debian Version Configuration](#debian-version-configuration) + * [Debian Packages Version Control and Check](#debian-packages-version-control-and-check) + * [Work Items](#work-items-1) +- [Debootstrap](#debootstrap) +- [Packages downloaded by wget/curl](#packages-downloaded-by-wget-curl) + * [Web Package Version Configuration](#web-package-version-configuration) + * [File Storage](#file-storage) + * [The process to use a new web package](#the-process-to-use-a-new-web-package) + * [Work Items](#work-items-2) +- [Debian Package Mirror](#debian-package-mirror) + * [Debian Package Mirror managed by Aptly](#debian-package-mirror-managed-by-aptly) +- [Pypi Package Mirror](#pypi-package-mirror) +- [Docker Base Images](#docker-base-images) +- [Git Repository](#git-repository) + +# Revision History +|Date|Description| +| :------- | :------- | +| 2020/08/21 | Xuhui Miao, initial version | +| 2020/09/21 | Xuhui, added Debian/pypi repo part | +| 2020/09/22 | Qi Luo, review | +| 2020/10/10 | Xuhui, debootstrap supports reproduceable build | + + + +# Overview + +There are many SONiC failure builds caused by the external dependencies changed. It can be built in the first time and might be failed in the next build, although there are no code changes. SONiC reproducible build is to make the build stable and reduce the failure rate, build on the same git commit, expect the same result. + +The external dependencies contain pypi packages installed by pip/pip3, Debian packages installed by apt-get, wget/curl, debootstrap, docker base images, source code by git. + +Some of the dependent packages can be controlled by its version for the reproduceable build, such as pypi packages, Debian package. There is an assumption that the content of the packages should be the same for the packages with the same version. The other packages do not have a specified version, when the remote package changed, it cannot be detected by its URL, such as wget/curl. The file storage will be introduced to solve the package without version issue. + +When the reproduceable build is used, to catch up with the latest version of the dependent packages, sending pull request to update the versions is required, the automation Jenkins pipeline will be created to do it. + +In Scopes: + +1. Pypi packages installed by pip +2. Debian packages installed by apt-get +3. Host image created by debootstrap +4. Web packages downloaded by wget/curl +5. Debian package mirror +6. Pypi package mirror +7. Docker base images +8. Source code by git, full commit id +9. Arch: amd64, arm64, armhf + +Next scopes: + +1. Support on the old branches like 201911 + + + +# Pypi packages + +## Pypi Version Configuration + +The default version configuration files in "files/build/versions/" for Pypi packages. + +| File Name | Description | +| --- | --- | +| versions-py2 | The default versions for python2 | +| versions-py3 | The default versions for python3 | + +The content of the version file versions-py2 example: + +Py==1.7.0 + +pyang==2.1.1 + +pyangbind==0.6.0 + +pyasn1==0.4.2 + +The versions can be overridden by the version configuration file with the same file name in the Dockerfile folder, for example, the content of the version file dockers/docker-base-buster/versions-py2: + +**pyang==2.3.2** + +When building the docker-base-buster, the expected version constraints are as below: + +Py==1.7.0 + +**pyang==2.3.2** + +pyangbind==0.6.0 + +pyasn1==0.4.2 + +The version auto/manual upgrade configure example is as below, the automation pipeline will not upgrade the version of the Pypi package pyasn1, because the upgrade=manual is set in the configure file, the default value is upgrade=auto for the other packages. + +Py==1.7.0 + +pyang==2.3.2 + +pyangbind==0.6.0 + +pyasn1==0.4.2,upgrade=manual + +To generate the initial version configuration file, we can manually start the docker containers and print versions by command "pip freeze". + +## Pypi Version Control and Check + +In the Makefile or Dockerfile, if the version is specified for a Pypi package, it should be the same as the version in the constraints file. The build will be failed with version conflict. + +| Version in config file | Version in Makefile/Dockerfile | Check Result | +| --- | --- | --- | +| Yes | No | Success | +| No | Yes | Success | +| No | No | Failed for version not specified | +| Yes | Yes | Failed if different version | + +The Pypi version conguration file will be used as a pip constraints file, some of the flags no used, like upgrade=manual, will be removed when used by pip command. see [pip constraints-files user guide](https://pip.pypa.io/en/stable/user_guide/#constraints-files). It will control the installing packages and the referenced packages. + +The pip/pip3 command in Makefile and Dockerfile will be hooked with a new one, if version conflict, the build will break. It only installs the version of the package specified in the configuration file by adding the constraints option. + +## Pypi Version Upgrade Automation + +Developers can change the Pypi package version manually, and the Jenkins job will be provided to upgrade the Pypi package version automatically. It will upgrade all the Pypi packages listed in the Pypi version files, except the packages with a flag "upgrade=manual" in the configuration file. + +The data flow diagram of version upgrade is as below: + +![](images/pypi-version-upgrade-automation.png) + +## Work Items + +1. Generate the initial version files for pip/pip3. +2. Add the new pip/pip3 command, verify the version and use the version constraints. +3. Enable the new commands in Dockerfile files and Makefile files. +4. Implement the Pypi version upgrade automation Jenkins pipeline. + +# Debian Packages + +## Debian Version Configuration + +The default version source files in files/build/versions/ for Debian packages. + +| File Name | Description | +| --- | --- | +| versions-deb-jessie | The versions for jessie | +| versions-deb-stretch | The versions for stretch | +| versions-deb-buster | The versions for buster | + +Similar to Pypi packages, the package version can be overridden per docker image. The different part is that the versions of the debian packages are relative to the debian release, in each release, such as the stretch and buster, the package versions are not the same, so we define a version configuration file for each debian release. For a specified docker image, such as docker-base-buster, etc, we can use dockers/docker-base-buster/versions-deb to override the version configuration file. + +## Debian Packages Version Control and Check + +Like Pypi Version control and check, if the build version is conflict with the version configuration, the build will be failed. The apt-get will be replaced with a bash script to check the version conflict. + +The version control is based on the [APT Preferences](https://manpages.debian.org/buster/apt/apt_preferences.5.en.html). The high priority of the APT preference can force to install a specified version of a package, even downgrade the package (priority >= 1000). + +Example setting for sudo: + +_Package: tzdata_ + +_Pin: version 2019c-0+deb10u1_ + +_Pin-Priority: 990_ + +When installing the package tzdata, if the docker base has installed 2020a-0+deb10u1, it will skip to install the lower version 1.8.21p2-3ubuntu1 of the package. If the version is not installed, then prefer to install 2019c-0+deb10u1. + +The version configuration file not only controls the packages installing directly by "apt-get install" command, but also controls the dependent packages. + +When installing the package ntp, the package is depending on the package tzdata, if the tzdata not install the version 2019c-0+deb10u1 will be installed, even there is a new version 2020a-c+deb10u1. + +Sample script to verify it: + +_# apt-cache madison tzdata_ + +_tzdata | 2020a-0+deb10u1 | http://deb.debian.org/debian buster/main amd64 Packages_ + +_tzdata | 2019c-0+deb10u1 | http://deb.debian.org/debian buster-updates/main amd64 Packages_ + +_# apt-get remove tzdata_ + +_# apt-get install ntp_ + +_# dpkg -l tzdata | grep tzdata_ + +_ii tzdata 2019c-0+deb10u1 all time zone and daylight-saving time data_ + +If you change the Pin-Priority to 100, then the latest version will be installed. + +## Work Items + +1. Generate the initial version files for Debian packages. + +2. Add the new apt-get command, verify the version and set the version preferences. + +3. Enable the new commands in Dockerfile files and Makefile files. + +4. Implement the Debian packages version upgrade automation Jenkins pipeline. + + + +# Debootstrap + +Currently, it will call debootstrap command in each build to create a new debian base host image, the image will automatically use the latest debian packages. It is not a reproduceable build. + +debootstrap --variant=minbase --arch amd64 buster rootfs + +There are 4 steps in the debootstrap command normally, finding debian packages, downloading the packages, the first stage, and the second stage. The debootstrap supports to only run the first two steps to generate a tarball in tgz format, and supports to use the tarball to generate the file system without downloading the packages again. + +debootstrap --variant=minbase --arch amd64 --unpack-tarball=tarball.tgz buster rootfs + +The --unpack-tarball will only do the last two steps, without finding/downloading the debian packages in buster, See [debootstrap git commit](https://salsa.debian.org/installer-team/debootstrap/-/commit/25d80b10319ed292827d016bfea6edcdb51b9b52) + +The tarball can be generated by the following command: + +debootstrap --variant=minbase --arch amd64 --make-tarball=tarball.tgz buster rootfs + +We can generate the tarball and replace the all packages with the right versions, or generate the tarball ourselves to support the reproduceable build. + +The tarball format is as below: + +| File | Description | +| :---------------------- | :--------------------------------------------------- | +| /debootstrap/base | The base debian packages | +| /debootstrap/required | The required debian packages | +| /debootstrap/debpaths | The debian package name to package real path mapping | +| /var/cache/apt/archives | Contains the debian packages in the folder | + +The steps to do the reproduceable build for root filesystem. + +1. Build the normal root filesystem + + debootstrap --variant=minbase --arch amd64 buster rootfs + +2. Freeze the versions, the version configuration format, see [here](#Pypi-Version-Configuration) + +3. Download the debian packages based on the frozen versions, and make the debootstrap tarball + +4. build the root filesystem using the tarball with frozen debian package versions. + + + +# Packages downloaded by wget/curl + +The web site may be not stable, and the web packages may be replaced to another one unexpectedly, so we do not want to depend on the web packages at a remote web site. Although there is no version for a package downloaded by wget/curl, the hash value of the package file can be used as the version value. Like Pypi packages and the Debian packages, there is a configuration file for wget/curl. For the package with the same URL, it is expected the same binary will be retrieved when using wget/curl. + +Requirements: + +1. Use the same web package, even the remote web package changed. +2. Improve reliability, the build does not be impacted when the remote web site is not available. +3. Keep update, automatically send a version change Pull Request to use the latest web package. [Low Priority] + +## Web Package Version Configuration + +The version file in files/build/versions/ for web packages. + +| File Name | Description | +| --- | --- | +| versions-web | The versions for packages downloaded from web | + +Like pip or debian package version configuration, the version format for web package is as below: + + **==** ; + +An example as below: + +https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz== ebef065e4d35572af5b03e2be957a9c6c5063b38 + +If any new remote web package is used, it should be added in the version configuration file, or the build will be failed, so as to the existing web packages. + +## File Storage + +Before a web package is used in the SONiC repository, the package should be uploaded to a trusted file storage. The SONiC build only depends on the trusted file storage, not depend on the remote web site. + +The file name format in the file storage is as below: + +- + +For an example: go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38 + +For the same web package, it supports to have multiple versions of the file in the file storage, for example, go1.14.2.linux-amd64.tar.gz-b73d6366c0bce441d20138c150e2ddcdf406e654 (fake hash value), but for the same commit, the same version is used. + +There are multiple solutions for the file storage, such as FTP server, NFS server, Http Server, we do not focus on how to implement the file storage. In this document, we use Azure Storage Blob Container as the file storage. When you wget the web package [https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz](https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz), it will actually download package from the URL like [https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38](https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38) during the build, in this case, the storage account URL is [https://sonicstorage.blob.core.windows.net](https://sonicstorage.blob.core.windows.net/), and the container name is public. Some of the packages are already in the same storage account, it is not necessary to add to the version in the configuration file. + +## The process to use a new web package + +Option A: Manually upload package (only for file storage admin) + +1. Manually upload the web package to the file storage. +2. Change the web package version configuration file to register the package, and use the wget/curl to download the package in your build script. + +Option B: Automatically upload package (for all) + +![](images/upload-package-automation.png) + +1. Change the web package version configuration file to register the package. + +Developer sends a Pull Request to register the web package, the web package URL and the package hash value (sha1sum) should be provided. The PR should be only one-line change in files/build/versions/versions-web, see [Web Package Version Configuration](#_Web_Package_Version). + +1. Upload the web package to the file storage. + +When the Pull Request is complete, and the commit is merged to the master branch, a background task is triggered to download the package, check the hash value, and upload to the file storage. + +1. Use the web package in your build script. + +Expect the step 2 is complete automatically. When the step 2 is complete, the web package is ready to use. + +## Work Items + +1. Generate the initial version files for web packages. +2. Add the new wget/curl command, verify the hash value. +3. Enable the new commands in Dockerfile files and Makefile files. +4. Implement the web packages uploading to file storage automation Jenkins pipeline. + +# Debian Package Mirror + +Debian official mirror only supports a single version of each binary package in any given release by design. For reproduceable build, it is required to manage previous versions of Debian packages. The designed Debian package mirror is to guarantee that all the versions of Debian packages used by sonic-imagebuild exist in the mirror. + +The current design of the Debian package mirror is managed by [Aptly](https://www.aptly.info/). The [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website) is used to publish the mirror. The [Azure Content Delivery Network (CDN)](https://docs.microsoft.com/en-us/azure/cdn/cdn-overview) integrated with Azure Storage Account is enabled to cache content from Azure Storage. To improve availability and performance, [Azure Traffic Manager](https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-overview) that supports failover across multiple Azure CDN endpoints, is exposed to users. + +![](images/azure-debian-mirror.png) + + + +## Debian Package Mirror managed by Aptly + +Steps to publish a Debian package mirror by Aptly: + +1. Create a mirror pool and update the pool, example commands as below: + + _aptly mirror create debian-buster-main http://deb.debian.org/debian buster main_ + + _aptly mirror update debian-buster-main_ + +2. Create a repository and import the packages from mirror pool, example commands as below: + + _aptly repo create repo-debian-buster-main_ + + _aptly repo import debian-buster-main repo-debian-buster-main_ + +3. Publish a repository + + _aptly publish repo -distribution=buster repo-debian-buster_ + +Steps to update a Debian package mirror by Aptly: + +1. Update the mirror pool + + _aptly mirror update debian-buster-main_ + +2. Import the repository + + _aptly repo import debian-buster-main repo-debian-buster-main_ + +3. Update the publish + + _aptly publish update buster_ + +To Publish the mirror to Azure Storage, [Azure Blob Fuse](https://github.com/Azure/azure-storage-fuse) supports to mount the Azure Blob Container to the filesystem, it makes easy to publish the Debian package mirror. The Azure Storage Account, storage container name and the Storage Account Assess Key or Storage SAS Token should be provided. The publish settings of aptly are as below: + +_"FileSystemPublishEndpoints": {_ + +_"web": {_ + +_"rootDir": " __/data/aptly/debian/web__",_ + +_"linkMethod": "copy",_ + +_"verifyMethod":"md5"_ + +_},_ + +… + +_}_ + +The rootDir above can be a blob fuse mount point. It will publish to Azure Storage directly. + +It is not guaranteed that all the Debian packages are in the mirror, if the size of the mirror glows too fast, we will remove some of the old versions of the packages which are not used by SONiC. + +# Pypi Package Mirror + +The Pypi official mirror supports multiple versions of each binary package. It is not required to have an additional mirror for the reproduceable build. Not like Debian package mirror, the designed Pypi mirror only publishes the packages used by SONiC image build, the other packages will not be published. Because the size of the full Pypi package mirror is up to 6 trillion bytes now, it is really hard to manage. + +The design of the Pypi package mirror is only for monitoring and alerting. When a specified version of the package is used in SONiC, we can expect the package should never be changed. If it was changed (the same package name and version, but different binary), we will be notified. + +If a package used by SONiC is removed from the Pypi official mirror accidently, the SONiC build will be failed. One of the solution is to add the extra URLs for pip command, see [here](https://pip.pypa.io/en/stable/reference/pip_install/#install-extra-index-url). But the risk should be very low, we keep to update the version to the latest one by design. It should not be the high priority task to do it. + +To publish only the packages used by SONiC, the used packages and versions should be collected by SONiC build. The input file should be like this as below: + +_Automat==0.6.0_ + +_Babel==2.3.4_ + +_Babel==2.6.0_ + +_Flask==1.1.2_ + +_Jinja2==2.10_ + +_Jinja2==2.11.2_ + +_Jinja2==2.7.3_ + +For one package, it may contain several versions to publish. The input file schema is the same as the Pypi version control file. + +The [python-pypi-mirror](https://pypi.org/project/python-pypi-mirror/) is used to publish the Pypi mirror to [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website), likes Debian package mirror, it supports CDN and traffic manager. + +# Docker Base Images + +In the original design, the base images of the SONiC Dockerfile are based on image tag, for instance, in the Dockerfile sonic-slave-buster/Dockerfile, the base image is debian:buster, we can find "From debian:buster" in the file. When a new debian buster image is pushed in the docker registry, the debian:buster will refer to the new one, it is not reproduceable. To make it support reproduceable, we can change to refer to the digest hash value of the image. The docker base images that are based on the image tag will be changed to base on the image digest to support reproduceable build. The Dockerfile as below is based on the image tag: + +_From :_ + +Example: + +_From debian:buster_ + +Change to: + +_From :_ + +Example: + +_From debian@sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ + +The docker file will be changed to use the right digest during the build based on a version control file in file/build/versions/versions-docker-base, the content of the file is as below: + +debian:buster==_sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ + +debian:stretch==_sha256:335ecf9e8d9b2206c2e9e7f8b09547faa9f868e694f7c5be14c38be15ea8a7cf_ + +debian:jessie==_sha256:8fc7649643ca1acd3940706613ea7b170762cfce6e7955a6afb387aa40e9f9ea_ + +# Git Repository + +The SONiC build will download source code from remote repository during the build. For instance, when building sflow, it will download source code by command: git clone [https://github.com/sflow/sflowtool](https://github.com/sflow/sflowtool). It will download the latest source code, not reproduceable. + +A version control file is introduced to control which commit of the git repository will be download when building SONiC image. The schema of the version control file is same as wget/curl. The content of the control file likes as below: + +[https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121](https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121) + +It will run "git reset --hard" after the "git clone" command. + +We do not manage the git repositories, if a git commit used by SONiC is removed, then the build will be broken. + + + diff --git a/doc/sonic-build-system/images/azure-debian-mirror.png b/doc/sonic-build-system/images/azure-debian-mirror.png new file mode 100644 index 0000000000000000000000000000000000000000..76f7145b945b6c3c3387819bc5ecc16a706161f0 GIT binary patch literal 19428 zcmeHvdt8$F)-O%tZja55>7tc+neJtoI;E+Jj#-+bncAd@ifNM~sTpbs0@ducrKx#I z%^RhnnHRDIuSkv|W8M-kl-KfxNJI*VfXI2!YMlMP@A8swn!E#EsiTa0W z*@Nd>rd7{ttxaCHE@|yrhxLaV4y`CrIfwcn4Q$UTE>47|{*0w~Duz8gPQQ6t`M&h& zY!Rw}tuS8!0=+$;x(fulruq+jELek|pI#Mblh+wwy`}y7jUsD=Z&$z}jX27c#_2uj zW+}Uj7C*!s)=aojv@&@!$Uqg@T7AkO+~rW;iY);L@MJ>OGe$UfT8eZBcx z(59P46Q01A4eqZ1c9pEt@KvVHD`Cf z_M88hlcx{XUg76>*GEZ&3v$^A2E_Ox@txVJo?_Sv*Fb^=7X5+T#z@d1c-LK>7Is)F z%;i!M7eCa(WswLL_=PCUpgm(~3!6oPhp-B9poNW}Rp3`KOv5WGk58}VV>tag$aiUJ zIXCP^v4V*}H~>nh|dwbz}CCgd+DT zSKYt5Oy&9YSSm921i7}XF2MMy$JriFZz`16OoZWiSM9`wdQIOQ&!LQude^X?|M`op zcI&+*-mvTcDJ)8{AgAFmJ9O2BSB~m_-96&wV|6^*V}q&$)9hzsI4xjz^keAm>ljo0 zJE-w~$M{A=bv}>d&NBAaw{Mnp8CG}-;COlXkztxb&hN)`UhRg94ddOeZ|J6lk*=@( zT>~B^X{x)_elUs*T}QY6Z5XQUN`IN*o}^v;>vnH$NN(5F`HF0KDQ})jNcV% zzv0%_1n`{~pB?7X%Z=hE6%d6eMK!wO>T4GRJ-V))?D2`15VSC4Ue^f_rl%>35dyKG z+F;icEu;l|Ul?+xBO5AarPsZ`th|s&MVmmn z*mJUM6TBU+M5>FE9=7Tu#1|P)%xA&kS=_4xSF1v9*YJsZ`v;LeU;l1vrZ2U^i#E#~ zA<1<$g~;;7b=_877o?B1PJWpX2fxk%7yL3?s1(W4pF52vVLQ8RNA2mAJ_s6d?)iz| zm*rl8g7#b2(0UJ78Wp5SFTSA&%_=lWzV<2vN)d+~lb1A^tWBu@la1f!z3X#;LwbLV z^wY5w{$G^*+*8OsEY~=I>nXUgR{@57<=^mG|If2#;N)e8{zEtLfA!?~$@i5pt-|>) zj69#IQyJYrNVCb~4PG(_!WGfZ5%}y--;naduRx$j5B}kgQy&>R_*VypZ1&Ofy^7<` z#MsrIX*31G=pM(5edN9XVtNIaoM=BCz(Sg6tGGrp^9w#5=Dnq8LqY`gZ==r2m$@4s4onH#gx@-k*AuCm5jiw~;Tc~*zhuj^Dp7Q^%e z^72H_=GES112rSh69#e?1GB)$R4{c%Mc*yhEA*{xaK~aOmSki_hBw_v+gDj;@^O3R zQe=;EY_Bu(YiPL#bJLtl>|cuI^}GDNv&-L(zbxIiX8t$cQfl~@nEyAlc_E`!nat3# zSzcF5ntljRE)zQq*JCkgAfMqYi;5DY6Wva-%TvXpI3V2D5#IwxIgz}nx;e6oN_$U* zk7-JQ)Tel1ab(#78uurkT^*O9ept^wP}gUuE>a*Ip~v>$Q0w!KDTC^qB%OT%)`>p{ZQ;)E#}4&x7jcz3+KL*ht&&r z%TjM5_x8Rq0o<9~bARI0+}!jiumABu)Uw^uYH%NU{3UZCYHcje5;tE(T@Z9whV;YK zYx~^We+BkYNoo6`=o>LRv9Rz9%l1k1=3RoJ*zA1E!OHy+zmuzrFb1`c&0g?v zW90V>a8meDM5g+2)dF5tImN6+lc~^k0#dYMp>BL&`L2HEa6z6zDo~t__)W3<-Mb;h z4s-YhQFN_Y&JDy7{}OH6j;(Yip23AHq%1+IKzUL2pLJ`xXg?KGX&L+25jFr016&9F zC?K>emsmiUBF}NI=h1a3$i#hWMnw97`s1f+ZC7Wxrvdy#VeYcu={cKlC3 zoL;izQ`SaHKfY@eq@tAMu1dVMv4A{*#hpjdYq`ZRR^Tu$d>AWD34n}rjZK)2HgKuBJ-$RB4D$x2wvru zA|t0q=t7mQ_ox#WvTSSOmE5(A%kCglYEF6}Jude*enF-s*{vSt!eq`cUV)W~0bU(7 zgwi&w-`Pi_yfsO7uYfhtj7b>xEnHRLT-mmJ8fF~ z;Zn|N89Dt}tc8S>B=nGtp3|b`?!^i|0_*QL7Lnu{Rt$ZS^S6561tKmw-`Rx1%`;yM zi?@j4Mo)JKz;p)*#`NIovIq4CH#%%*-`zhRNHX046-?eT-2t0Od=YAdMD@2U7askZ zyLF#bEm8ZGG|;keRdt4xLDK6!41dog1e4)>AQ$6g>f0U>+Vg@!PwJUND_#ezJ!L6x zv$1C|OtzIxW^}_@>u8Tp`0bB#jLr0&^fWXkKe*}bC|i(nN&S%vaz1`|sM6c*H~uID zDX57imxpENP25en+=C|Z+qQO^`tDyV1Eecur814mIJL%=RL{>4y)mP?3)e^b#^qS&T11x3=2$0%So*GM?v2bx!n&M#$0ERA;QzU{^Svw2fG5jb z47Dj^;6uaj7P;Swem-SYyejgLyrE2+IUQ$pYIq*DQq=$S!K7zjPLmf@5&M1iVMxJ- z-DP@{;WfAP;~cRSujy4Y$Qb%b1|$5hr7ZHH)Fh+G!Ilj7P(l}S2(`wJvv85ha>=kx zV%$cqj#p}1o%W|Tz8HeMT)XNE(K(gbN+ev|i6bT**npm6IK9{_FSWVI+Lpgco^fNZ zy0+G{zm2@8T&EH1p{q_2X0FBx9&YL@C}45iQ=1AHOgcj|av8PS`n#yG!EtH$#2Zo6dN3rW&j zU}1qWLw%7)uy`Z}_4Ro44NgXN_F90aLUupXu8)Kg`Dxzzw*{rV+VhBEW;Jq}Dd!;d z{_1gTv<{tp;K|n6LdFEsf?R*DqptV;=HiG_jtnESABiI4o=$jH6H%2}`>@oAHFiR~ z1Sg`b>&!F%V1DQ_-fCF!1aW;ex>V&AT?x@`FAFu-Eh8u^lOw;CPaUHjy$#0(l*42+ zhrVBwh%;df@UdmceAO<2I}$G{Kdhvz+MUj*6U%|L!@ub9e`^-cY<8%y?LkD>Q( za1tyR9smU@jkpM(+NaPB4#fP$S2lvm;#{BH#pe`Q?c)|TpkeWxySDe7vP1A{{D1jL z!AXGpqoqPlm!Q@!>)f}`&wh*q&BuF2duc!RkonH3^0+0Y!B1^i#0})LrTo z?LT`;Wf7(T1%xhuVm>4sn1O)VCbMGxaAg7g?D1re*uG6R8+XOv`SpQ>zSxcW!K}?T z3COroKijhlk`r`uw-)w7P9R|wNCvB2)A9bCY5{Q7LG66JLN4YFTPo%SkxrTO=LQ6q z4rpx|y=dy+v1#_foQ@@~PEAXyt}o*CSN2E5@$VdGwE9`*kx`*zSrDlJWS(op{rFzH zz=!EXl@|OsM8o4{l)Xg49fdMaM?fx#NHJ+C(tvlTT?xY7KX;O(GlGIp6cij0r*FAS zJzOhKzBj0Afmn_lKx6?|8)$!6A1Jkf_TP{evb)@wa#K;KBwQ|eUE$~`6Xb5!O7kuz z_;Sn|nGv0vMm1)t#PQG|Eg2P4HL);w#G8pl#2OA{f+OeC!Q?1P*0ZoRqGn4SXoclv zf~zJ^Kh8Q(#?RjJd=s+w-5ZEQL;k8d?TXxxb;46hl*Xe@Ql*nH709H>48IA9F~8$y zctWS$%-E8o{_CwwXo4dGR`Vm5+LU)(zp3D+`J}nWWNK3mqcdH|wG_oI1U?M2(0aC` z9bSW@0tis1@J!AhWg>ZQ!xgV#ePx)(p?gy^yxHRXFwX-`ugqpK;!}*C55v^apHdzn zYGbZu^M@4+VjeZXdk_sg5S78>8P&#O9BIN9U>9Z>KFr3SuwmS{jd)ccEgMis<`N znsWI1^lu{Tm=b&OwtTh|LI6B^n2q{54p7R=&e#OIrP%h^ne$oF~xAlrc- zHq+_71A6Fpu*l~2>u2=2%Km!8a zz&TeIqMAC!_!IU>%7RdPw1+lr25MU&ocRV|n+It}g#kmiLz+(%)B^IKuk=AUM8#OP-f#C7P#!6at6bQT)g z#hbR!!ap{ijiNOX>1ORjY-y~EE>CB8Xxwj4_IAjDgw3fG?m*01p;?U9*)JlcGlC%& zs=KmxUnax^g_d~eCeSXJm$LU~ThHL2snVVqkA8h51LR{#4xdv!05`aYJDL_2x`Eo; z=x!;A)&5m7Y;(f=NqAj~cDvb0Mll~EDc2~jKl@T)yhB|@EQgUidoe?mRi`dnnpj5)qaY8#c-OP-k0dP ze*X{N?x5CigSS}Hh%Zn+3A1B7b zhb+q?5i2@mwuu_ULw27 zu?TbK<#rmn$Zr6!<;Ygt&Pp5U-L{R_J!@#)ZkCd%C{Awe zF@`NHuN%(^-xHJ!Y2S*JvHO`~C>!3VT|w(EjPP{Fyk|<7yy+V8RW%n~RiU>dDNXh6 zrlDr5 zn{>nbAOV#1Mw|Qbjk5uy|CtYSq>1$Plb`1GGAHsg0AG_G0-5ANdfz?E^FcSU=1ej% zw1}^+9em@bWpU)cm|y;DS?S+?tH1`aU)jIuw5fR0YCcQeldh2;s?3izTi#swfm;PT zWngq-^p)}*m39WqK6Zke}CFrWdv$BKF;%ik@Tvxhjf}Zi;b zokeb#ek%u>5TyvK#im5&;gB4Ru+!;RSnPL|K%-nd&8I1FxWhwzvOZUXY7*jhy_X9L zYPm(~Iq6NoHU0#g7N=T(N?&F*^BCqYZhX;ts(Orf!bcz0SCwwpt9yH8HM3RQ%Mi_- zDz2|*sviyVJ0o8hFEf6YZKCB>{QO)fY>x=}U9;lMQBt@=O~B+P>^>r|vjWZtyzbnR z&?ymhRZ!TLWUh%^YCU?2uujxzIuZgClIm4%iOtT8+Hn&2{uVg!$a%|dh%jHlX>x=ZN>fNI#7jN z^fQb}J)nvv=&Mc<4yaC%=(hd2cIL0c9s1yW(KXhu4|(X)RrptiG1GtnMtlz#p43|> zB|F!#baWY}VSjcdU+P8rgAPry`U4Ozlsff7DY(HYNqiP%-PpCr5j#!sE6M>Z7yq z#`dhtA^yCk1fvC$UcA>1t4(kZS++w2dq!4hf&J8Urth8BK319M$MlXdi`2*I&m zZJ$gJJEz+$8{G#J7KY%0a(NtyJ~bG)6t9Hr{=&hy+Pr6hSqtr1ye%{S<*cx%@$6- z2Xn}O^&mvy7jT?s}7}AZTC(I%5&AXCbGSd=BTSz)t8FjvUUcGE)BhD6iSScKkRjy?n1f)qQsR(1YPE9My^J#-bH84 zGEMU-mX#ZJcYlZ;!_LUE`S2KE*oaVSR9-&#hk!5!9-0p^65frc>&^~h42r+E+xS-# zH|f5c?kB?QlZeN+{{#!LUwx}7clvfy9ug|cXS<^;2GgpX_R8&{tc@&9!<~bL1uR6C zQRq$@NfNqsxJ+zjTwio*UdX?iT4opMSL~HpLap!ayHT|0JkZczGdgT|IJV&jx`QPw z(vn&>Xj}4kvE(jmrMn({O!$Ch%!{QT!W$IQ>^rxui~Z%rFXq}pD|RQvkojCrnV>j92oZr+rvY=?Sk<3vec2^GCKY?t2Y?Z>^H_w zv-o#~-GTni2A1@pMb;A!?>j*BF#}J`VJsFLOU`R4F5)~C(AE~z;$_PI;arB09PNaqj!&%#x%Zoa9NmL z@uBnZY6%{Wh;Xah5-kbiv{fP##Jb-oaBysL@jUn)0dQjabyi(SmWk3C;?9d)uP#%1|q5Jzd>)K$H zF1^s}Q&;Fn?fs@2wM7Ior**GCZ^YMpvG#S?HP~N+du9=SyuETdI3Xr>da<50>`NOu6k65}00C#Y<1yik#rzM9+up6M8U+U^A^k!X`2#W0va<+0euE3tO1Wuuy1G<0U{_DoAtc z*)e%o&YTx*p~tCg+M9v&+)H&c=ec#I=Q-iBB@!sRzwyyPukb~Wyy|es45pfql?a)L zlTKb00YIun(5rYQ@!Kzv5RG@+`@uERD23t^@-7r3YAAPirCx7Sr)VHkKWR82s(pXk z#=V1AINADBDE(n8tgi#n1lhh809oqPG|VXVO14Ws{HbxJzd@IKWv)-4fo{CTkJAlz zQt_L}dicAX^sud6hSKi?VTU?tp-~sRkl0D`rhrUuuGM$laJI4pGN`h>>2Y+&Fi>V1 z7ApVk~fkM-fX0%J?*2~xqC8P z`hD`%Kc{G$OQxqX>l2Ree?+77jLB!j#899RFEOdi3W;4b{}!>m_^2M?u!OpQur}3n zJEnDu2tJj}Z|TrSP8_qCC5HA$5pZpMY0dz}ear5$a76tHoTbo=S}C509b@nf;r6D@ z0c{+&JP3U_`C~_HE<)zgkdm`I0K+rm^Zt!5NTM%6 z8#Yupf}df)yLep~QCzBST+t4G<%KLM0tz0yl{x$7ay{hE%OFRm0hf4>)ddfN$cviF zwjSNzEas3Wpn+}S4CSN8eRZ+cC9%Bx?F6wE3y9vLqw$56!Y3 zZHL_QfT-N*$-dl0l&!WJ0`)a2V6E`HkPuWXR#Lc$1J;6+3YeCmdH#M(x)E zTuzB_{6;g^)8T^3T{|JwLFL9LaixH0siw&t8=LtuZNW}JUP*c&pmx*CdMd6wGo&o7 zq>Z=vODtFq-ih7RGXt)jTv5jBBZ12;u1Qu|$UV-2>xJBzt3T$0O|q@|qnpSk1n)HM zZ>#Iz3qbj9v|FL8TXuZ38(+CodeL-P+h&|nusb+t=uY~AevflR45Dad>W!Q6w6gI%Dwx33~v;58n^Eq+Fn6*K6 zHTrzFBRE8DN<5WiSvfJVlgJ|nAv?$u>0~sj_6$_h%e4ZN`k^snG;Q`?JoQS>si|jS zp16fqm*X+Ln(O`Cr(perz-C_Oh5wJz@*CF$!ic_ncWbG5^O>}&Wix?{mSXhVm}iAoO^7oJ zYe&sA0PzIVqdb&AC5vhh-J7|$@>9$!Gr@SEO*p%2PD>T}r3w~5WkKhqvS_d1D;kvH zxdWrHW+365xy_nN;1oQb;q+^k2Sd$YzxAQ$!b_fm-(ZZgn$w;$6Wyf@=irB7K;-BD zS@^uu=_)!K2Ps1ZWY(J{$T^L+;GID-puk=_&ggjiXmPEKsC5(FXh4c{Yf6`+vDy4p zvGr5Ev_P~0R#_F`<0q|>EJ{Xp*FG4n(0ddj6;(?MWP(xuOG?yMd(Y3}8&n|V-&q~= zm*e2|kP|}JUghM5-aX6bLNmMx5lotYce>y5m%PiE(*vd3SlijjB^*@5qTY92Zqa=F z2Nk{4(U8W#`WNN}N_sD`bHEC}zn%KBBT+v!mVOhuYY_{=Oec~J3VTuijxpvN zwn-s4>DybYZZBv*IL3a}$Odkhs1#3SF6jN3Uo$ zNWnn+5{#%Cd0!;F#I&~Q2cQfv#Hf(}wSd<-QfNl)T@d0Ele%;~n?k3KWG*c#1-__U zGeEh{7Wb{Wt-k}*EXX+g=V(lSo(kmR$coCatT9I0*rGv-iStAKIUTWNf!&Os{|GQJ z+U}h?6B&ZB0H<5C4}SJ7lrg7~^Emz`&p9+f_-BEg(*I1?PrQ3!Zshuw6dW5AJRC0j`C{SiZ&2Fx1o9O|9{FC|Ez@csT;`~f(3fz5-%Mk z{nqaMGIFOJr_=NP!mRcZ9q zHoS;J2!Q#u!j1ag=5k;ZJ$f*YTn+hUiH!f^96GWUDBtp7UC{!ddF>Pe^v*wYI$4_P zINz#xxzQl)3x)f^-@Z3qo~XJj=U|YKm(-WoJ2Ct1C%>|x?}g-xye$VO8z|9`jW_mb zeCCU-s0-ZJ;<|}NaPKdSO1#f@h`&+PwzSb29+^n2CzNrXAQj%b^qYbL6wN)Q54u97 zx1Ia*5VZInYiB)(4LgCv5e#>7(}8BeK+{7DXtxvl_OOTxmyc=i!|R&vF){QmoKc*q zbc}VMKnweXr9B4|FoIpQG)$`!vOY?PcOxv_4+qY+Y)T@)15vq$0RDl%pcA?+Ay-=2 zdg#mjzCoH^9aPCSiTpQqD>EvQiA@`ll4$4tnN|1W?Kb|Pl?TqH9A7Wr!eAIvd!xN$uTwZ=?XRum;R^ttANDfv zEkV!C%f-VAtO78ABF7EzT2hSgs$pPkNJC%{mZVy+o-#TwjPwZQs~Q5#lDp&HPg*Vkg5j zb^1jf;wS#~FHHE5l}l(`H>rEX<2<5{%q{sl4s%j^*V*;mT5!R4*c;x3;2&;B|Hx@b z)*;n%WbQ9+NRcqdA$_3J?0&@()rZ z{lR-@qVG(uyoY-rupol#>c=}9ov@-)P}=481Qte%gdYuGxYo=#?P3FPmW+5L$&Ek7ab;J&rRu3AWz$Llt#ScWLC_o|(ZtALh3kk!9 zTI?zk74cZ4zV6)_6n&un9(jYpDiaL-9bKMWnS%dw7=<5uRjR3}Gl)eTh?X zwDSeWXDrZO83)k>fLJ9A`_-3hb`2WpZcmKYr;s-|{5wWm*xJp1BUp7!`{eYyI5~0E z*;7`z7F*jl!ikKP$Jx4P1>Yyv!FdfkG-8`}Yb2EEEu&GmkeSG5;>G&OLWDK@mY~V= zZkV!r!-gGUFqJO1RDU;ho;SiHU6_mXbc~hJu@Q0qppu!^13Vs(H#?zdvBemqJ3h0?uw7U)^GRNh zsBoIl3~m4uj#21+CS=Jr@}i+}!$g4HW4GC4gj636>CM?rZ;ZKg}g|pqO;0-kqXQu>}%cYrkUb3U0p%{+F^5zn&8HDvsR7d>G91&O_CnKkqr=I;b|$J!t})Sc?kK0p2V8l+rpwg$-Q_ zJuB&3I|RXFfgrSOY}~7@p^>&g%Q{pbRBXQpc8fBIyVDZ&xc1j94K4o`bKf;Qzp!BO zp-=TYvv`ZE(~86S3L5$Jyw16)U(U)DKbjp~cm=r6K8tjlrk^c(yw07Bi$BN(z8cx~ zsfroDUdJ38o~7849~4LgW2HyJsAp*NyMHC}a-$#m7LSMWwbB5pG=0r+Fm^jx9(e~6 z*Un2y3?0So#lzUi741bH+N{5OTW^N43mT(3T+s6{La3}lwlT8=Gpfl!`dZQvQ#G_~XNgAjiq|GQX-j11BCi5v0i21Y7b+fsFjKMKco%@2j`fW(kYviMHv0V~mLd+RR@j zoUQ4A_IoMSq*UaMpjI zCt+l;kqV9Me#ob-RyFe9o}2>1MgLq}9+;P61u6TGpKmkwKfDx9{}VL(^BnXqZ<>Hf zs`Vhyp2>)ofrseYif+?zmO1UxlTbN(;dIgGnNiD(S8wt}8*uGMIe|ZiLMv z23g6w1c?0{V8mx~UdJM8j!=%s^zb$q_hVv2Ewmz!z5^P5p4Qspq0r~|f4@TRtKY`y zH;0h#+}7p?p-X+gpEL(%J%diZcUpbRJYIE&TV zX{oo>>3t_>;LSyR>Viaa8G%=`)6G!k{X;!XDKs#$lGP4wz467ur#-qw-0f=CoY|`+ z-|sziEuA?1sbMtL&1w#z2YCI(KFm3hg^snJ&4R`LAXc>?Bt7?3GI8|UGd~E@wxRpl zH-Fna9xG+y&wL-VLLPVOc6;7e+ZGI9SOJY~H~J`KJ}y~RlMb9_T{cYKWjMzEEunch zn#h}8HK-ggCYkD7iumF-+tm)}ZmXy&F1mEqV{CLw=45fIx{mJmOvOD<`+lZiigjMs zpu)>ggsgz9-lc}?E3m9uBQc%vn-QNw)aKL9IM9NUFpL%cP3T^rTC!sOx6fQwtPP5D zQ4m=VD;#G{qDy#q{U)X)jk1cTFDlYnm8Hf#djx3GIv9YSt3CPTTq{AIEB;B|=6tRE z!4wUi>R-6PsO=s{WpK*tyb$ajeWS=3@pSQy6l1?e^kTSF`#=781p<-7&tyL+v0pXa R4+LzG-SLyh%8vTn_#YQ&GxGod literal 0 HcmV?d00001 diff --git a/doc/sonic-build-system/images/pypi-version-upgrade-automation.png b/doc/sonic-build-system/images/pypi-version-upgrade-automation.png new file mode 100644 index 0000000000000000000000000000000000000000..778c91f648cd176634f06ba3ed757d1185489239 GIT binary patch literal 38968 zcmb^Z1yodT_%@6VDX1u=2$F(?64KHojMA;dfGFJ(Lx-e@(%ndkgpv-!fRyym(xEa8 zHH;E73^m`z-+9mbf7d$aTkAXPtfgCK*n2;9KX+W$^%$qG3#Pfuav1~y(L8zl$Pfe~ zRRMv>Sg0t0Z(hr$2m$|)cpHLMLA6NMP2dB$gNhDt$Ctz_r#2M8XKK&KX5JtWz0bv8 zl4W}if8a|lAN6NGYF>6WK29F)oW@RWb|A3_Vh?0Rfj?rLq7t$a60#2-zG^Y!1%Wb) zo;*@9_P4^Eg?O^!^U(pJe|sDzf*GkuBlykI_%%P+cJjE*3N55?%xC1LOw(A`aer;_ zK6Ze0;vPWd9tMGzmU{AUyc)`yQ%-E^jEfuF9sb)jBlK{Jv0}m9jC;Y*f|lc|S)#;U zgEVpyawo-{QBHP$!Wl~BQ+>VIr~2?f&9##o28Jdd^v%*KOhdQuXX_%j-5-*JK%90g z7$OM>WEf2gd@_vt?^?hKGVwzV-hg@jjDUr%4>LPCJE!!wI^c|UxBlSIvHwuDwMMG3U zw16paW~@>rB`j`ij4uAzhCWfm{DcL1HCxIl+;?a!XOtb32-@^@P$;1Q$4ihyo{d42A z%`np;7#YB9?QiHVq21q8eMM|Mctp^HrI1@^hDnA)Q)j}fYbbC$-y5Ictjx@nYPn^MYqJ9w9?{saHX#R4Sl5?-LRRmCh!WP^^?4{usOLhC9I$ z>l1zZYZDbPgp70WvKb7Qov3}3?Kq%Ns;D_R8_W=6UuZ2fGP9Wa=~$@s?Cr42?WPtl zh@7Mrk|W#0BW}1*$@=_2k7#T^iVSlHO~eB;nt~)-E2IeQ94RU1xlvjo%p$8XI$W=1 zDZAx;Sl;aw^7Au9OtL|^d8jS7|5>)`wefb@UT-1Zo_*Ty)!=Z zp^h3P4;WPOfv10B5YMxtH$xmNg&xPA^5(n8ziUJWHYQ#|pAal9aIa5a9oC^VjG!qR z7?p{VuC5Jl9EZVrm<~vpiZlrf))c5}{E(CTcPlh7DG~ ze1TJk_ps4Tl#E9l$ZUJ#H(d4--ANux1DinJ196yvu#bEc6iQYf!-0YS-JlwyU@LQ? z_AnLZqfut^AfWSnHy!n7Egu~(o{=Dl^Ptk(fJG%FDxE(=7LF#YZNR=oq%t~RPv{^o zLkTcw2HRC>iWh(SEnQCR-|xfUTsyq9={+;EIzIE=krt#5bor&NZI2a$Q-K_v?B!Ai zFT7raKg^A@Lf%9ua04OrJm^4%La(s+Kyt(7Z}9PE1&$?Lf}5PA{}u%pqVWm)?fm$D21%B+|KrxqPL-e$w>Q`1B)?B9HO{*1o+_lv(+diRu%8p& z)ShwikjvEdu5e<)YCsc!Yv$};xxQBDI3^DLD1gp;K9dpbSU0_VsNRQZIz?~(?cgQ4 zP$cOBYy3U^{!=d`om9}1(9z+Ixka<$zV*U-X-fy^i6c{_o40i{D-MsP5($zSc`}IaFLTdsI?kFmX)wr>|OuD z(vfo3|FwNJ3bFohLC`@rPB7>oM^N$#uibQqXV)0_v;wJ%ab&h+s0{J9&{0lA9kX&P z;LdK%)Iu6&7O5$3=X&d<7K5X75**N!Ij#d4NcpG8^{~<``ni4CJ5z#Soov?vR(#B> z;YwY9YMn4FhV1b8U80F^Y0?rVf^Ei$(;Kj!mFU}N()X(gJwWoRG5(uw!9L%Gp&emoft$nyL4#(WGh=lj*JMR99VCN7?V+l6EtPlotjUgG5bljnXk=x6g5 zVGj}q_D;BH*m-b}P}me$@|c&!`2FwkzmK|;s8uU+P_AoSHdaU3d{5y^DOrSkC&ybu z`8;1{-((Oa$sTj4(t=(h^IR*lS%-*@j(O>~`B^SUaY{NkB`d#7;LlAN_A$m~+w9rxM`a6yQ4ro1)0CDGc7PIG7gX*fOxZ z|J}~1e6zx!7WMwJ3L*E2U?zmpp z8Jf1k<-8pEDhRmSWwF?rZNQqxxxcoR+n>SqUx^=dz)>RJ&rT+`>(}*oQN#Po*x?jd z82wAD;?I5cIdwwKGkMorzV)YHPVf|^Coi?=rin#dfMHdasf|Tg z^D|}C;Oq&Z=Lv<`+rBpS1{v)gW-PpmW;INUUOG2@*4aCuXR4`rA>PS|yY&J4_7~|6 zf`ZcxShVLGm5*P(#Bu*TCY(;{4qN3YkbKPO&Z#@%-Te4v{nL;b$>S08yfD&dM3;#f zP0)Xf?C9>Ek?YYuw@lBhkQzohVHV%#ZJ$yQQaow(2_e&t{&1$;X&{IS(>b5H%$W~N zvDp0l{ElSu7cHHD5S95`3N8a#2o!mucj1VfAmZkM%&=m|{IXZqT>Ys5XNb^oj{N_A zaACn)^F@inJ~(pS@2FfU&owQ;VYgn|*`f1sW`M9} zKG2$stejj~*g<&QA(>6b;jUtlSqqb2QMKxYp#lz!&gUaDs(eol zIw(Wj*x1-`H~;hB#SQ!`L0AMXV}Vm4kheons$>7HjQMcM-H!ym-IkVn+TcQs)>493 zvrom{*3~v;%|dtwm$~(Aadonb<^$Y2&Y;h-e}~&6_Ga1)KvHWF}lJsytTwnZpUAN zk$GW$HoC<3t>Y5BPfgqIzl=}F-G861N|u;`XdJ$I)u3m@RtmowZv^gNNX7 zo~RX?;`m2P`(3RJLD3cxWx5CuqlAP6PhH#ITLdbjFhjwUOCmUnJ+EXRwKZ_6;Vh|$ zZlhM4L7%BGB)~8ZNOfq*6MJj-)Ql^iv8VN@;DufLgdMY4BE@y zbonPv<@jshhaWOS--n4M8hP@^Sz*CpR^^u-J8>UKcMxs`XT8f61UAL;ZJgEIf9+@1mX%4iW1Uezi$8R<4u#Q0Z29 z1hDxvZ|=|;&ObnGG(7dztu=o#qdSrk&oMG05cV->7Y6=OXYJuZqAYPyun>-ZC)yFu z&$7GG3iB-us~L+%&S|Ne;>WgD%W;syyyw=ZznEk$UsZMm?B1w+p9+DScaaHEWydh& zAP!oYnJ>$IbQ18nb_a99tnexGl$`@ZB-LBm zyIRyxX09DbpBk7Z$+C*yW61@QE6_q}gX$zCCFd*%K8%i-`6k>=knuIZ3z>}MO6%>mZVLz9UpYS9E`!15 zmPx}0fds9xwQ`(O<2A^b0k~~i{XM1VW%8mNU;xf{9(&3>?9KoDw zYHDIqQuY~};3P_}#DvRDEiJjp#p>}l=gL6(p}>Ux!xF@|_Q$~-1He3ECj2GHmqJf4 z6LWJ;fp0)Xc|e~Ab%!hhbsx|u^63>cYV9{QHxB}f?Tow&J=idi@!QQid(sZT3Jfu( zbyjYfpl=7jzD*m6W8l;NLjcUh*2`-`ctjFNhhUiIg<>8P>RO)94Ui^KJveg$Dh@}*x zdDJ^-XC5(m1tkVj0bUi*=9P)u1k98fE@3S*2${UmVp%?F04jj-nK`;FQDXp zI=`Pz=2PLx?1~bNn*V+ko;JC<%C>Z4+Mxs`OmmGTG=-j3Vr0APtB!|Ger;6`CtV1<-eBJ_y|jBG41!1^5mh{9BO2D~x&gWQurh`5CSO3bbN z2R_G(S)VP6v*>N3@dr6IHnumaFqieW7Q%4A(dlr0u%tA;$wgK9`6BcKd-Ayk+4t|? zcP1=Zpl$V3s9K*fuWV`Nm&Pz{qHx>5Edm@gm`F=*tPC_?w-)UKEPBFK=YMYjFdqJr zhbZCvzP{XHeLV*f|3=Yex!+;N{^gdGY1g@^vUr7DHWK2!m<9#hVedNbKi{HK`Nrr7 z)AG}gZvS$BNo6zIi>dscJIwX(uik7rz&MlvaDub^@84Gq;u9%Pi}S|;%ugI~I}NU{ zYc4#Q%oR8iz6U^e>ODe!n{@Ilq5N7!EC-cpG-|*~eFR(-moTK&7AhP)2+`5eSQ)v$hty3uZzHXX2;Y%bjBDTEvF>5uOMrX?3wImrYJq(jsX3I*QqA)rj`ER2yx z#c%P|C6rj{WiBf5A1gGhX-|p!4SOEv$FcQO)M906@V4c+ltBm?+)k^mAE#hZHYA*r zkQ^ds2VN<9?2xz<=-kjahkB#35|5srb0-1D{NP|Y2%ByxJQ=3fLF}_0>^T1!CgRN` z(yaB`)}@hni5{$!*5W#s0%K)8k~?9T`Sp_vtw#L7su*Ho6@-x_Z0d#GJHcbG(FBrb_@290|Uakbcd6sbs6sD-NS7L=9Hgwc&H^A{E&c63u#}OwBrSR>{OhSTeVV` zC%bf#kC75wmSbyvKlymb+T}-eGBy+sU;i-UYnGI7>kyCEvbHW{mG>V@`ytFeE8?J6 znDL3Ciy=uOijxPqx@tz)<~Y|sFUI#lg*eOF_%Lqe0R3xfAM%vKOwPMIh}LV}df+u{ zqBYS1cJ@PLu5d#T$%IlmuOq5tc(m!UaQ?6d@H8&s~DqNS?I6$`2Zrz#} zCO3b-Z`|&&tVTo<|0OE|tnH*Tb`PTzB+n48oA=OW5ORV$D_QihfKw~0$aZ-H*^ln0 zMT%jcPOofs$(yFalSPbznIuno9@Ely-=>faJx)1IB;tu?!mLW}gO&zhKZqReHeoub zVNGrYrXGrGn86r6#p7J#2G}G$>l~saxMP>~v`lfY_rMX0bors1f)8cahzK0wZ$UiY zq&(Z$=52K^>?D_hf*MlyKE8b!`iH%*1M)68q<%8QE4jpQ-S4_<3MBFRP{OxVywdd> z4iRP!o?iUN%~DHzkl?chbr#X@9zOJ z-ltKP>AidL)~Bk=K+ZP?nwp=#XMNPgw3dtCy7Y&c50npuhPr!<6hIVodFVF7+za;& zu)7a@3QOb^>WJSL9X{(7;C#n5mOUJ2<#iQBK$}S6(12=}Pl`Awx@D0q)l9EP*OZOp z2OAtKgPFUlfV{%Gv#>2srgZjW7w+*~uF-9FZCNqv#DY8a+C#Y|-MPl%{wto%GYQLF zE8c$#`X;mO*S;>yn;a1WHE;@U&>z`uD9h$tkS$i<`0wG>o$1wLYzZPUBux7=|DWZw z>V7FfOaQr6rjF06bE~eFkCdEen?hbEF*NnI1N7e3LNYEO=b+VvJ&eb~P+@g)D#y9i zc_NxP$C^w%C9X=lRlq({hN@+<@;7D48}oLPZ|O863_-3doqiVZla!Qrs6_p)tI;i^ z&v(=&ft4j$?b2`sLVlfC<&G8iUrkz~(^$cN9Lp~XO~sMw+XWf6HdB(B=DN<7!vC%s zI7c@}E-!Pg&m3Pr?n=eCNH4T4++@Ia4RgM|C{L9@%FB$D(%i*2t!R?r|Sc`vO zHXGQ+d-%Q`|3yn`E94H@ALASIF=a}L$O5?c=&DGy+!q8 z_3bq%acnGN0hRFMNdV>>)^>D-h!Du!j4D-DxIX!{!@Pdwe<+qNdQu8u=k z0wqozsor|~%v3?o#-8@s%@W$ASH6YB&Tsb`^N7i*26ZIhlD;uW@Q27Xrs)$v5|BAe zQ5&0khb?8K#2{8rFl$B_;88~CY?ph}`h=}g_Hi3xY_PM~u%k->_$7cF zOTWL4e;ueyrMh&BjZMN7Js%580D z3Fj8d4)BI6KC>sj&3)OD^8~4t-!XCMDp7&=xHxiE^=G4CLC9^LfZqFYy~5GGbbb~y zY;uMZEB7s++t2~rkdKnpcf)4}p9c5k2MKt?9Rgy2FW-60iUX)JU$pMUm<%{nvbz1r{YmWJ6$hP0y*TDkJ&JsD56pH!zA zKD+v=#dqmdi<*O8qK^w`?n9^AkZC6+MpMoz25qq3k$DHZNyHJCg>!e?l-()(>Y7Q%Bb$#-=B&5N0WM?%pL^7H@sNu3g3+aM>Oa(!n^~<@H z=3II7ic$VM13u2_fm^<+hWYPAk9yVlUQQXdWDGSP)S$$KqRFOJsZ^&OVcTc;6={?( z=K7ddrY`bkDc}8RvCL5aEuybi_d8*~wBBu@fp?Am-KU7rbkF&>t{mD}y#7U;h%z71HpVzls4raMQNX_u1ue1yziY`f1h)%ubZpC|s?1!A)FpS$8bio5xTT**v?<{cERZL`q@_ zxm3Soz5W(fqg9NJ4}g_FT9Yr>_gg)svprWFGd!_8x`poqE=DPBL<(cw;UHuV;p z^a5~=;|5TMOB6COFfc%!gwH;d7T!3faOxyo3GfVe0^*h3?{qj2cnh<=W|FzVcZMLn zG2adDBXtjD{K#LtjT9Fjd^{z(v3x-CCE-?Hdtd;2y5VPXcPE%O-{WboIV};yBiL2D z&d0RnI-}gBE2{_e>bX1!k5@L`E0qlM#Nsk_g7A^sY`IeAbcjB;{wF@~JL>%TaH0{`|hamSW5E&85dfD)c#5CXcVqZHpL@>s;6P%EBE zBm*0amc!jlN~BHGA5}c5wruksU#?<5Go7B8c;Ny2Q46_`U-V8C2qRIJbdos-$o#~e zCsS&F54BimV=kVN-+w3nw}$~|irP%JmVIqD%c`U~WCd_{-J|B$$ig)z#(uM<(Wz_g zT~|CRLQt%6lixLNQcD7F|KSM!^P2EyTmakcD5IUr(8N0`{Q!jW+FgVVlQ7S7Fls1; z%`YsBP)Qal@n1C6bXkQxzfdjl$5=c&j7~lA^5lckl1D&UpQbtVF9K~Cd^9)^*XSPI zn*M~L2Y@bp)oEFB!Y1`~#PyGrlH;K$8qHeN$u?1w0M^DPosSBUaG1|K zLUwXFG+!d?I~;fmx=6*EFy0uq18^5Uu%M669Ds+3i5tx9kJ}~leQV_7@eNIq8BO)utzdnf`n)x9 z<|3V)n33KPKG1b`W{L9V_!h?g@iUsfRTqrMDg08>bKVH7Sq+IC#bsoAIQaUq}$A)jR&78Ykuo`%! zt>q`yW>gDREg@wW)?u86AZ#}L05|)d1u)7zjtjKko0;E?1 z^BJ@L7Jdc>(o@o;uv|Cnt{GT8~^lantK$f-~4H+P;3(1w5_3C@Y40X>PPZ0{d6@+_1d23Hq14=IN2}nv4zIMzI5T6b5 zIN`_ysP)G_^J#aD8ow~#Sd=YiuW zx-d*}&vEqocMYShiDYYq*6s5XEP6wDiULT*Gctw_+LX+NgH~T(QdN@hmFOpq7=h!R z%iQ*VyT=Wb7d;oy)`%&4E?UK_**Y|MPwzRHFXU`(?A z;N?yT=wpxuAD zR7BZ>CQWSla;q}4SYB2cDpF?toQ1+xqbC#;g zuM0|Bp$4O0W?iy>omd?pY7&5?4jkS!2$40CM+x)La8bpT0p8%nj92zG*o-=?+NH#T zetX*+db+gduj^Mk!?P%Ykn`?X+8$T4fcDL*=P*Z}o!ud;pJDyZXfhc04E@c0?kD?s z+CsGjWn>6Il(WQE^Lg_??4w>PLp?Tq3v9}OWt}^(werZG+a-*OF1y6RmLFnkr0bvl zwBIrWdLrIN$&oKm;Kh^@6A(dvZ*!mAy- zluqi4ZolP&L8a@7yR6;l9_odGb>t0(W4O)s)qqfgZQo4vXzcGy!V)HTO}Q>cS?}CZ zYB!_-Kc?2+GB7E#O{JJwiC$+yE52&(xQrYs8U6eBCmXWiG9LGCIyYE&H89zApu{4rXf#x2gIqUqF+9DG-%YKHNt!j`g7kI1TjDG{X+V< zD_SQa%SX1Wq(m{08pw5)0WW0JcuAZZ^2RMIU$)Mv4wFHM@=Y%OVk(vC-vAVSp7xJpBJgJxoBsl4z{5}rk_Hd= zj4Fz{Lp~h?6b7BfWOP6pZL#Q;IWw#WyTbq9lFd)DrNk7gqQpF6OTBMy`W6|pv>jr# zyZwy+;07T8n8xK#c9)adX=V05_Y1)!l|KOwXF9$^Z7r+?Ei!utF7Z~I*) znnyMchd6(9zz0iHB0y2Z6atX*Tp6Kpxa_XJPyk7#;=kV7^9p4Kq>35}YZU+lfLqMW zA2ryON}>OLrwFv747>7J_}|9~Zg-gW_52@nltN!NEFV>}(#Xkf?E?U5HZ3CnCF`$H zsZgs^Y4QOtMDPNXL+k#uTD5^)yc zmc=q|cL0g@BN=#fVa~K&`og*Ffb{^XCHl9tQ7-^iCw5``nnIOjxgD!Qz4rAMJr`y$ z`tRR_-3dZS*9LTAwUXNjnC0xqi>2o5x$uokM*v$JpD_d=@D716?IFnW8?!H7nN~hi zm%a_F{cETE*9HEdAB1>>Ic>5mmkXhJBwk}#-U$|Axm+S>)i(R?$nhv!A*dO0`8n^M z3k#!~OwaIQMhSxq{>_sr3>b$(YxNk_laxU_fZ%%n{=L`!h_ulzso_84~dAj#yxLZ zY_cC=@%{w_2T=KcD={TUqXJG~6}W)m*A7n@@B3{&o`|5hvSSF!V*fYtB%W7;b}B$8 zfb#jmS3LdzQ2Go}2OIT3NtJ6SdnuWcJwM6-peS>I3KdYY0FQ~8k*uvK&L&_7O<}xR?ZYb8?FgZyr6JnfZP7>S{X69 zx0hI$##b0@dTrtE)}IEw!G?1RkiI&=CYns#tEurA&=M)EZy_ma1_;+G`?Ky9+JL_v zcLguy2NL@B)>387*Q`RdpONa@08dOe{JOuFnG+d$qw;lrpxHUD*Obti7|NqGG1MXE zHz<%#q6`CgOylXdb@GDSD}e%j5*t%bWMJp)W)2>MZ++|A4}}6^8Oe(-6b)6b`N1Q` zy;d1r6m_Of5SuzuWeR}9G+ugJcUl&#W|hXYvUl_0YjllYJ*v(q7FCm4Qy&2k(ihJ< zKaUPZ{XE3Ab{|~MPb>6)KOVOKq49k6WNz;k$XL0XX1!Fe)S)0iqrm?cbtOH&2B@j9rguL7>NX0lnM*`x*%B-~vvbB{^&x_oZvBpIa<1vV1}7_Qea{`~?q{ zvQ9MwZVEUF7mv6_uOT6FVEO>MSf62JWTZYc1owadFo7#pWZ_#ytQq=E{IU6fv^{}C z$=!owQd&kvz@wc|E@=5x|3YO1XXHP<5Ev8+t}g8Rre0CQd~lzoHhiDSUA=v%c$-T@WU18m+fl@x7sa#2(j?(rDF z%iCV$dx1tq3^diCtqEWk>;4JYD2}xS@B*Mhj|a9-(^VC~V5MWl$dnPVU_$)`28#n& z!e~)Sn*V(w;D4SD_&crcHxZQ+iO)rl^ST|kuc(&1=Z5FB7s`z)KGFr_Et z=T@STbP@Vw08!ykcc4{qu;b--d2UE}bFZuLAP&nUHwdb3eXhp z+{PYgY70Q~h0K{ZcIIyGk<9bv7S7*$a{S>c=rJ#nEkl1m!7nR;6yvF?-{QJrB){V* zzL}tyTu^Xrb9+cA-`4}s#Vwx96;KNt9jO?{ypYEiXng$>xuU-F3$M~i2rh8HMB&v)~}lZR!FHsJa+(XqFh2-`7`>We+>o;_cDYI=KT;r4{}6jr_fd(M?8{rRciqhlMa!ZH+{ zq2|Zg&XyIrx*ZF(M;t$k4B7K}M2r&Pnmyb^em_Gzza@AcIBoo6rP(HR{~7XkP(JKO ze=!+y0$^1-mvYl_v#&8Q6Wf-Pp}7@b;d2nL=>)j7RX8i^<5PJ3wGIs!?#|cS*o@a$6 zKId=&YFnq^B&INgkyRKlx4C7+B9DqSg&(}H#Pv55*{tRT)uH(dUhI`x&F(WyDjs10u)13hC6h$`lg~3)Q%Q{+09YoC#o7v_+@=iQ#+P)csXk-^Cu}DH z4wSa}U%|(7zR<=6^lC%Zl38SJ^U+PAQ$K^}Y|N8k>_)_ecl&dM!S8zc zaz@wS-+683-UQG=4!)#6SPYtP1J-|H0r#2xG->F_R^Tk3_!uy~!H;=Fd*{}r3mU$7 zaEJf(ezI9RHOVF07O7MyuD@1Wq>Ft{67>?ZRD6akVBeG~VQ%G~C=RmjF2{GWZv+ik zmM?bDO*a0(zMpb@GJOA~e;JTrSXs=N;?K)xUIO*3@YuG^4702`Eni!Y!T)HueE5Gi zywy6){`H>LL2d)V+I(w&JKfr+QRz0&b*UvLTWRr#CyHuIr&AspE0B8#E?z2~VMA zQ#D*ym6hAb&f&vEez z;>#FcXr}x+d%SCyi2J_hx3rnMw)#?Tv?Gs;Y@!=9XAb`&eojShq5c`OkZb;YoI*`^ zw|A3U=%5h6_J@s@?m}>)bu`-GS#k$_xIYJf8faJP@HDQh`l4f}K{W&@nP#7D9CiBQ z06l}U-q!r5{R8suEH`6}r{dJmO6lvy-`HBgEJ3<508^*(99?7Yl1s~@)H4Q!`6(ZY z(x>UaBR4U8&3;88C#E>Wm^{o6_3bgOCIFl#RowUXNtn4qhfcfn?ck5!C1XJB&i_pT zZ}fNHiUifO!gisB0fCY<|G!^(5@f>CbqBCY;N(JNo7}XK!4yLXn?c=|tj$w*fnV<1 z*?R>%DniYYSdoj2{t@#mrl4c6Kd;2G!oR}AaCaEq+2G1I8eVX-Z8dwQgd*%|i-o6L zy4s?fg3&TVho;-)W>s%LU2@E>4(CB8 zHhIc7F4i?}HG>_%Fa7+$Yx%}v7PszeE0rFSS^AM$DUAa}{)EgdpI+wdTww4srJaz( z28YZ8ALl~c^0#r(Go)_6pEU$PJB9I;HOV1*xwNSP5TqTPS`>GULi@}wKR*qwBGysC z0^kte5GK|9}0Yz+#R1gt=n$^QRW|NKno-0P&v0fPajKz~;pRBD|wP6@N^4Uc|PE>X7AW}dNsCxMw z5Wsi^`d^-Vbh_oDUO8Ns^k3EQ%-7YQ`LTK_=pVD5wK^{P+M^R10s{xz6)s4vL^^NW zUp`RCUl9^KcQNktH~}a!`PZQ`x3VM3j|v*yX!j)m23|E9|Lg*4EmMLzj5a&%R(G#l zAvXI9iv;%)PN?37y6K_bEvDgGp8Cu6LMkSbpNN2Nuym4dk#SK8cv{BfkN(-=I-H9S zh39YCY#4p1g5Xy}aWLP;OHkfd+GszxZONQ8UUs3FbRS;rEX{pgExAuy?@-}ySC+{0 z_pI~{iOY?h?y-g(a&i%ACR;dxKUTADq7N-k>ja5dZpSTb05Bor7)HPR$cYv;tp;*V z9zgskzS94BLy$LFcHw2_^i_-!^^Qc>vBOrrfatuzTdRH9VDDw)V@ciL3Qh~BUoBqL z3-CX?e1GNy;(0dLXF&#)F>u>E`??>=U}tyd66$c(DxYQ<-N$L5?6f*_Q(buz=%5L2C> zk~qA3a{e;A$Mcq;mFL`U30c@DX|?a&pH$b4Qs26OP{sEoI4o|=D*`7T-}ru?zghMN zfzCd36tCa+yX{}odoldl$$&*Sbq!>=tO}K?tfN^sTg9jo=`@=$UIXqYtLWPPg-O`L zXTa2m-hJ2<26HS!!@H{C(aUXZz6Lh{#2D*YfF$`R;`K>~Yi^iJNwNxRaI!~0#nX$5XZ&%nbN<2~8Keo^eJ-%+|n8d-v(#S>CyFC@Q~upyBJ+!o{6 z^BiJZTA)@2U8qyPiPmFfHnGcA2R;;_3m}F3VI#Uz0YVDm6awnP{~Iy(MFY+q-itFi8|x?qU`kvi*zNIjl~<^>~_vgAea z51IN;JPsUuDB_JMQF%v7Qh9MAI)^IZpW1vri;ut9SyFEdP`VFuA(dq;J3=8=ftz<3 zLfg*|>$KF=BG0HFC|tQI2&@4ufWDZ>kpsH(-2CYaO$#uwK;5f-WWz{>`Y}Gv_5UJ; zOITx`#l*x=DOXa_D3JqrR;W8rfzL0K0=jD2E8<5V4EbLdr}!}5exoWQSX>3Uc5!}T zXQgAsa;C(;6%HTq1y%9^Cl6DfT8B(8)Bk_0nlr&1AyXH2<3-rB3pgjBF+!@emrGcI z|5)IE9i{(2isk=t#Qy&u-%mxpUII+Zo%yr$m5GzxMLuqv{S!^I>!4vzKm(ivX!*Q{ zZt^`m5{6fBa8W|HUt-ciqz2wyy)Hv!OZjAl<(XJCJbC@x2}fy;+NNznxjp zxsI?7m?9wJ4>%YHj~p`VX~`kZ?0Nc?l8O=BfF$#Rh5vgqac4n5?pW3<CF(p&zk!c)oLCE~q zGbtjc0qcs_J#ujeS7k<}TOKe6xBpyt@Iql}K@D8q#m;>!^L4>EB9UWOV;76QVUiEW zq*T=3PtHfa-N&M+;N`2Rv!GWyl7EARl4@y3>L%Xj1?WnRz+tznxx z0fG`yX+!<(6*{1a{LH6@^X{Zc2sLt}iMeTRS=010NZ4~0ALzOoGGRGH8{2+1YM1zM zlC7ovd!AU)vIoE;3(q=C@#b=1q377@mtY^YM*PwFOtZr%l+V=hME}o5Qz)9?{NS&` zb;wpwyTKjnzA?}7m603%*A_;^)0c&GIhhAD&Y>`0!@2DslD^oG4hLZZYU(utI23Ih zfPd8zP3`hp+2k~YIcaG(zaIZe*ojJ6`6IB1SI-bQem=y;weREORs;EiZ&+oY(#X-!Ia{0UBi_zDD!3ifWu3bpUq2la zkM@c%{er%I_t;n7#8cwF_Nt7Jf_=n0#pW>ba<^{Y{nprE--+597`G7;L5i9p87Dh0 z4Jr|-gbrwgyqB?Y%fVg=ztEG%o&Fue88pi7Y<_1#xOaWD3) z^SbvX9XFcyVvC0tH7KkWs-WJ)&YxW_{UPlJDHDZ_ntRsBH(7I6y?R2dUBksl(qxLm zRsXYCK>P%Ip%Hq*&YaP9R%Z<;`xX)gVz|V={Z5bD`R&&nE3f%qsch<~FwH17W9*)o z#Kt_R&7Fjy*bZHkeqqve@h^nwoB zJ9cFN?~6^qrHOxKG}oKHP~-XksL%GeowHngwPkpEZjyV+{@}Zd;HcZFFd<~TzEoJc zf@nYUtaB2Bb;6E-nPJxU}?tu>A zB8=Oalh06>TLgX(-&^Rh(PhcVNYw>2TL&V_Ul^Ij5&W0=`_l$}m+7pKtz>Ba$&%E| z$Wzzmk&L{P4Q$q)UU{+fufO>#BZw6rLbHv%g1NI-!HV%H!t@?CQZtpsyAH3jnALGqVew+Tr4r^#RZVp6X0OC z+(NG)r#9~DMQT^qImOyKPGDGeEMcjJ_HMzC@!ch?c1yoQ3q=^)f zqEbUuU>lUKpwdO@Rk{!$C;}?IOD~b86zS4cRC?&Wse~FJ5JEyt&I;`Pe9!ZqbKW^~ zX3l*7ILw&2?{%+wt?O5=)%rT=^J;=9?5w=?uH08-zB1ReTp^rpwo0&;zM*+&a&zQf ze-*9>`AKxtu3cB_S(`&7!zkwt=N#9ijA37{dB5vluhuC_f0Bzd`ME;Z$*hs^PDdkw zrt`Gb6P2{;E02zr)Aw;`T|1RDa_0BnH(ZEaD<-l1DLlx|-<|y$$W=lzgF$E|IyrS} zyzy*eg3@Mwnd3G`AEf;iI3gqmqij?79*N$~?p9pSu=DMgYo3}PYf?-aL~ zn$002O8DRmu&6SpNm+vmn`o1xXe7v<4~m8yi3iM~5;Y^EltP2@pQ1+smcX_0|2{nW z2T1wD@MN_8;lp;JL$4sFD&f{YNPD?RMaW<41MC zB7#Vsxp)vSRf54B2t!D`)&NYn5lyV%0bL{nd5V6Jo{%yVu z9A?m+N#c~q6~I4zBO?^4Q7llfsaad+CEcfi2)#J~$u3i5aA;(tVvM%pw(0=}{!Bzf zq_4Nv{OB2A`yv!{__MD7px0QVxW^bb`i=s%oc)mS1{pJ4l0rTqVpyR@!7IdNB@XCp zkX#DK4^WNEfjh^DF$LX7L>pDw#nVdp=)LPZbBSIZMAK5$%?p0xfOZ#c3Atu+2=`4H ze|~l=;v)3Pw@!-YGg&<))46U%%B>e;?*3h^nF3v)0p-rqwwR)Rq-gv&VX(K5$3lnj zr9uzCwKLWH-u4FgMQ$#C|1>h|FBszPbqNUzPBSrGn_TK^`iE~ysOX#xip}S@=E$l6 zqDf8K4$N+n5XTwZw}c3ppk8E$yNSg8W3MSt$SZEWaWH}!Z|<_Eos@*ypYC0|UvMLy zaLOB&^q1s)`vT3B0uTHfE4zPCRLL+9g~sTm@>|niG$i!l9!f6kDYLzjIGrZ%4TOey z-FMV2!NX||A=r5L)d{G0QdV%7vL=XdvVULt&{kvkN9sN%+7c|GoFuk@L!h#~%ih(W zyierab#jjbydc@Kdv>vIl?$)qABu)%cMZt8=9E|xmPb9W1XS=Gpu4kY1tf$45FfS^ zhXl`bS5;H@20^=NuazCWDkaEJKl1+g0ZjRh!O?XH7ag3TJj=V1S6~628`q~%lx6ev z<#F)5q(k&K?ruuzi4!M6rOv`NQEb#!Y$ZvC)hea8omV`vli1#k9*%;sEL}EzbM1!M z5V}zO8iq!P?RnRL587yehwO9dA$=hDbaAGGw+gCi8-GA@1o5-rlT$qg@+vC_MK$we z{B!|b6cyJ2hRcfj098HBkFw==m$EFGh>zo+x9j!ine_aTTmZl_t8#kGCJhv_et@l0 zMXnY$B&AYDX`wgzRJ-i@g7F>MnB1&<9r4Ow9ueL?4f@NB2k%~-lZ0Yk1zV$1OVu}k z(16V!|FQsd42OAt6l)&%E~tR(@t7Bs@y~Sc1M`Z1KA$Nuw-oea#S25&B>rAulA$*G z!^YJk`DO=mDknq=iV=Ke`WC!>>-VXp!QYbiF@n(%=Yx9tW;nbi%%|9s3tsFTpx-^6 zc_B{rZR%cSx)N*8VZ!>D?>VjKz-t%bs^XG5W(Pf)oa8RBBsp{5&UNTs;;N3VYq8i4 z$@YTKfJPBEqtT1a)HatI;AoTg&<>sBtKg6)@S&UzEGnz_ge>WaRka zl2bR?N!uIqD}BDc&9Dpm<5z|>8XQJ%h@au@8u)@ge4bhTn+?T%eE&;L)PYb^Q38rQ z4g7@+3QlfDEWR<8RI}&K_B!>B?O+PoVf%8_lb8}Iy~z=uaAQQ*Tu3WjJ_-8v6PO#M z@Wmi2JMKgNiJc0VmJRGbl2aSD%_Jbb=rGr{py-i~`D9TSyS>1Y9mibr8A`8>`rgUjfQTDk* zXD(@>=`0R^2sMo@UwxGQ94xDV3hase%6s626(z~M~+esX7y^2ig z62K3S9{g~dT;&FUh|CI0F1o8tCUOE zVp(YJQ;TV>(Ozl6zER$~^f6&aPj3fHf5mq-^|zOl{hYyamjV=*BVex+*GKLv=@2E* z4x?p!Zcv#uFM>evk7l3MoKRQKKlkfYJ5TI`ph5^h;PJy|2nFEgAU#FTF4anGBr0t8 zB{nke3YMI)_Cv8*JpbOnF$|DolAL7*mgt^Dd(-ZdHi6;uL-p%}^MUx@$4CTbM;ZZ) z5v{=4JT?bGD_=??pnh6m3~g}K3HCLbOcXgynk*izXJ znc^DfE9N4S0{7NJ5F*wm9G`z8JeDC?B9H?l*fC|(F<&eMuaZ{6qI_p47_29y(zl`mJ761|kRbQ<@qf%Q|h zf19X>h^hQT(w^-2x)_M6t$$JtU6$zS}B&)ToAFPYvR-F*9V;niK0rxg)Lfx3#^)VkiE=$f2X z*=7(jU)eUQbY&t3*%XZ+-L&XRK92i(qGmSRY2K)~>HB*HnFhm`xRbVO3=J6xMX4v; z*pvS9@$^Q@9>_ES_U~+~$n<5dNA0ruby-v)>4uDPCRuO8}zT zo}siEW+XK}IHP(tnlw*<)(Iy1k6=P-SHIktm28-PoFTu0seDsE-Scd&w5%TTKI^h9 zLOf1@Z6O3$A#CILggC5BLQvW93*7Tc>bCN&E0| zUh=vUiP5#1w7l6IBUd-{nknG6Wmkr;5vMo$%yuGTT{20@K4EW}ODX0fkf`iJHiJ&T zs`Y&8dJkp%F0Rpx)OV^OB#rA%$WmqLw!N1vBGUgPxf=H<&(YNhP}DI~cF!J7?DxIC z*ryP`v9f)gtk^c?y!M5M{d_i-sAsCydj=u~8QfXJe-TujI_Qdz<<>^r1M!4ie-op? zBc+;B4t+Fd{q>BX!)f|9>_7bWL`Ui~+f?bKxr5jNjBkbQD zj*_2RrnelB1WM!(pZ(i5?4pi^^ZND zXLNt?oNRs`(eY`crZ^-EyT;*E!m|G}h5jVLpJOEpdBXrDy~Sa8v>*h+EK1z`_<8)M zCG1k!-R=Fw{HfaVaY52dT~jQ*^D&>-X2h8?O(sqLo)PZehSD$7!7Nm}e6eZE`S<2? z=?U19fnkoSmHJ;pn7yghsOV}ys{e#I_HistQidqwH7KErm|5+Xsh(@wm8X ziH$F1xY$SZQ+59WBb$zm$)%1<^^9C&(x=;~aP*hYt3NLs6p!&piTS|#V=>XReBW*U zV>=)c>^Nr@nc($(lG)rZ+f)^9gQRR3#xf>ySbOANMO5JpvB8CqG=>+%&CMkK`A>{;9KU;GrI+?D%WAhJ zEA0+j0U>2jD-cpz#!t=PmDf~FagZ|kwBJ`JPJUe=l61~uSE7vAPiLT{PP+6%hpySMj>CCQ4_@iqMOO7T+e^KNOcYyRivJu`8-UB zu`&2Dx{vNEIvbL3Y+^@WE4IM@(=J?^06UHyW!nE&S|-_j_3{Xahc`IK~dDRf>oDu zzp)kM?bd42bUE#cOvNz--L*Bx3}`usbZD01^Lgv1YIZ6LY9%dRJ+y`o8%%q=-wI+e z%}ioD*9-_7`q)^$CBaAnk>ZKj$eCe0CYNblFr?y;H!*XJd%fafu~}4w2B#u9Q4M(3 zZkN4Bs+s5EA-NPz?&gJ;?hA%ZrM6pdcDVbmY;V2>1}45yM`&=3H-pl)Mv%Nec36S2 z&y_=0jy@0@e=t>!V$fv-+eBx8(p}I{RUaM-3K$KecM^ zZpxL$lVHjP8O)t6RS993mAr>><$!*W;y6ME@t%`&))eLI!SCmHy@4eaNHCXZ|7^`u?m00Fz*;Dw@% z=cgDM=lw9a{=ShkuPIw)-I2mikZb2;-()8=Ros#NKs7WnVxtJhXY=F(kEb?pMa?%1 z;*KQHELLA@V78ALGR?9>TsZO$idRqcA7Zu(9wKo$vqexuc{zh1rAN6hQUs6`DD9$X ziJyvYNC#%QM>aCVUv1H2-~|AravoajrWbAJ*?j3lVu@}mW%jk!VtQbm& z)Cxr7QE6yX&j-INu*d2=Og24&x1-G!@^*FJ2xR_skN)Cv;)3|k;S-KlLt=WAXLZ(4 zZ~8of^hM9Et~xC)E=B`w;|;OXxnZYYmGy4R;7@Nm`(ca@d>gBsOcs9Q#(F*@$o_7* zHVPk9U@;%&wmwmH8z_!cBkjy-&OHtk!J4@)(%dcWrsB9Ygw)}JFJJFp?mX2i2&|8U zo`Cd;D@fU-^kxBLot{$$Cw}Y?J*1r+C$^?5Oj#y=7O+CpKH-&|)7XONKKluO9E^)( zpo~8Q2M4%4c1-3$tL&-M~YV)h8}7VW6_2*ettZpb=zbcbPKl zZNjwjeeA;0={Q!0oc(uRJX)T3YU!9O)%69yXEq<3*hfUM%cFebXj^cDdxzi>sasxP za#;mPnAH57)~L(l*k!=b@sw%8+g1x&B8h9-|jwNeR@39XB(q87Rvlwkb^oF zcxrSYv<~8M86%!@;i1}P&$gySWVt}z8EO1B)#^+Gi@$k^W{_gK`@94^WV(fezSA3V zY+`sSdRS&v(bSW}Wm=l|XVuxdsq}Q7#f>UI6#T(*1D6ZulMwV&kx^~kZq78ZUZHd< z)|Qe(n1*M$Ar@bklHl5Qk&NNJZOv+@!3kJUnbm8b%OLm4RQ>7HvzmlXp-~OT-5m9ERCGX3z1&Fj{-x9Ze%}#>~(&1 zuXnX>9ar{OmZDKv=GM}z=;PdzcdP;pQ$6UH2h(av>0TKtD3OrW&wg@!t@AVWb9v(I z9p0RWC>fCJCfz_P2D_OA2EfV7T2n^W1Zy9c3^N3FHFZT&J>-B z&<+2vW%w8sH`#uQf&d0;*Ap%5>RpKxy{<-$oG@fj&Be>BdRrJAE+=eUZ|eir^LhI{5)$qn? zOicJroZGht_>JB~X<#&hZJdoUnXgJ^>E$!3#xN}>&ga%3yWehjo%S-Sk%dx+AS3(x2-&9{p|eF>2;8cADli@cBrIev{?O0awgvkrQo3ZNx_q zE_Hk9?0DKNURHwqD8DPiOGeLVOj$#mYjpQGT)C;DY6 zjzSBjM^n4!y^IRJyngE7GVOJasBlGV*Jlbi(P_k%Cp2Uyr#5YOIk$t|*CaVrmj$d- zbwkd6t0g7!hLYHb#{Leb#2Tc!0;P*neGQdDZCYWbK_UMj1bSDw8wtZSOl3K&z3 zmE{d18k1>#{WH{!J-X4^m@?ioY=)kgs&)d>U1^owrL0@vo^>758ROjVmM}Y?F4;Vp zrh8eAJwu`#1iQ`e6AyyEdt+yWek;%-d0M8?g*^E>u^R`5BmCCDregCYxG&}QPfr8o z(_9MDw8HkD^&9#=#;49SrjasN+=U{a*Po#UI#SEP$5#r^1D?Iq5QSM~_If-goX6i& zfXn{fMg%Y=rWnY9}1~5241}UQ32-bP^o!EZ!bWYz>vVyNS`{Z0@u*pz2P| z7Z#`nRoz;SdZcx|TL1k!tezEKbWWQkEu>*eBq-)qx@vSCTds% z07)jFk_^=?`yF#vUGGO-oS0Ae_N}StP+@BAHK2C&?EO0$e{pt?d$WjijP|K(ncc4_ zE^-QVM3guW@S$oQlKyt~xEbho^+UycY%e7YMNJoUx?gNSZDDnv-F6%wjurSheQVkA z9M2G(Ycshy-D{8d-QINw@h~VkwenSwg5r*|p87b*E`lRDN@geg9{ZIBD?7(e!UG!n z{TU>8yRHG?%<(m|TOr%}>Gx_6j$3V=59B|Mi)d4&XsYY1t=IR4ry^f`V@M&eS?%U{ zl#~bMtuth+f|N z$M1?yZFJhB!yj395?dAS_9)TKa@=AmUihc!SopMCmF>ueC@GyPCAInXBgusYT@$eP z^EoJtom8WnqrIE|zMDypcIQQr<@E*)EiGVec-}-t7(rYTC|gHtMdBWsbt>h+bgTqo z`(;*K5&V7Sez6x91~buZ&NomK((^Rg#-&`^UUr7(OQpNIt3&b0Div;AN*jCPWw7f( ziUQ6NoY>9tCW*tte-II)pQ5C-BZyl^MxpR;FK?3wr`r9}UhIYxiKF*VF-`dI_)K`e zFWWfr!+$#U-5G16phmHDFfE>~a&3&Pz|&SvyMCK^O+?R$k%A`5Wtv)Gws#DhQOT0_ z3+wCV>6z#Qh(mB}VIiMZ85YSIe`r~OS1}I3Hb*FTe3E;;tR?&L7s`&VegC<>d6l;R zR{Qdr+@GNE%c1*?gX>k`gaBZM3fxy7h+XGsN>C^*A7ACuFJnSnFIZhHN!pQ{-qy-Z zl3i9U1}>FJS^f&`!ifCoQK&*5u&Yuxx1CRaqxecX=cYb;B*sX=>1tYY3z#-BE{`DK z(Utf;a2M=Hhp~bT1*-_iqoSnI&bd;W8J_cTay1-#Ah2ImRZX)O?3vHzOF{j*9JKgA zY`h&T*>Wfg$fGvOfgJh#`cW&kc5~XlpbkqExP?vpna&j^{;SCXufV+h+ z@0gL;X(ttIy=Q6&xU{DQ6Oj)sO}Xxxa`k9P2N8?TkKcuP&kv8;4y_sF0NluQr7vqjT{vJ zP*xtH0a(_?`{zg%;JsO<{-V7|x*!wkBsCY!gK9%3%vt8?bY$})v+U57Jc6ruY*-f4SFf)*Tfvlng5br^30)$@KIqcT=7O_1#+km@poS=kBF zcTeSOXYpI_r}Uomvj%pOaqgbHaHj(|mf`$&H^Lq8E2SyGrT(qPJF=K=@ zQNRlF?;rDpCP-Ig>3g|T@F3Fph8_)ta$CWn-ftARkwqU4m(xViD*y9o0Q1Qm3iW?{ z8{je?acZ#8C|{S9)J=t9+w(TAS|H9p9F90&5LB#QKx(0v00MeUc;4eKKbWr5)!L8N~-8V zZwzL!OHU;>lE+W>?Ua!2HkSQ~tKXYxijHL@tJP~Tv+&qh^)nhzvSM-`iC1!Z7V)vh zp#ToYEWi0Zr#9TyRSQUL!5}O}HS*<|c^i`E(9Jj`d@5SN%q||W`Oqx-9AQ4slvKLc z6)Lb&^k6%fR^4a+ll(4P8!b|9{C>A-8`0*q%1_nh4u+8PjoyN4vBdqvpC{kBfRvDF z#@w7$9kcD>osOfg`hC`KqE@Ci%%XwGc6B}cC{T0y?@0^HnwLoA=9GD@x~N-|FbqT& znp?JiB=0jaGO{@I4N{vV7i=sN4&0m9t^gQ#Jd!E#8z2RM=_`v4ViEtdF|?Da%U!M1 z%8n^DY%<3_g}q!}nSGV1*MEEI34o|-d(*s=L7?6?%6{i6cmRtC{0N`~9*nRl4PC_Y z1@n%Voi@FpID>ZcXA5tPJx-VbJl1QK?R8&Yzbg)Xr`$vWa?Z}RaVA2pvns2N^#|D4 z^A@LNmOTc}Ful}jxZ1SWtcmNHOmg9FvGw+vhLx>l)s-VvZf!j5aMZ-MrzAQj>^>B$ zI2rw5tODMzDV@aRQdL=*#BV3EyVXRtp^*W;!oI-cfS-B6uR<@kNiNprjk-X#?@Hyc z#;CfP+ZsHx(Mp8HpR|#YNYQ-a_d6?A0@^=6U(ZgJars7Nefb#gEsy+PYEit^*T+gp zC*~Bt1}|bs&r*DU^8G{+`c$~rtv{Vv?1)Z zgp6vzyp1LywAkn2BA;WGcdj0TdW?o;wE*Rptp%q1Y+NWn^{+@Z8cMKk)F z02+$;ZE1&F;aC>YNDvinG(nGA?8N(YNhdKv*-ls$upGyJjE>bE; zASWyY4M-1^(*w^nHm^v80?DnsLiERoDrWm!0$ai6Jtx_Aq`9yb*b6H3GERcX|NP;W zvy^0=9xagm{M*#g59$DR=e-6#w83@M9cKPV3JOQ+g7ED8KU?YPr6B(`cr3%gCT4M4 zz$&bnvQ0A$CJq2v39APHLMw0ov*YPFSAZB(UzXLp6zl}0wvLXK>Tcdyce$>9X!EAn z(DDv|i&VqEPQ83LT`^GRO?U6hV(V?d>ijcrl$yYgJyVF{x{p8KT|H2Z}i<44d zsVxMoZD5F{f|zT9916AdKm_zP6h$h8y$IUxL>vhKMo#}JAkS~^R=lFE2p^epz3BfS z-4|!DE^;Iw8F=&N=X*?(UlbxReBRq}Q?h(wP56XD@>il|+V&O50L6wps3;cXjL<*K zIL<@&u6pY>@|cy=E(~-iB>+8opQ*n?V2rGAw^f4D71E2#5J46|5`Sy79v@=4Z-i|v z$JX-!KiRj1JwbV<2~B>b`u=N3uC_=#F7Sc0U_mps)d++5CF&(QZ9PJBs2qP0L7f&? ztz`v!fgZTylF+IRN{?mK^SR>U-hIvVAj)Jd3gFO@W}eQ+4uAjS?YW{$MOPX&v0B>?2+{%>F z_ySE530eHXu)W2eOP>2HdK2UNGZK<1%;xm$Ub?E!9LP1i5lH`34S`g8?rphjPh$NA zBIR+fOrr8B0~33t2#DGs@nWkvM`m|q})B_;otBnm-N85yYh67>hqD3UUDNu z9&$!K5Dd}1!d+xbVH;d_he#qj9nqhe9Bo#Mt-XDn5k4?aZ$uldGFzck>Bol={!Z-y} zr`LUq@cmyX`QWK(?{saJg)BkBXO#FePlm=}TkJU5nD9Egptufj$mdH_TZW5`GybIt zWY_tl^FYB`~*8-w+C8qHP=`)+bKFXkcdv*=_@FwVFPvNwbwOYd{nA*-?a`) zNjUhIRLp0-Rbs4mKr7y!bK!lt^u?6^CY!|a%;>cEco_C6Q4O zR6#rYzJI$1kH9#@c12=ZFqx-Pnj3F}3}kG8MT)caN3OZ?uJp?fhljq{yxw@uF0}}A zNkx{IVSGB99@>lx@5BMxeDco_?_VBcEuEZ`QsExeKtOnQmvP5aZED6bhTic|K={Qo z_3auF<(+8p(R*WOwk8$wz=1vJ4UiEg1?Voh>D>3`)UBUe3~$Wr{AU=GrcZu*_NC2qANyap$FRdSGgFy03$igqiLQ*;ISQ#_HK9#?mvoy4)vZ zk&JPo2ny3bU1>F=yE>1#G0QA>w8y^6`32N+*(VUD%?fO7%}cb@vpn+c1lt=4*0RR<3#~lJH~t_ zgqX#-;0;9T0~C?^@Je_eMWk+SbEVVZ!RjzTP4^Z0H?9MS7KZm7*$j?gWUs*3cC1^u)1I&f4YAzk{Z68YPk*sfMuqLH`XQs5EEtwZ( z;&$0r1%o5LVA33+JbBZ9jke%76yr2$ZG^Ii^y&M}yb@DhqJJqOu4w2lcvLzjIObxc zKIpMDOid7<%f4->0-t)+!JzEG=9Vh<@pOdQCocgsR=&N0Ollr@zI)_8@Y$nxZ+p6G zE?Cw)E%qdM9-e06hF>Ft-!3m#0_FMdT@9pt)pX9VoInWny4&+K z>^SLQdcUf?_=|TY-etn|_L1mF_7;iZB8Sr}ShvXAKJw#PvimI$tm<~*2kESm>~`BT zm@ar^UNeyxZ4i=>S~cS4tv>^0ew;G23wKl3^PzPjbn5yiMOPjfcDZJt8Gq*t`AjsYtOEdd2V+IuaHs+O6_tlSfKjs^-GJ08$Z zsr>?h-*!{5j!WAO7FUGiOM}LqW0D^Xlq1v6BaZ2Sw8r1BYj0oKT|8G5Oix&8wGH!R z&vApl{PsEW%x7l1Vrt509ud4+jjlAo;~(CK662|JOa*RvID5h0bXq=)bY~#6_5bK& zg2V-Y049PWvJOm6M_)(qdH@Jw;*Ngc35M%zRGALobxW--tV7796T23$%!|25l^Zh4 z6UQL!K>C^KA9gAf#l~DvZsHm#ee9qxL_XKw{?=k~ZdbCZX&`SYK^*VL_YM8qX5?F1 z$RnA5C=XRjaTsE-SxlVz!m~cNwlk^93z?-5`^{>r+S6Bj1RpuMRiuX$!g=R`E>e+R z`XWL0Ec#!Ul@Ik#TO!&>`@3D6fX>^fDLR3BISSN82X@3IzS1el_K8*6^A4wUl^=| z{|mwDb?qq5_B~3uqVL~dnH<&VMx)VxoYN`mblkEj2i)vAY@*!QD= zE_`4Om7PdB1WlcjSb7ilf{UC=~O>j z{vCghS;<>n>un+VAwY)=%oFTseCE@++K9tRDQYyg zUj}@z1<|UP9GNSwkt2vb+e4sG?l!%6Qp{*E=i)Xmiej`m{fZqY@T2NEO13cc!ZzBM05$lJPx>?##!f4Vm! zQ?ml{glT3kdRp&H7%Lm_(DA*DU~XcqDVQ$bxv75k< zPwstYEjWtoegJtX-S1^pq7VFMw0WZ%WgSW!Us{`jk~JE!pJhd(M1&frn{Hx@HYj4pU{qUpnd;?$J znK1?C@i%9o;gzQSE=_o96efo#=I6O6HTtP-xgZCaiioSYJf2&+M7`^2VJ+cls=nr3 z?Kr}GVX~(`Fs>@1;7wh_ALo#*627_GE}y-Io4{7sLCHw8ocFH8z2Tac7t{L68(`mQmFI|>!_L(XL?vwJLNGBEZrN(kB5Lx$_ImZh z0ngU)Q#thsG8ieI7TUW>8TOet=O$5aVQW7L?PG&q01xj6sAq8AtK!3&YVaKvN!m*> zd3~{bPpr)UC4UUK@!5Sb#upUGJlqH~Gph&;(J0LrY1r>mPjat3Ch=+XgVPiZDppii zM^S+QF35JqnQwlq;Dqh-a0tK{{$B}550j4njf@l+{ZfLEgWS)XH?&Z{or-_onc8Lj z_df3drduqG)BC&VtpiTK$!1Zb?v#6kx}eAwgw*eLsLlWLUbp4^Z)7 zT?xwuOfUC3+|S!W@+oCHKhy$rvuk@kyE9Gh;s4%dV=C;zk()W7Xa7S?x|IFD5tFJu z`!_M^)PWI)UDki~Rk-OV2Ra%E5yp(00Bk{_nro!yGx0@)hu;CoJ=WuPj*gPVUfpaE z+ooWoC*5Ny0fzC3r6f`s`9FsT?8TcJ0;(5TS)(cPRD8eoNiF|;u3Ky`>1=InqxC;N zxC2s(?(Xh7QYt2YV+4RhBhBXPK4(37W+5j=f@U$Nygc^iE3otrGLC@SBvw)A0bHWJ zLCFbC+pF#GCE7>%tG{~`m=_z{+a$bUr*|!Od0|jxfs2uloQQ%-|FrxX{?Tf*?@b5F z`LD=Jwqnsz5Bs%Wfr35%Jdtq9>XIHNA(9aN^T*tdp05?|t-D5g zZ2Zi;#qV%?Q2%3vtB)Tu?o`($dgQ|)z6IQ$vl4*ANayI3m|3GkqQzn>N~iyPfK3vaf_x|r(Ic|T#XkoV~%BPSq`+`%z`9ufz?6rKrNCkd#>0=2H5Y@-91W4@d zSq-J(aLbiAb2t0ZV^Osu*IMKWB~~S|Bm;Zg?L<_Dl5_g_*T559ZpSGW1dt~+VAX@V ztxeauy8YQS7&>bwUuE=bo?Pi`-&N_i*0`{ECWlDI9v9NwEB49c*mgR)*6@QzKF4Ncqw^$;jpp*Vqkrcc)U)&s57MCgqN zYdv0AogMqWO9IK~J6z`Pwb$ZZxIU9g-8E7x)r~2FB^jSn=Rkc3+_ke*+;I_*B zQm1Sd!_M7xx3J#a;t1a7iNuK+|J?4dm{0@kV8q-jLYxgDi?cP z)f)${UcJSq^4AEZzHWGq?d7eumpttox71T?M+hgR6v|e<1%Ze(paNW-TU}<`&$+#; zA^Vzz7&?V}pc5nZd@*C=WuD8=7xKH%sX;fYd@I9>K5G_F;R4_W8+dn=`f>3u3Igzg#InPUah4P*@LypjpZZ4?nE9k3$IMB`4_IP<^7k`lqGP-Pq7;53Z zBXLc8n(Z^{vgjpi>9(zJ=REIhhl|-sTHe@if@K6Iv~ z?_pu8;YV%X*rt^*F5KR5VBHL_9|-Lwc+Ay!}>sMw2H8(xzce+Jr`JUUc?GEKz(%U|E z{4B!N!%N00NT#8(ZjagLc<49MudByQf!l$o2_2W@(Q6x&e%qPha)aHj5N5vJp(Z;Y zVeQbXNid{Ek?v8GCe1p&yjbFVshEYll_2G@YZfwH@N+`EpXp@)9xQb6PQZg}I1kbr z%h9GjbZ_!mvyBAAM(iW_=lwVAq{ZC}G)7TgR7T^fTQ)jt7i}LFt7f>WEb4k_6w-8_ zdd2-Xm=c)C?=F`EF(nsR1baLShA)_Mc3|54_<;+X&mZ|R&(|zPJER9mXE$dd8o9~n zXn!_e%C24Rh+I9!ZNTOPxD}UM#k=KswdL1eH5|8CJfo!q%Kvn2Qq0aMXum-@KhXhG z(m+V|1s+XvJ6y?%y!+*@h}CV}Cb6^E+zxP!!iw9|kymFHw9)bNX4TTh8%_0qDm!^O zkGx(oyLP7T+RPCHqSF4&il+j5^Ovr2?pM1GuMmP~+Fy|M{dcGKaoej*xbfE>K}ibu z2njP8+hY4;_!kC^CpNd+7ot1QSO8CjcwnXe#rPqcKgDW7t70iA1FVrnL zUx9cca)dA63-~XIj{x8FS&-XmV`s%B`qG6{b)|Q5vLTR`#SL@Bh&@x4wlT66=UxWM zaqdr0%xt7gHF?%cRW3rJ@UxQFu_$VG`kSM(KHFH;NCmmgJ?+cRKhy6kxgG7HHP{zx z*jiksqKnU z18)=Ai+o3Gk*1HAIAWR5K(?Ij)_^Wc11dvAPYcRUQRz~7Bv)0nDgm`$-;khg9d9b< zY0S^h-$~O05@q{$WcI6v2L?b&PjtUp8l>vB`<3?ZQ?j8TVKv{KA?Cw;vhU8ZsEEg^ z6exPM7oOMoyt>9j(Z*JMIFPe$#|W-zK{*)8E9P4lPtt{rygDBL@;9Fnw7lhZg(S3F;Ta)6Ns`2dLA>H^|v zMMf}}V~i;f8P%_SJDClOE?`VCnq5gq>DtV6HiL`1wTFSD^v0?j)K<`$+4_eUdV{>8 z^2GCa)~%c^wJ0*_KK-q7Ex*emc@aq%pM<@{IjSK`1rRQ{N7R|9eSA=Kg9p|TJcedF zA8Zo_d@p=Jg=?+m?q)_M;pyEeq6Xw1N%s`l`|Ha+&HiwvUSh|iov4Y;S^Z0ST57*t z+EXkVAGk%<#}7mNW&@>F=gZQog)i7ebQCbU?!L;#V(cz-5H`Z3&L`7x!?*U9M*$y| zQ#a3i9<27uYY=e|p1;fb_AL_FVK~HYTztbIP0C#iUXSwjbiX_|y&Z}6fhlf>OUakG z!(I&ZrF3p~U>jym-Zs6zfpOg;Z*(lkLR*4Qg>b(6U$9}ytdvCb;i>O=G>ZRaT`j(+#*KOw=cO! zyDxkXeCRxGQ*1q2me4TU?i}UWVKBuy=BLD%^tAP&6PIALWxILglvrLmIf-@OgL*}qhtG4`YIujE!gHybk|Ez5vTY)3JGm2wLF%Ll(U{l?Sj ziDh(v|Cj)B|3>;%FJA*E30P#l-dziAog-}i5>(#q*{WpL8HI4v`4L}9_V4cB^N`5q zsiW6obs3J`Yc*r&KYedgGu?^6=d3dlxO3iE^%#7Y#L z#L8pcePhXK8`gssQZ!WI5>_g=X5y8CrPUse)LUNt>%oLr$4tVvpT&1(2AAz|O~K$u zGOKaTk&w35&N}kg3Isz}a^!p?(=QL&pJHIMHGEP6b<$`r6b~faJ~v4^=G~LxB^aF0 zpD|+ZbVaz3Ma`DAN6zKL(z5Uk#&N>zp>!2%KP5K|6>8}#n4y@FWga4)y;8z@>)9Os!qUuKO6TZ0vXGs#7#!|(~3`}4rjsykny?R zy_LbCbR{DxO!35CmO*t}&ClihH-ujU4yra!H0JzqwGt-n>EHAPynH|M?NqSrjE(tf z{_;t`cKbzuJE-3g)$OHwYFe7ZQ-KyA~3>+nrj>%QrPT(`-}HWnYes3Z-v!i!!Mb zbc^KZgvx%rcO$Fg;n9BH5r1*GBRciwU0BUtEU(>_sOCQ}@Sqvj+*RM_@Qth0`#T5< zT%)pu`23*dAi0GmQO{~m-{2D>(`eQ#JpCWbx?uFwJYIOqH_g9k*JLdh2Rl~vZKytO zw{^P<@MxxfA$!k6Y1sGhaTvna%y}=8AJ4FBg6PQSd|Yo3F0XzJi@TIx&imxu;pq zw{c}#nC~;|!R3;NQ%vIRW*fhHd@tgf?tM3a3$R`uSabXB+Kj3jKhuxg>Z4n`TJ$rE|Ak7| zT7-kX{NJ~EfBVIYnsZqK)0_3g&V2b<68~QzJ?%y5>uV>0BXs%sj2^&ipBn=7_kNi) z2Y8Ouy&`_w>&ol7Oy#Og7%~MeU0k^0q_{Wm_VMPkr=K2PHap3dJ@&o$y80QHuP>Rb zWcTAiGqm@~VQOWi_5X|ie-+?TscqhO6ohQF)=12qz8p9rk?C{o-khKpRemP=6Biy+ zU8W;GeO)oI+ia=20d#+i`Tn22nR$A03mBFF2e}=Vcu&_m2^^IP150K&+D0?mux!dGo8*@>Bg= zlXnW+zRI~V+blPUg~LG(7%mZ@gI6qTer(wL^wOsr52x`-W<_O9zv6afVrG?}L0Su& z0I(EO*ake1EaDq*KmSbk6N~>^Ss9s36}nmS;>@)wqbmvPC;I&Yu9U1Ze9rTe0eCDk zQ@|w9E@I&MFViloKe?!U;$D7Kx$&HQ4 zpT6DBe=cMD?BcV`xzQO*3OCNli3%yPo1e7he)?45nlP*B*G_UTEEaf~X|QY8E{VU4 z0>I_e_ov_Y|KA(4_m$jJ(It!?%@=OA{d~4UzjR|^-^?!tJ>S0S%qck? zZFOn$mt(2MpM+C^7nl}-7L4X@zneCl!2vW(ad}yy$LwYHz`Z9Y3`E?RFWp@B^Vu)p zcvspHb-nj2E4JUSvv#jzoFN1ZvQ1XNVO1VJzEd9`AAbrQohZw!%$VN~yvl6DcUA9Y zdWV30_CwV*QmflV1QA0+wZF0;f}#?%f+3wl>OjvvI>FVAfxu#p@QnF2=C^ z=Ia#`@B5nHP5HMUSnL-wHh4kq7n*gia5V!11Mng)CI*I;zk7 literal 0 HcmV?d00001 diff --git a/doc/sonic-build-system/images/upload-package-automation.png b/doc/sonic-build-system/images/upload-package-automation.png new file mode 100644 index 0000000000000000000000000000000000000000..21581a81f67595c0ef14b38bf66dfd49d2bacacf GIT binary patch literal 13209 zcmeHuXIN9uw=QZ#5T%NA3>rkG2`ELH0fB&k2nZ-mnxORFi3SCw#;73DLp zU|=|^i@0LUz_1^~z_728i4oj^vQAZj%RXOY9Sw$}ZoxV51L6!ffHN?ZCbDeV9R$Cb zy%0Bj85lVI_Wt&b-#HP)z#u23dj)P9WJ4wrW2Xkw6@JaKrFWJD+U&1QO+|nB*xdXq zcGbYX!?3}dv#B8UsIraVX^+Bd8F~>OneWISNkgm?ICzxd@J8U1ra4eLW;iC;imw|zH60x75ga^*aa39MA z*0QLDhA`ZgM!`T=j{izm4X2&Wp{DL7V!ZV)JX&<|t6 zs9zPz6~==cXwthY(t1wU*@RUrstj+(Ky=@IHn$BQsFyLH&z&a*lC1yUMJOY*@l%<* zTh2L#HoI%KyP^mU5zrglZ`_zk?Ibnd4PU{Dp)R4md^-Tv*|p z=T0Ip2jHAvL=YNU5M3WALPH5W{}`&v3ei0i4Q1x$fZ-lMnWx!dStn62C2$c$!A!59 zVTVw#=bHbnK?UUyYHCfY%JmjIUhav`v(=t!T@?F(fDe@QA?oifw}73X4b9F;Ds_gt z`eH@b5Kkt#kTCmI+|MfQt4Fb9kVMDGrjNQ>CUN}UkA!tK&c6#x;Dbqbd#pWt6`n4O zkh1xpjwY|%>R`F;0ACp_G;~SZ_1NC-nk^k@X2yJ^?FoG~a!N*D5kZ42Y*c)unB|e2 zMiO>MSpJcm&nUE(Nr`Gy&gpjY-B?DtZ5rOgR&9K%0fV_@cW1owc%PqU_e^k`PocCX z!9HUW?n7NDkn~;OPLwn^5idRwon?tvRu(vnAv)PgwFIj)t%XoGDs7QJJ-J4yHMgQ@ z6V}OpC7;KFAsP*^s>C9z*>K&O0|Tw`gH&lv$JupXftibJMyj(cMK02omh{R&TW4&t zSNJAP#XJwGQeWeO9seRSul2b;%;WG2EWAva6y(vBhm}}fxU*9@WE1SR;k)ixO-o^I z*irkdaiyE+MP@JRN4aa#L}+=Ko9H`uUXE(G`1z^>Ycyo)#x=`^p#_eA3}(s^?lazU9D6JM*4u)$*Ja-v zjvw!n$QWW(CC+-t6bCci>T-3eyBZrTcmdH(LW&{e)zd3;Go!0lYr9-GEbzWMkyBxf zu|x55X1k0sncZc0Bq@UEWNldTH54oSSG^u(IxBkDjOL=61}{)EDmy#s@WeYdS5Mi! zBZdY}pL@O5*j)rKQw?9I@`V=22*dhTG6R#m`K)GV{R3=Wf!M2pi-%7AvjaXaqoz)V zS*^`WY7B=++YX=Js?58&Ro|rGn;I;fy-12bej4ujcj;XzjLgA{?LV$YG%TJ63H*P51-G>0IN)Vu(Vl-+x1wL za)aLs`sHauBU64ygi|87u0YEQsO$bRuqu7B;_v1Dj4;y<=8ofF2U z`(J_uM@uzx`;L!42$DK{=81oIhwL;|smZ{0!BFCQOZ5gpTP;gsoDe_(c)D^DU`?(t zj9}-wW;0A@hk3^QHFME}xEUUCcp1T=R@QxX{9A0tr7SnKrVpWdiQUDe5uQuAGjgsZ zWJ7*$sH1d2V|6#xe90GUHAAJ*l@`NGuEGCV#?y%qn6&2Tg#Wq z?GsJTq&hbZuq&xGwX3|eF+f&I5QSUX@I>}e|F!YKJW3-zC`|IQCx@9YnBpD|B5uwi z5>{I)BW)BQHn*Z0gDK-tHdsS1tF5tD&JEfRYnLbjRA|i)i&ZD|V(`wqYhdKP?0Pq@ zzc7lp9$lq8$ac|_2gcJ^q^_peR2zW*;iL9+X{LA}qPwi>C{fy?4PD~|{`FuXN; ztf)#+$7h4mnbFd_v9=X>66?!a;vJ6U1i^2jXw`{Q9fAUl%8J6 zdPrgBQ}K3CYe@h)?nJh8F_(mN9;M5Z@3+~^NWXnm>_)+Mgh8lqVNV@xD}c>O!n|B? zmb~7_<3c_Ccb7bmbTr(Qdd0#`NaV<+2WNp;WSe%kDb7us1H~Z<^YMG>S)AjuTHt`&Pp;C zE)(L{*M!`mX5w4!(3edEm&Q_5Lsngs%YeuOgJ&K=nN7t|KOzsY&A>~gSG7vwY}(Zv z;IdCPbQpwQAyuUJOML8S#$3KQ(c`yznD(f?-{JS_T;Fv6HgF;C9Egq(l~X-cp(^`w zub?3@N!@l~lko~6EmmnQHCJB}E+S5!ZDmc55dx%bU~YO z1;pvOlFgnQKjun8Ah++2ffrr3|D2x1Vq^a$MDwi8VFkhCO47rC`^`e-`qdvN$jX0egA0s#;3rg~=gYYa#Cg zL9m7-d1BqKiHQkq#+Al==*aO;BYO=@n9K4fd9^PK^3%N+2Os5OG|~DKRDJ<*IW_5O zdq{#V8sae~uj-cSVI6c%=^$A3uoCV2ekU&21A$U5Z&)@ zC(cbN3MvnLK>M;`aJiN;+V|DlZY*6M0d24jvKA z<1af+%y-h$9MqQq3NVY)#D4Nb3YwcgmN?$dM*n1VRDTN|IT z9ZuV(ga_`ZW+`x*F=I^U1~PZUL@zY@?}Ts*JgInaLb>=WHmQRdL%6Ce;V?^>X=wSz ziq3DOV=2XH3e1(jjX&)U-~7aetk^QRd9TL*M?MpVe)fl1Tn_TuM-arhJP+awKk~`A z52!&4-H_q#(e{%1^11h|IFb9`fvb&Itk^E}+-_M+MAlVref4WYi#xQGwK|B7*Zneu z;93wZ`faQR_2IdCE&9{)r*ojpAHYh?zx;UriZP~97d7=`TJcwSm~mMW zZ%lOl$u76~Pq^*oNBNwvYo`M4w}%Wn9MWS`9#es1&KbyDsq?oJrZHpkXO%-Lj+`h@ zxv-fts>&tK)W947#cC{Q^YyGh($a48UqOR?1t99V?|UdubMN z7!%EH=e=ka$01M5XHK|^es}~3+K&mdk+NCHYxf1BTLbrYy7%yDW4vvh?0`sJ0~01V zR;Z|y<7lpzxb{nM;xLqml6Vr!7}fs8ccHm~8PmtEeA}VgKCUTU#_Y!?L|2icT1p}k zsw8QG7XjP5p!>)|IHB97n2ZxwBLEf8>U9222e;^5=pr0D~$? zfihcqEogAU%hU*h;a;<6?};LsnPv@oMG=FY8$Bh4=!e1tE8HB~EFGfT3=o7r>%lE1 zOi9Ju;G_1WQ%snOHrDCkm!-YhaeP;Cz={+Cs$<^;brGcy5Z%pI)|p{|f-ZHj+@ioH zFw)^PQAF93bd!505ysrnSv68{jJuL`6hBbtrzgvTF;JYE)kmFth;9tGrWfz~NKGzS zZGwB$F}c&Cz&RKTMjrCjN5P`MpFZ?l1O1ja)x&va@R1-2ip$zqvIjalElHc7bPNbx zO9Mldy7^uZ)exJyf_B7=)uB^QB7EFRCl|Inay8J7##OZ|y~Bxu&!D)oTA^|i5G5`c z&IZyF8_VKQ`t%?ilgb{A?P&iPPk}pp_u+!*mT?D03|ELPEwQJ)g5nHfgl2}lg_UKw zl*bdgyw0IuH+5|>9@dB=aK;9L1?O-W%UsFVo`IOsPXWVG0NbbT>}`^cu{m|YSST(& z=D{aY1f4uC-J}>Mq-m%4G%XGEkqD^CwytWrYSNZaXDQ?3f`W<74nERPf#QzUaG%Np zOORb|18de#hvJ^Qg^Ft{FD+h0PyL)Q>KrCFjfYZHqi?-D0>|XX2(1}Z+xwm4sON+! z#SGW`FKFn&5VHOrZJ$M$F@pVldDj#&lDP!zU!H?wj9U}7a;XE^njh^v+nF$et-1Sr zh;`_*F+$mm@~UPs{fG4p7NPES`>qtQ^#Bc4yL(zQnaE#I88A|UeSY4y4(3|M0x$FN z{(Q3eyA%-itHp2P`3e|g!AY~YyWjo3oJ5#OS%0dY>x72qyIo4YlPt~sGR8erbWPj% zjh}dXsV}K|PUW%t75!qJatV}UoBeS5suuh15|{aUn}a6!*T&d*>EmNB&%e8&1i|UGy$B&o-?#Wg5O1>NX?}08*Eb(m1a{(FPmY4AP==O*>QvHGbmHZ1Y`UP|gzX^3lv4(cA4)GHwvgOARS?Ja9m3o2O8J!#ZNdvuozCeArq?N*P`8TMPyuoxHu9XQSW zL!p@o3f0=E9pQZxo1y6JD0a ztj&YicPTw{v@v4sgY8$f-`^vhI#r4!A|2C$2m54Xf}%iSQw5X?C)(`Rlv zv}tD@^LEOKAG72V#%rP#6t>*PGb@THGCSM2IsajNJ+I&3#_IPMCqwA;afiYeRbto4?gpl$)%jm* zO}q4_&cR@nkwZB3-3|O+W~|b@-MAvsQLrjHwTIOzhY`2Ch2OQ>Id1qNJqv=%h3I4iT}{M0SK|jG2&XcTu2lnX zb;|{-J&5xXoo$N+8$?6uS&Kc*)tI4Gf(XQHqVU*Psg|~p9@h>ZVn$*3*boN4DTy#N zs3u$aV&{lEUT08_*{jgt1vBRs#TsPkS^OZoW>no&T@`CMEl5@B_YO2&g6;V`dd)Il! z2=3!XFAv)mz_6&X^1nxts z3cVjtUs09ix;FMzL8EZ!CIKo%*qNUs%adN>Jxho#+mgy8^$++Q zsqFI*z#e|*u&`t|NC2@_d|BjFnTV}?k4sJIem}FW;n#&1CU&@kOHprMl$4)5S^!BF z9~qq_4p|t6Zp4;4ow@c$u}f|8_0_~u`SRcERb=_rNmqeK8>@-VJUoTYg+lx`ZqQLY z)@8q&3dLvRmWXVUyI(|lNyBDs9Cp_K1Kz!$VK6sXGQ!5TiWYiplzPiHw{g9MR;|k- z`vC2(nY`R-zR_FJ4^M(FS?l~Wap>kB$3D@H9ov*I-%Y}zd^T}mWeziL|wFz5C zt?c1rjO!tsH37`jy`qB_mFK>b)3kD_aKlQ ze0bX|!Fs7Qcr7f-ZbwNf7aLM=(Va5e8$r(4X!rf5t+3ko$3Tu2+Y)I#?pF(|QtqW{ z%OIC-gxa!TvdMn0$qG?J#5P~n%JCUJVAv;MeTRL=Z*nAwf8I()XGPj3Q_V5a*c&xD? z8{e>cs?3B2BscL|{*|jH1V~2!`kf?X+`d*X`mW5@#srWD1ChQXG6%~$oM+nCTU@yVnLbFm1=;bei|B1fTrYa zqa3$toI*(#tQVF$r+MA-RNehNmg${y=7YqR;%X&Mn1Da<(aC%G%E|@1Nm1e>F!hNd ze7Nr2^gM`*0>!Ppi9Lb_^Tv)3m1@q#almSsD+{Xsv;PJ;OS_V@!n<{F6igu(wPuEP z0|7X6Z|nV&UYz8B34n?i!u|aFV}B}RKY}3tY3#iMnxZhgOay_6>}Sl@16NRIOVIi6 zVIKV{F8*6(Q=$IQf)BY~Pde+rqPM3YGIV49Louz%W*FZjx*DzPQ*$V z^!px0g8?k`zc`a!B=XiV4|l@M%jw$cQEt`fg^Zg^$vOT?>~2rgM^Q z=825{;<*c{nWNZ7kRojXAQ`nqvBMjfM9D4(Qi9Fz^8cK7EX#G$h&-VeZ zz8kjT22Q^Pmlv0JSCX3+t?$y$vEmuF)wWmrI{igih1%x!Uv2_PF;6GDteK2t-zkLI z1n!v_R@|d{9Gq6CKNWg2MY?IL zERb?Of+re!KO!9aG=EjES;I=Aueyc1#mOJZ<3HP-J}sAW4jki>8fo?zP>kKPRv8P0y|M_AR){U)wE_a<7#(0^WeNUhm!)hOY~+Sw|h!Drx+7X+uz zkA6N{tUoMmm_*CDoL@zb*UvG>aJB`8rH%Fn_Ua3cjyqY}zsoFR!YslX_3brNSYz{i zEXDPnOF>=Cl`Yz}E9t+9^?}LwDA`sEG?B9P&V}NA{WM7Yj9e8Qf9xwh`Agf)!n)T$ z(mpRAEAq%Vd~3M_tCKaMyxW_t)0zJ+9m-sC0FL>lg;-S1Ii=zB@Tv6Z?&7U~$UojUD6c^Jn+P&@c7>es{WoMs@JrjR+=8a&kA^PnF@8!;63v^>O z-zkL8h48?hQs+)kFL-`3;aD45l}!2h?!xi6zNC>TDDHZWOiP`uX?LaFdx&nX)P8$N zTUIGFJ*HI(qHAdHa^#C*ma=@;d2STUggg4(q|us4JL=&T^ps;%`|CLAi_u0e!8z^h zJblT)9lX&h8j8bdxwuidJAQCS8~pFGVlZ7faIlem@}alw<1rWqz`TPs zf#MvG#77>F_TS6ydo=ebHslNOZ=*nFOZM!t6TetO;ClNH&}8^E{nKZVGlJ!`D3^*P za((%8Q`C?sa~xL71=|+@1*x#!_BZa8Ee_9Fj^ds|bxXe`m&%+`0#kO_6H<@!lpgx6 z1zA(0RE!e)Qj;B~R8d+AB{Mx!hGXp8cwLFoccP>A@^_lL4)Cg`Y~wac{6X9?Cd@HO zLxcjOXfzDrV?Ag8sqs@QWgP1&fjF&&etVehgxaDIQj!%ZfQ+{WmMb2Aa75u{&=}`n zAcqMIVI2RPlTv@`%0u1ZB#;W6c>by}eS}J-2B)Gx{y5)I(Ci*5+l==%4+>WQ*u-VR z^i}^IV9WDa+Sx`qdZ%-yB=Tzagx*5U z#HGr3)8%bD354EhMDzV6Ev4RUc5|djwZkFYpP~U__cwV&eV3}vjNv*DtRgMaT>^0r z3i>Ku8E8~Ce9xP_GMT7uW2og0h$@+?bowHlL|r?3rI4CP%EukY za@sA>{eaYSG%AYx7(MsCa}#i8s#>x^c%;}SYUn>Fz#_Z!Z0`l95Y5N!CuJ_6y|yw! z1jP{%|4P%3(0Q6(?n)qLoPV}6_OiaQz8nwH9fk{b&ySBT^3LL(7CRio)qI?(C((Bn zDWIH)JPxDE_3*-7? z#S142%?Uo6mj>1myBL**Q^W=cxPQ3%e* zji1H%{==^z@_NHt6ZZ2So(ZOp&qu*J%I`{icJ36PeX!T|K(Rpe!cH2sC|3)eR#oc1 z3|v@e{(X05jMvFt7EZCnz)<+am_)87IxV*}`#_?w>Mg^Apt|*tD^~ck;sf>cGTbLmT#amzXkp_e5oP{Uzw>fIX0yOZa<-7 z>t-XhK2G^TfD%lUqS{e@uMUk#z&TH!DVk4U;N#({;a{r&jZg5G)F%G{b+BWt3=aY* zzfM%jH=BhidKMg9_EfN zk8UT(S$)}o;1&d2Q6np20MPS)!nw4#5#Z1!ttixZ9a$30KV0WCoDV@`=f>*>qzH>a#FO> zV+LV1%St(Ti5r50Qchy0QB#3U>)Ruw!mFkTG-UVzv^HH4zucX!EdE30T!78Q)@O`i zK07QQ80Tut57e79&00OQAdmSi5;inYR7Jb(EVp-&#}fzmVNAb18Zvk#@%mu~Q6ZEH z1YTy#&mtQxPmUoas~mSEDPg)3-yR3m%9%Y_U^&qK+KL@!o^yTQCL>p325z(+6-Ha6 zlv9wDdg|Z8rA948_qDm;pgiOBCAg2s9)1>>rU~4nTcFi9f3V%btFEVy*s9UpHU!Q* z10~X!lJgfBogTguyOtr5R}c-Y)xm&b&&R=&7aFPE%Hkq#s%&ulk&X?sKi%DSiP;kkBcsx-b?bTR z-8*xcJG)=F33C$yvj}5w4jR6hwtGCfVe$WP0fuB&`e zOfGuf@?De>_tUNcxf?;TKrdGpkWuGuhoWF!UK}bQB1qdDzyiLJV#o5Q$V1o`W$ZSM zFpzkV+3hr9zmAOjZGT7!lcR|#P#wQl#22K%_f=38f7i~M+P)EFBDV(RQ*HKs52sVu zZ)kYRc!hv*IW>8F^vPTG`HVP=C8k(1UJYQ>M}Ht8T`jLTD#e&Rb=2aa2@3}J7zyWF zGp6k*2#7Tx!=Fr1j>;v~ouD@;zFZDC3DFvU!lL1`Ak(Re@YsVi2rDS=m-VR~OC>0U z)iUA?9N_%=f{KiQJYcw*=HFsub!O#fZ}#*t1+i){v#Q=uxWB)U*NVg6VN@@*y>N#9?oCJW6k?WAhD=71^ISUc$F}N)2bakkW2_FmR z=Xh{)7SLm<-?^hB*i;#(Wg;CRt&P6$YiC zrhnx|AY6fBIKI)%H3HiT8WNnvx-3E;M5m)kXawBuq4tw($SX9#Z=I~ z3Ax)?3Vm=AxI`gu(md0XaMh$#)YPz23a|jZQ;7Y36)m=bN4{i1Q;Ikhj>C+&;GC}~ zx3woE1%b<;R@1N*xN1g=KbWE+c6knv)w+0W!({#( zA&+yL-_R+Zq~g_^jvuG*Rb?`4Z}%xkSK5H+V|Vh+sh5g?ZpNj0&zslG-Ijyh78Fv( zfAMm|;h8O`c61_bLsXR2l|6StnZ7o7 zkX{*xB@4L*Q#SLJgD`V|O=s1!sV@IWMZ`D`0PdM#g%f>`k zg&w#8$(KA~%lJ-9QRwg;mgIfXHtHHTH+OF_S#C90EKP6qMN+4Ai-+Eup%&OMWT%7^ z2X&#yJqvybt?isFWVp!Yl|Nf@txH@uaTm`uBs=yMui%Dlm%VLsC1?Rs*XHWUXJJ`{ z`i_Az>)!_(BxR;eKDMVn*$VRVUMfcEL z@tO{Sn@(N}rt23&POubq%u>R4e6LX^S1P4Hs6Kl!h;DxS<;}!u%w7oo p50mr1`Z3ABK2cHay0d%e;90t^LT(Ncd``olt7UklNaHr@e*jQAue|^O literal 0 HcmV?d00001 From 5f3f688b2a6df153160f6f915777376d6b8bfac7 Mon Sep 17 00:00:00 2001 From: xumia Date: Tue, 13 Oct 2020 14:17:24 +0000 Subject: [PATCH 2/7] Add apt-tool approach --- doc/sonic-build-system/SONiC-Reproduceable-Build.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index f7b5d8cf35..52638ab365 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -247,6 +247,7 @@ The steps to do the reproduceable build for root filesystem. 4. build the root filesystem using the tarball with frozen debian package versions. +Another approach for debian base image reproduceable build is by [APT-TOOL](https://github.com/pauldotknopf/apt-tool). It supports to freeze the versions, and use the versions in the subsequent builds. # Packages downloaded by wget/curl From 420dab781bf2722d8e0b7e9009f4ca594610170d Mon Sep 17 00:00:00 2001 From: xumia Date: Mon, 2 Nov 2020 15:09:42 +0000 Subject: [PATCH 3/7] Add Golang, component level build description --- .../SONiC-Reproduceable-Build.md | 24 ++++++++----------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index 52638ab365..65eb453699 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -36,7 +36,7 @@ | 2020/09/21 | Xuhui, added Debian/pypi repo part | | 2020/09/22 | Qi Luo, review | | 2020/10/10 | Xuhui, debootstrap supports reproduceable build | - +| 2020/11/01 | Xuhui, add Golang, component level build description | # Overview @@ -49,6 +49,10 @@ Some of the dependent packages can be controlled by its version for the reproduc When the reproduceable build is used, to catch up with the latest version of the dependent packages, sending pull request to update the versions is required, the automation Jenkins pipeline will be created to do it. +The SONiC reproduceable build framework provides a way to lock the versions of the packages in scopes as above. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc. + +It supports to control the version of the docker slave image, and also supports to control the slave container to build the targets including docker images, host images, debian packages, etc. + In Scopes: 1. Pypi packages installed by pip @@ -65,7 +69,9 @@ Next scopes: 1. Support on the old branches like 201911 +Out of Scopes: +1. Golang reproduceable build, suggest to use go.mod to support reproduceable build, see https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum. # Pypi packages @@ -102,17 +108,7 @@ pyangbind==0.6.0 pyasn1==0.4.2 -The version auto/manual upgrade configure example is as below, the automation pipeline will not upgrade the version of the Pypi package pyasn1, because the upgrade=manual is set in the configure file, the default value is upgrade=auto for the other packages. - -Py==1.7.0 - -pyang==2.3.2 - -pyangbind==0.6.0 - -pyasn1==0.4.2,upgrade=manual - -To generate the initial version configuration file, we can manually start the docker containers and print versions by command "pip freeze". +The current versions of the images will be dumped to the folder target/versions during the build. The initial/upgraded version configuration file can be generated based on it. ## Pypi Version Control and Check @@ -123,9 +119,9 @@ In the Makefile or Dockerfile, if the version is specified for a Pypi package, i | Yes | No | Success | | No | Yes | Success | | No | No | Failed for version not specified | -| Yes | Yes | Failed if different version | +| Yes | Yes | Failed if different versions | -The Pypi version conguration file will be used as a pip constraints file, some of the flags no used, like upgrade=manual, will be removed when used by pip command. see [pip constraints-files user guide](https://pip.pypa.io/en/stable/user_guide/#constraints-files). It will control the installing packages and the referenced packages. +The Pypi version configuration file will be used as a pip constraints file, some of the flags no used, like upgrade=manual, will be removed when used by pip command. see [pip constraints-files user guide](https://pip.pypa.io/en/stable/user_guide/#constraints-files). It will control the installing packages and the referenced packages. The pip/pip3 command in Makefile and Dockerfile will be hooked with a new one, if version conflict, the build will break. It only installs the version of the package specified in the configuration file by adding the constraints option. From c7172baec0a6a239f405d815d17d67efdde195dd Mon Sep 17 00:00:00 2001 From: xumia Date: Tue, 3 Nov 2020 08:27:14 +0000 Subject: [PATCH 4/7] Add more specific feature description --- doc/sonic-build-system/SONiC-Reproduceable-Build.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index 65eb453699..8ef48d35bb 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -51,7 +51,13 @@ When the reproduceable build is used, to catch up with the latest version of the The SONiC reproduceable build framework provides a way to lock the versions of the packages in scopes as above. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc. -It supports to control the version of the docker slave image, and also supports to control the slave container to build the targets including docker images, host images, debian packages, etc. +It supports to control the version of the docker slave image, and also supports to control the slave containers to build the targets including docker images, host images, debian packages, etc. + +It supports to enable some of the components. For instance, only enable the version control for py2 and py3, while using the latest versions of the debian packages, web packages, etc. + +It supports to only enable the features on some of the release branches, while using the latest versions of the packages in master branch. And the default feature flag can be overwritten by the build command line parameter. + +It supports to upgrade the versions automatically by Jenkins jobs, and developers can freeze the versions files on the dev box and commit the version changes. In Scopes: From e410945b1f3cd15c41fd76b792f211a4b1b465ba Mon Sep 17 00:00:00 2001 From: xumia Date: Wed, 25 Nov 2020 07:12:43 +0000 Subject: [PATCH 5/7] Add Q&A and simplify web package process --- .../SONiC-Reproduceable-Build.md | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index 8ef48d35bb..1adff00c38 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -37,6 +37,7 @@ | 2020/09/22 | Qi Luo, review | | 2020/10/10 | Xuhui, debootstrap supports reproduceable build | | 2020/11/01 | Xuhui, add Golang, component level build description | +| 2020/11/25 | Xuhui, add Q&A, and simplify web package process | # Overview @@ -51,13 +52,7 @@ When the reproduceable build is used, to catch up with the latest version of the The SONiC reproduceable build framework provides a way to lock the versions of the packages in scopes as above. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc. -It supports to control the version of the docker slave image, and also supports to control the slave containers to build the targets including docker images, host images, debian packages, etc. - -It supports to enable some of the components. For instance, only enable the version control for py2 and py3, while using the latest versions of the debian packages, web packages, etc. - -It supports to only enable the features on some of the release branches, while using the latest versions of the packages in master branch. And the default feature flag can be overwritten by the build command line parameter. - -It supports to upgrade the versions automatically by Jenkins jobs, and developers can freeze the versions files on the dev box and commit the version changes. +It supports to control the version of the docker slave image, and also supports to control the slave container to build the targets including docker images, host images, debian packages, etc. In Scopes: @@ -458,3 +453,27 @@ We do not manage the git repositories, if a git commit used by SONiC is removed, +# Q&A + +1. Is the reproduceable build feature will be enabled in all branches? + + It will be only enabled on the release branches, master branch will not be enabled in the short term. + +2. How a developer adds version configuration for new packages in a docker image? + + The developer should build the docker image or any other targets, the referred packages will be collected into target folder automatically during the build, the developer can run a command "make freeze" to freeze the version configuration without manual effort to manage it, then checks in the version changes by "git add files/build/versions" together with the code change. + +3. How a developer upgrades the packages to latest versions? + + The developer can upgrade the packages for one target or multiple targets, and can upgrades only one component or multiple components. There are two steps to do it, the first step is to make the targets with version version control option "make SONIC_VERSION_CONTROL_COMPONENTS=none \", the second step is to freeze the version by command "make freeze" the same as mentioned above. See more options in "rule/config". + +4. How a developer merges code across branches? How to resolve the version conflict? + + The process to merge version files is similar to merge code, we need to resolve the version conflict. To use the higher version or lower version depended on the usage scenarios, we would like to use to latest version better than the old one, but for some release branches, keep older version may be better. + +5. How to migrate from one debian distribution to anther one? + + In the version configuration files, different version files are used in different distributions if the version files are relative the distribution. For example, the version file name versions-deb-buster is used for debian packages in buster distribution. But for python packages, we use version-py2 and version-py3 by default, it is not relative to distribution. + + When migrating from stretch to buster, we may need to add additional version configuration for debian packages, but not require to add version files for python packages. See Q&A #2 for adding additional version configuration and #4 for version conflict. + From c8695d0bd2332592c8b519e523aa33182f3437b0 Mon Sep 17 00:00:00 2001 From: xumia Date: Wed, 23 Feb 2022 02:43:56 +0000 Subject: [PATCH 6/7] Update scopes --- doc/sonic-build-system/SONiC-Reproduceable-Build.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index 1adff00c38..ac94417271 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -38,6 +38,7 @@ | 2020/10/10 | Xuhui, debootstrap supports reproduceable build | | 2020/11/01 | Xuhui, add Golang, component level build description | | 2020/11/25 | Xuhui, add Q&A, and simplify web package process | +| 2022/02/23 | Xuhui, update scopes | # Overview @@ -61,10 +62,10 @@ In Scopes: 3. Host image created by debootstrap 4. Web packages downloaded by wget/curl 5. Debian package mirror -6. Pypi package mirror -7. Docker base images -8. Source code by git, full commit id -9. Arch: amd64, arm64, armhf +6. Docker base images +7. Source code by git, full commit id +8. Arch: amd64, arm64, armhf +9. Branches: 202012/202106/202111, after 202012 Next scopes: @@ -73,6 +74,7 @@ Next scopes: Out of Scopes: 1. Golang reproduceable build, suggest to use go.mod to support reproduceable build, see https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum. +2. Pypi package mirror (Multiple versions supported) # Pypi packages From 2794527f8e169f9caca9d9dea7d1f50840546e4d Mon Sep 17 00:00:00 2001 From: xumia Date: Mon, 17 Oct 2022 05:26:00 +0000 Subject: [PATCH 7/7] Add explicit note for file hash method --- .../SONiC-Reproduceable-Build.md | 959 +++++++++--------- 1 file changed, 478 insertions(+), 481 deletions(-) diff --git a/doc/sonic-build-system/SONiC-Reproduceable-Build.md b/doc/sonic-build-system/SONiC-Reproduceable-Build.md index ac94417271..0ff8371f93 100644 --- a/doc/sonic-build-system/SONiC-Reproduceable-Build.md +++ b/doc/sonic-build-system/SONiC-Reproduceable-Build.md @@ -1,481 +1,478 @@ - - -# SONiC Reproduceable Build - - - -# Table of Contents - -- [Revision History](#revision-history) -- [Overview](#overview) -- [Pypi packages](#pypi-packages) - * [Pypi Version Configuration](#pypi-version-configuration) - * [Pypi Version Control and Check](#pypi-version-control-and-check) - * [Pypi Version Upgrade Automation](#pypi-version-upgrade-automation) - * [Work Items](#work-items) -- [Debian Packages](#debian-packages) - * [Debian Version Configuration](#debian-version-configuration) - * [Debian Packages Version Control and Check](#debian-packages-version-control-and-check) - * [Work Items](#work-items-1) -- [Debootstrap](#debootstrap) -- [Packages downloaded by wget/curl](#packages-downloaded-by-wget-curl) - * [Web Package Version Configuration](#web-package-version-configuration) - * [File Storage](#file-storage) - * [The process to use a new web package](#the-process-to-use-a-new-web-package) - * [Work Items](#work-items-2) -- [Debian Package Mirror](#debian-package-mirror) - * [Debian Package Mirror managed by Aptly](#debian-package-mirror-managed-by-aptly) -- [Pypi Package Mirror](#pypi-package-mirror) -- [Docker Base Images](#docker-base-images) -- [Git Repository](#git-repository) - -# Revision History -|Date|Description| -| :------- | :------- | -| 2020/08/21 | Xuhui Miao, initial version | -| 2020/09/21 | Xuhui, added Debian/pypi repo part | -| 2020/09/22 | Qi Luo, review | -| 2020/10/10 | Xuhui, debootstrap supports reproduceable build | -| 2020/11/01 | Xuhui, add Golang, component level build description | -| 2020/11/25 | Xuhui, add Q&A, and simplify web package process | -| 2022/02/23 | Xuhui, update scopes | - - -# Overview - -There are many SONiC failure builds caused by the external dependencies changed. It can be built in the first time and might be failed in the next build, although there are no code changes. SONiC reproducible build is to make the build stable and reduce the failure rate, build on the same git commit, expect the same result. - -The external dependencies contain pypi packages installed by pip/pip3, Debian packages installed by apt-get, wget/curl, debootstrap, docker base images, source code by git. - -Some of the dependent packages can be controlled by its version for the reproduceable build, such as pypi packages, Debian package. There is an assumption that the content of the packages should be the same for the packages with the same version. The other packages do not have a specified version, when the remote package changed, it cannot be detected by its URL, such as wget/curl. The file storage will be introduced to solve the package without version issue. - -When the reproduceable build is used, to catch up with the latest version of the dependent packages, sending pull request to update the versions is required, the automation Jenkins pipeline will be created to do it. - -The SONiC reproduceable build framework provides a way to lock the versions of the packages in scopes as above. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc. - -It supports to control the version of the docker slave image, and also supports to control the slave container to build the targets including docker images, host images, debian packages, etc. - -In Scopes: - -1. Pypi packages installed by pip -2. Debian packages installed by apt-get -3. Host image created by debootstrap -4. Web packages downloaded by wget/curl -5. Debian package mirror -6. Docker base images -7. Source code by git, full commit id -8. Arch: amd64, arm64, armhf -9. Branches: 202012/202106/202111, after 202012 - -Next scopes: - -1. Support on the old branches like 201911 - -Out of Scopes: - -1. Golang reproduceable build, suggest to use go.mod to support reproduceable build, see https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum. -2. Pypi package mirror (Multiple versions supported) - -# Pypi packages - -## Pypi Version Configuration - -The default version configuration files in "files/build/versions/" for Pypi packages. - -| File Name | Description | -| --- | --- | -| versions-py2 | The default versions for python2 | -| versions-py3 | The default versions for python3 | - -The content of the version file versions-py2 example: - -Py==1.7.0 - -pyang==2.1.1 - -pyangbind==0.6.0 - -pyasn1==0.4.2 - -The versions can be overridden by the version configuration file with the same file name in the Dockerfile folder, for example, the content of the version file dockers/docker-base-buster/versions-py2: - -**pyang==2.3.2** - -When building the docker-base-buster, the expected version constraints are as below: - -Py==1.7.0 - -**pyang==2.3.2** - -pyangbind==0.6.0 - -pyasn1==0.4.2 - -The current versions of the images will be dumped to the folder target/versions during the build. The initial/upgraded version configuration file can be generated based on it. - -## Pypi Version Control and Check - -In the Makefile or Dockerfile, if the version is specified for a Pypi package, it should be the same as the version in the constraints file. The build will be failed with version conflict. - -| Version in config file | Version in Makefile/Dockerfile | Check Result | -| --- | --- | --- | -| Yes | No | Success | -| No | Yes | Success | -| No | No | Failed for version not specified | -| Yes | Yes | Failed if different versions | - -The Pypi version configuration file will be used as a pip constraints file, some of the flags no used, like upgrade=manual, will be removed when used by pip command. see [pip constraints-files user guide](https://pip.pypa.io/en/stable/user_guide/#constraints-files). It will control the installing packages and the referenced packages. - -The pip/pip3 command in Makefile and Dockerfile will be hooked with a new one, if version conflict, the build will break. It only installs the version of the package specified in the configuration file by adding the constraints option. - -## Pypi Version Upgrade Automation - -Developers can change the Pypi package version manually, and the Jenkins job will be provided to upgrade the Pypi package version automatically. It will upgrade all the Pypi packages listed in the Pypi version files, except the packages with a flag "upgrade=manual" in the configuration file. - -The data flow diagram of version upgrade is as below: - -![](images/pypi-version-upgrade-automation.png) - -## Work Items - -1. Generate the initial version files for pip/pip3. -2. Add the new pip/pip3 command, verify the version and use the version constraints. -3. Enable the new commands in Dockerfile files and Makefile files. -4. Implement the Pypi version upgrade automation Jenkins pipeline. - -# Debian Packages - -## Debian Version Configuration - -The default version source files in files/build/versions/ for Debian packages. - -| File Name | Description | -| --- | --- | -| versions-deb-jessie | The versions for jessie | -| versions-deb-stretch | The versions for stretch | -| versions-deb-buster | The versions for buster | - -Similar to Pypi packages, the package version can be overridden per docker image. The different part is that the versions of the debian packages are relative to the debian release, in each release, such as the stretch and buster, the package versions are not the same, so we define a version configuration file for each debian release. For a specified docker image, such as docker-base-buster, etc, we can use dockers/docker-base-buster/versions-deb to override the version configuration file. - -## Debian Packages Version Control and Check - -Like Pypi Version control and check, if the build version is conflict with the version configuration, the build will be failed. The apt-get will be replaced with a bash script to check the version conflict. - -The version control is based on the [APT Preferences](https://manpages.debian.org/buster/apt/apt_preferences.5.en.html). The high priority of the APT preference can force to install a specified version of a package, even downgrade the package (priority >= 1000). - -Example setting for sudo: - -_Package: tzdata_ - -_Pin: version 2019c-0+deb10u1_ - -_Pin-Priority: 990_ - -When installing the package tzdata, if the docker base has installed 2020a-0+deb10u1, it will skip to install the lower version 1.8.21p2-3ubuntu1 of the package. If the version is not installed, then prefer to install 2019c-0+deb10u1. - -The version configuration file not only controls the packages installing directly by "apt-get install" command, but also controls the dependent packages. - -When installing the package ntp, the package is depending on the package tzdata, if the tzdata not install the version 2019c-0+deb10u1 will be installed, even there is a new version 2020a-c+deb10u1. - -Sample script to verify it: - -_# apt-cache madison tzdata_ - -_tzdata | 2020a-0+deb10u1 | http://deb.debian.org/debian buster/main amd64 Packages_ - -_tzdata | 2019c-0+deb10u1 | http://deb.debian.org/debian buster-updates/main amd64 Packages_ - -_# apt-get remove tzdata_ - -_# apt-get install ntp_ - -_# dpkg -l tzdata | grep tzdata_ - -_ii tzdata 2019c-0+deb10u1 all time zone and daylight-saving time data_ - -If you change the Pin-Priority to 100, then the latest version will be installed. - -## Work Items - -1. Generate the initial version files for Debian packages. - -2. Add the new apt-get command, verify the version and set the version preferences. - -3. Enable the new commands in Dockerfile files and Makefile files. - -4. Implement the Debian packages version upgrade automation Jenkins pipeline. - - - -# Debootstrap - -Currently, it will call debootstrap command in each build to create a new debian base host image, the image will automatically use the latest debian packages. It is not a reproduceable build. - -debootstrap --variant=minbase --arch amd64 buster rootfs - -There are 4 steps in the debootstrap command normally, finding debian packages, downloading the packages, the first stage, and the second stage. The debootstrap supports to only run the first two steps to generate a tarball in tgz format, and supports to use the tarball to generate the file system without downloading the packages again. - -debootstrap --variant=minbase --arch amd64 --unpack-tarball=tarball.tgz buster rootfs - -The --unpack-tarball will only do the last two steps, without finding/downloading the debian packages in buster, See [debootstrap git commit](https://salsa.debian.org/installer-team/debootstrap/-/commit/25d80b10319ed292827d016bfea6edcdb51b9b52) - -The tarball can be generated by the following command: - -debootstrap --variant=minbase --arch amd64 --make-tarball=tarball.tgz buster rootfs - -We can generate the tarball and replace the all packages with the right versions, or generate the tarball ourselves to support the reproduceable build. - -The tarball format is as below: - -| File | Description | -| :---------------------- | :--------------------------------------------------- | -| /debootstrap/base | The base debian packages | -| /debootstrap/required | The required debian packages | -| /debootstrap/debpaths | The debian package name to package real path mapping | -| /var/cache/apt/archives | Contains the debian packages in the folder | - -The steps to do the reproduceable build for root filesystem. - -1. Build the normal root filesystem - - debootstrap --variant=minbase --arch amd64 buster rootfs - -2. Freeze the versions, the version configuration format, see [here](#Pypi-Version-Configuration) - -3. Download the debian packages based on the frozen versions, and make the debootstrap tarball - -4. build the root filesystem using the tarball with frozen debian package versions. - -Another approach for debian base image reproduceable build is by [APT-TOOL](https://github.com/pauldotknopf/apt-tool). It supports to freeze the versions, and use the versions in the subsequent builds. - - -# Packages downloaded by wget/curl - -The web site may be not stable, and the web packages may be replaced to another one unexpectedly, so we do not want to depend on the web packages at a remote web site. Although there is no version for a package downloaded by wget/curl, the hash value of the package file can be used as the version value. Like Pypi packages and the Debian packages, there is a configuration file for wget/curl. For the package with the same URL, it is expected the same binary will be retrieved when using wget/curl. - -Requirements: - -1. Use the same web package, even the remote web package changed. -2. Improve reliability, the build does not be impacted when the remote web site is not available. -3. Keep update, automatically send a version change Pull Request to use the latest web package. [Low Priority] - -## Web Package Version Configuration - -The version file in files/build/versions/ for web packages. - -| File Name | Description | -| --- | --- | -| versions-web | The versions for packages downloaded from web | - -Like pip or debian package version configuration, the version format for web package is as below: - - **==** ; - -An example as below: - -https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz== ebef065e4d35572af5b03e2be957a9c6c5063b38 - -If any new remote web package is used, it should be added in the version configuration file, or the build will be failed, so as to the existing web packages. - -## File Storage - -Before a web package is used in the SONiC repository, the package should be uploaded to a trusted file storage. The SONiC build only depends on the trusted file storage, not depend on the remote web site. - -The file name format in the file storage is as below: - -- - -For an example: go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38 - -For the same web package, it supports to have multiple versions of the file in the file storage, for example, go1.14.2.linux-amd64.tar.gz-b73d6366c0bce441d20138c150e2ddcdf406e654 (fake hash value), but for the same commit, the same version is used. - -There are multiple solutions for the file storage, such as FTP server, NFS server, Http Server, we do not focus on how to implement the file storage. In this document, we use Azure Storage Blob Container as the file storage. When you wget the web package [https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz](https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz), it will actually download package from the URL like [https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38](https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38) during the build, in this case, the storage account URL is [https://sonicstorage.blob.core.windows.net](https://sonicstorage.blob.core.windows.net/), and the container name is public. Some of the packages are already in the same storage account, it is not necessary to add to the version in the configuration file. - -## The process to use a new web package - -Option A: Manually upload package (only for file storage admin) - -1. Manually upload the web package to the file storage. -2. Change the web package version configuration file to register the package, and use the wget/curl to download the package in your build script. - -Option B: Automatically upload package (for all) - -![](images/upload-package-automation.png) - -1. Change the web package version configuration file to register the package. - -Developer sends a Pull Request to register the web package, the web package URL and the package hash value (sha1sum) should be provided. The PR should be only one-line change in files/build/versions/versions-web, see [Web Package Version Configuration](#_Web_Package_Version). - -1. Upload the web package to the file storage. - -When the Pull Request is complete, and the commit is merged to the master branch, a background task is triggered to download the package, check the hash value, and upload to the file storage. - -1. Use the web package in your build script. - -Expect the step 2 is complete automatically. When the step 2 is complete, the web package is ready to use. - -## Work Items - -1. Generate the initial version files for web packages. -2. Add the new wget/curl command, verify the hash value. -3. Enable the new commands in Dockerfile files and Makefile files. -4. Implement the web packages uploading to file storage automation Jenkins pipeline. - -# Debian Package Mirror - -Debian official mirror only supports a single version of each binary package in any given release by design. For reproduceable build, it is required to manage previous versions of Debian packages. The designed Debian package mirror is to guarantee that all the versions of Debian packages used by sonic-imagebuild exist in the mirror. - -The current design of the Debian package mirror is managed by [Aptly](https://www.aptly.info/). The [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website) is used to publish the mirror. The [Azure Content Delivery Network (CDN)](https://docs.microsoft.com/en-us/azure/cdn/cdn-overview) integrated with Azure Storage Account is enabled to cache content from Azure Storage. To improve availability and performance, [Azure Traffic Manager](https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-overview) that supports failover across multiple Azure CDN endpoints, is exposed to users. - -![](images/azure-debian-mirror.png) - - - -## Debian Package Mirror managed by Aptly - -Steps to publish a Debian package mirror by Aptly: - -1. Create a mirror pool and update the pool, example commands as below: - - _aptly mirror create debian-buster-main http://deb.debian.org/debian buster main_ - - _aptly mirror update debian-buster-main_ - -2. Create a repository and import the packages from mirror pool, example commands as below: - - _aptly repo create repo-debian-buster-main_ - - _aptly repo import debian-buster-main repo-debian-buster-main_ - -3. Publish a repository - - _aptly publish repo -distribution=buster repo-debian-buster_ - -Steps to update a Debian package mirror by Aptly: - -1. Update the mirror pool - - _aptly mirror update debian-buster-main_ - -2. Import the repository - - _aptly repo import debian-buster-main repo-debian-buster-main_ - -3. Update the publish - - _aptly publish update buster_ - -To Publish the mirror to Azure Storage, [Azure Blob Fuse](https://github.com/Azure/azure-storage-fuse) supports to mount the Azure Blob Container to the filesystem, it makes easy to publish the Debian package mirror. The Azure Storage Account, storage container name and the Storage Account Assess Key or Storage SAS Token should be provided. The publish settings of aptly are as below: - -_"FileSystemPublishEndpoints": {_ - -_"web": {_ - -_"rootDir": " __/data/aptly/debian/web__",_ - -_"linkMethod": "copy",_ - -_"verifyMethod":"md5"_ - -_},_ - -… - -_}_ - -The rootDir above can be a blob fuse mount point. It will publish to Azure Storage directly. - -It is not guaranteed that all the Debian packages are in the mirror, if the size of the mirror glows too fast, we will remove some of the old versions of the packages which are not used by SONiC. - -# Pypi Package Mirror - -The Pypi official mirror supports multiple versions of each binary package. It is not required to have an additional mirror for the reproduceable build. Not like Debian package mirror, the designed Pypi mirror only publishes the packages used by SONiC image build, the other packages will not be published. Because the size of the full Pypi package mirror is up to 6 trillion bytes now, it is really hard to manage. - -The design of the Pypi package mirror is only for monitoring and alerting. When a specified version of the package is used in SONiC, we can expect the package should never be changed. If it was changed (the same package name and version, but different binary), we will be notified. - -If a package used by SONiC is removed from the Pypi official mirror accidently, the SONiC build will be failed. One of the solution is to add the extra URLs for pip command, see [here](https://pip.pypa.io/en/stable/reference/pip_install/#install-extra-index-url). But the risk should be very low, we keep to update the version to the latest one by design. It should not be the high priority task to do it. - -To publish only the packages used by SONiC, the used packages and versions should be collected by SONiC build. The input file should be like this as below: - -_Automat==0.6.0_ - -_Babel==2.3.4_ - -_Babel==2.6.0_ - -_Flask==1.1.2_ - -_Jinja2==2.10_ - -_Jinja2==2.11.2_ - -_Jinja2==2.7.3_ - -For one package, it may contain several versions to publish. The input file schema is the same as the Pypi version control file. - -The [python-pypi-mirror](https://pypi.org/project/python-pypi-mirror/) is used to publish the Pypi mirror to [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website), likes Debian package mirror, it supports CDN and traffic manager. - -# Docker Base Images - -In the original design, the base images of the SONiC Dockerfile are based on image tag, for instance, in the Dockerfile sonic-slave-buster/Dockerfile, the base image is debian:buster, we can find "From debian:buster" in the file. When a new debian buster image is pushed in the docker registry, the debian:buster will refer to the new one, it is not reproduceable. To make it support reproduceable, we can change to refer to the digest hash value of the image. The docker base images that are based on the image tag will be changed to base on the image digest to support reproduceable build. The Dockerfile as below is based on the image tag: - -_From :_ - -Example: - -_From debian:buster_ - -Change to: - -_From :_ - -Example: - -_From debian@sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ - -The docker file will be changed to use the right digest during the build based on a version control file in file/build/versions/versions-docker-base, the content of the file is as below: - -debian:buster==_sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ - -debian:stretch==_sha256:335ecf9e8d9b2206c2e9e7f8b09547faa9f868e694f7c5be14c38be15ea8a7cf_ - -debian:jessie==_sha256:8fc7649643ca1acd3940706613ea7b170762cfce6e7955a6afb387aa40e9f9ea_ - -# Git Repository - -The SONiC build will download source code from remote repository during the build. For instance, when building sflow, it will download source code by command: git clone [https://github.com/sflow/sflowtool](https://github.com/sflow/sflowtool). It will download the latest source code, not reproduceable. - -A version control file is introduced to control which commit of the git repository will be download when building SONiC image. The schema of the version control file is same as wget/curl. The content of the control file likes as below: - -[https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121](https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121) - -It will run "git reset --hard" after the "git clone" command. - -We do not manage the git repositories, if a git commit used by SONiC is removed, then the build will be broken. - - - -# Q&A - -1. Is the reproduceable build feature will be enabled in all branches? - - It will be only enabled on the release branches, master branch will not be enabled in the short term. - -2. How a developer adds version configuration for new packages in a docker image? - - The developer should build the docker image or any other targets, the referred packages will be collected into target folder automatically during the build, the developer can run a command "make freeze" to freeze the version configuration without manual effort to manage it, then checks in the version changes by "git add files/build/versions" together with the code change. - -3. How a developer upgrades the packages to latest versions? - - The developer can upgrade the packages for one target or multiple targets, and can upgrades only one component or multiple components. There are two steps to do it, the first step is to make the targets with version version control option "make SONIC_VERSION_CONTROL_COMPONENTS=none \", the second step is to freeze the version by command "make freeze" the same as mentioned above. See more options in "rule/config". - -4. How a developer merges code across branches? How to resolve the version conflict? - - The process to merge version files is similar to merge code, we need to resolve the version conflict. To use the higher version or lower version depended on the usage scenarios, we would like to use to latest version better than the old one, but for some release branches, keep older version may be better. - -5. How to migrate from one debian distribution to anther one? - - In the version configuration files, different version files are used in different distributions if the version files are relative the distribution. For example, the version file name versions-deb-buster is used for debian packages in buster distribution. But for python packages, we use version-py2 and version-py3 by default, it is not relative to distribution. - - When migrating from stretch to buster, we may need to add additional version configuration for debian packages, but not require to add version files for python packages. See Q&A #2 for adding additional version configuration and #4 for version conflict. - +# SONiC Reproduceable Build + +# Table of Contents + +- [Revision History](#revision-history) +- [Overview](#overview) +- [Pypi packages](#pypi-packages) + * [Pypi Version Configuration](#pypi-version-configuration) + * [Pypi Version Control and Check](#pypi-version-control-and-check) + * [Pypi Version Upgrade Automation](#pypi-version-upgrade-automation) + * [Work Items](#work-items) +- [Debian Packages](#debian-packages) + * [Debian Version Configuration](#debian-version-configuration) + * [Debian Packages Version Control and Check](#debian-packages-version-control-and-check) + * [Work Items](#work-items-1) +- [Debootstrap](#debootstrap) +- [Packages downloaded by wget/curl](#packages-downloaded-by-wget-curl) + * [Web Package Version Configuration](#web-package-version-configuration) + * [File Storage](#file-storage) + * [The process to use a new web package](#the-process-to-use-a-new-web-package) + * [Work Items](#work-items-2) +- [Debian Package Mirror](#debian-package-mirror) + * [Debian Package Mirror managed by Aptly](#debian-package-mirror-managed-by-aptly) +- [Pypi Package Mirror](#pypi-package-mirror) +- [Docker Base Images](#docker-base-images) +- [Git Repository](#git-repository) + +# Revision History +|Date|Description| +| :------- | :------- | +| 2020/08/21 | Xuhui Miao, initial version | +| 2020/09/21 | Xuhui, added Debian/pypi repo part | +| 2020/09/22 | Qi Luo, review | +| 2020/10/10 | Xuhui, debootstrap supports reproduceable build | +| 2020/11/01 | Xuhui, add Golang, component level build description | +| 2020/11/25 | Xuhui, add Q&A, and simplify web package process | +| 2022/02/23 | Xuhui, update scopes | + + +# Overview + +There are many SONiC failure builds caused by the external dependencies changed. It can be built in the first time and might be failed in the next build, although there are no code changes. SONiC reproducible build is to make the build stable and reduce the failure rate, build on the same git commit, expect the same result. + +The external dependencies contain pypi packages installed by pip/pip3, Debian packages installed by apt-get, wget/curl, debootstrap, docker base images, source code by git. + +Some of the dependent packages can be controlled by its version for the reproduceable build, such as pypi packages, Debian package. There is an assumption that the content of the packages should be the same for the packages with the same version. The other packages do not have a specified version, when the remote package changed, it cannot be detected by its URL, such as wget/curl. The file storage will be introduced to solve the package without version issue. + +When the reproduceable build is used, to catch up with the latest version of the dependent packages, sending pull request to update the versions is required, the automation Jenkins pipeline will be created to do it. + +The SONiC reproduceable build framework provides a way to lock the versions of the packages in scopes as above. It will lock the versions for all the docker images including the slave docker images, host images, and docker slave containers which are used to build the SONiC targets, such as deb files, python wheels, etc. + +It supports to control the version of the docker slave image, and also supports to control the slave container to build the targets including docker images, host images, debian packages, etc. + +In Scopes: + +1. Pypi packages installed by pip +2. Debian packages installed by apt-get +3. Host image created by debootstrap +4. Web packages downloaded by wget/curl +5. Debian package mirror +6. Docker base images +7. Source code by git, full commit id +8. Arch: amd64, arm64, armhf +9. Branches: 202012/202106/202111, after 202012 + +Next scopes: + +1. Support on the old branches like 201911 + +Out of Scopes: + +1. Golang reproduceable build, suggest to use go.mod to support reproduceable build, see https://github.com/golang/go/wiki/Modules#faqs--gomod-and-gosum. +2. Pypi package mirror (Multiple versions supported) + +# Pypi packages + +## Pypi Version Configuration + +The default version configuration files in "files/build/versions/" for Pypi packages. + +| File Name | Description | +| --- | --- | +| versions-py2 | The default versions for python2 | +| versions-py3 | The default versions for python3 | + +The content of the version file versions-py2 example: + +Py==1.7.0 + +pyang==2.1.1 + +pyangbind==0.6.0 + +pyasn1==0.4.2 + +The versions can be overridden by the version configuration file with the same file name in the Dockerfile folder, for example, the content of the version file dockers/docker-base-buster/versions-py2: + +**pyang==2.3.2** + +When building the docker-base-buster, the expected version constraints are as below: + +Py==1.7.0 + +**pyang==2.3.2** + +pyangbind==0.6.0 + +pyasn1==0.4.2 + +The current versions of the images will be dumped to the folder target/versions during the build. The initial/upgraded version configuration file can be generated based on it. + +## Pypi Version Control and Check + +In the Makefile or Dockerfile, if the version is specified for a Pypi package, it should be the same as the version in the constraints file. The build will be failed with version conflict. + +| Version in config file | Version in Makefile/Dockerfile | Check Result | +| --- | --- | --- | +| Yes | No | Success | +| No | Yes | Success | +| No | No | Failed for version not specified | +| Yes | Yes | Failed if different versions | + +The Pypi version configuration file will be used as a pip constraints file, some of the flags no used, like upgrade=manual, will be removed when used by pip command. see [pip constraints-files user guide](https://pip.pypa.io/en/stable/user_guide/#constraints-files). It will control the installing packages and the referenced packages. + +The pip/pip3 command in Makefile and Dockerfile will be hooked with a new one, if version conflict, the build will break. It only installs the version of the package specified in the configuration file by adding the constraints option. + +## Pypi Version Upgrade Automation + +Developers can change the Pypi package version manually, and the Jenkins job will be provided to upgrade the Pypi package version automatically. It will upgrade all the Pypi packages listed in the Pypi version files, except the packages with a flag "upgrade=manual" in the configuration file. + +The data flow diagram of version upgrade is as below: + +![](images/pypi-version-upgrade-automation.png) + +## Work Items + +1. Generate the initial version files for pip/pip3. +2. Add the new pip/pip3 command, verify the version and use the version constraints. +3. Enable the new commands in Dockerfile files and Makefile files. +4. Implement the Pypi version upgrade automation Jenkins pipeline. + +# Debian Packages + +## Debian Version Configuration + +The default version source files in files/build/versions/ for Debian packages. + +| File Name | Description | +| --- | --- | +| versions-deb-jessie | The versions for jessie | +| versions-deb-stretch | The versions for stretch | +| versions-deb-buster | The versions for buster | + +Similar to Pypi packages, the package version can be overridden per docker image. The different part is that the versions of the debian packages are relative to the debian release, in each release, such as the stretch and buster, the package versions are not the same, so we define a version configuration file for each debian release. For a specified docker image, such as docker-base-buster, etc, we can use dockers/docker-base-buster/versions-deb to override the version configuration file. + +## Debian Packages Version Control and Check + +Like Pypi Version control and check, if the build version is conflict with the version configuration, the build will be failed. The apt-get will be replaced with a bash script to check the version conflict. + +The version control is based on the [APT Preferences](https://manpages.debian.org/buster/apt/apt_preferences.5.en.html). The high priority of the APT preference can force to install a specified version of a package, even downgrade the package (priority >= 1000). + +Example setting for sudo: + +_Package: tzdata_ + +_Pin: version 2019c-0+deb10u1_ + +_Pin-Priority: 990_ + +When installing the package tzdata, if the docker base has installed 2020a-0+deb10u1, it will skip to install the lower version 1.8.21p2-3ubuntu1 of the package. If the version is not installed, then prefer to install 2019c-0+deb10u1. + +The version configuration file not only controls the packages installing directly by "apt-get install" command, but also controls the dependent packages. + +When installing the package ntp, the package is depending on the package tzdata, if the tzdata not install the version 2019c-0+deb10u1 will be installed, even there is a new version 2020a-c+deb10u1. + +Sample script to verify it: + +_# apt-cache madison tzdata_ + +_tzdata | 2020a-0+deb10u1 | http://deb.debian.org/debian buster/main amd64 Packages_ + +_tzdata | 2019c-0+deb10u1 | http://deb.debian.org/debian buster-updates/main amd64 Packages_ + +_# apt-get remove tzdata_ + +_# apt-get install ntp_ + +_# dpkg -l tzdata | grep tzdata_ + +_ii tzdata 2019c-0+deb10u1 all time zone and daylight-saving time data_ + +If you change the Pin-Priority to 100, then the latest version will be installed. + +## Work Items + +1. Generate the initial version files for Debian packages. + +2. Add the new apt-get command, verify the version and set the version preferences. + +3. Enable the new commands in Dockerfile files and Makefile files. + +4. Implement the Debian packages version upgrade automation Jenkins pipeline. + + + +# Debootstrap + +Currently, it will call debootstrap command in each build to create a new debian base host image, the image will automatically use the latest debian packages. It is not a reproduceable build. + +debootstrap --variant=minbase --arch amd64 buster rootfs + +There are 4 steps in the debootstrap command normally, finding debian packages, downloading the packages, the first stage, and the second stage. The debootstrap supports to only run the first two steps to generate a tarball in tgz format, and supports to use the tarball to generate the file system without downloading the packages again. + +debootstrap --variant=minbase --arch amd64 --unpack-tarball=tarball.tgz buster rootfs + +The --unpack-tarball will only do the last two steps, without finding/downloading the debian packages in buster, See [debootstrap git commit](https://salsa.debian.org/installer-team/debootstrap/-/commit/25d80b10319ed292827d016bfea6edcdb51b9b52) + +The tarball can be generated by the following command: + +debootstrap --variant=minbase --arch amd64 --make-tarball=tarball.tgz buster rootfs + +We can generate the tarball and replace the all packages with the right versions, or generate the tarball ourselves to support the reproduceable build. + +The tarball format is as below: + +| File | Description | +| :---------------------- | :--------------------------------------------------- | +| /debootstrap/base | The base debian packages | +| /debootstrap/required | The required debian packages | +| /debootstrap/debpaths | The debian package name to package real path mapping | +| /var/cache/apt/archives | Contains the debian packages in the folder | + +The steps to do the reproduceable build for root filesystem. + +1. Build the normal root filesystem + + debootstrap --variant=minbase --arch amd64 buster rootfs + +2. Freeze the versions, the version configuration format, see [here](#Pypi-Version-Configuration) + +3. Download the debian packages based on the frozen versions, and make the debootstrap tarball + +4. build the root filesystem using the tarball with frozen debian package versions. + +Another approach for debian base image reproduceable build is by [APT-TOOL](https://github.com/pauldotknopf/apt-tool). It supports to freeze the versions, and use the versions in the subsequent builds. + + +# Packages downloaded by wget/curl + +The web site may be not stable, and the web packages may be replaced to another one unexpectedly, so we do not want to depend on the web packages at a remote web site. Although there is no version for a package downloaded by wget/curl, the hash value of the package file can be used as the version value. Like Pypi packages and the Debian packages, there is a configuration file for wget/curl. For the package with the same URL, it is expected the same binary will be retrieved when using wget/curl. + +Requirements: + +1. Use the same web package, even the remote web package changed. +2. Improve reliability, the build does not be impacted when the remote web site is not available. +3. Keep update, automatically send a version change Pull Request to use the latest web package. [Low Priority] + +## Web Package Version Configuration + +The version file in files/build/versions/ for web packages. + +| File Name | Description | +| --- | --- | +| versions-web | The versions for packages downloaded from web | + +Like pip or debian package version configuration, the version format for web package is as below: +``` + **==** ; +``` +The hash value is the package's md5 message digest. + +An example as below: +``` +https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz== ebef065e4d35572af5b03e2be957a9c6c5063b38 +``` +If any new remote web package is used, it should be added in the version configuration file, or the build will be failed, so as to the existing web packages. + +## File Storage + +Before a web package is used in the SONiC repository, the package should be uploaded to a trusted file storage. The SONiC build only depends on the trusted file storage, not depend on the remote web site. + +The file name format in the file storage is as below: +``` +- +``` +For an example: go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38 + +For the same web package, it supports to have multiple versions of the file in the file storage, for example, go1.14.2.linux-amd64.tar.gz-b73d6366c0bce441d20138c150e2ddcdf406e654 (fake hash value), but for the same commit, the same version is used. + +There are multiple solutions for the file storage, such as FTP server, NFS server, Http Server, we do not focus on how to implement the file storage. In this document, we use Azure Storage Blob Container as the file storage. When you wget the web package [https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz](https://storage.googleapis.com/golang/go1.14.2.linux-amd64.tar.gz), it will actually download package from the URL like [https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38](https://sonicstorage.blob.core.windows.net/public/go1.14.2.linux-amd64.tar.gz-ebef065e4d35572af5b03e2be957a9c6c5063b38) during the build, in this case, the storage account URL is [https://sonicstorage.blob.core.windows.net](https://sonicstorage.blob.core.windows.net/), and the container name is public. Some of the packages are already in the same storage account, it is not necessary to add to the version in the configuration file. + +## The process to use a new web package + +Option A: Manually upload package (only for file storage admin) + +1. Manually upload the web package to the file storage. +2. Change the web package version configuration file to register the package, and use the wget/curl to download the package in your build script. + +Option B: Automatically upload package (for all) + +![](images/upload-package-automation.png) + +1. Change the web package version configuration file to register the package. + +Developer sends a Pull Request to register the web package, the web package URL and the package hash value (sha1sum) should be provided. The PR should be only one-line change in files/build/versions/versions-web, see [Web Package Version Configuration](#_Web_Package_Version). + +1. Upload the web package to the file storage. + +When the Pull Request is complete, and the commit is merged to the master branch, a background task is triggered to download the package, check the hash value, and upload to the file storage. + +1. Use the web package in your build script. + +Expect the step 2 is complete automatically. When the step 2 is complete, the web package is ready to use. + +## Work Items + +1. Generate the initial version files for web packages. +2. Add the new wget/curl command, verify the hash value. +3. Enable the new commands in Dockerfile files and Makefile files. +4. Implement the web packages uploading to file storage automation Jenkins pipeline. + +# Debian Package Mirror + +Debian official mirror only supports a single version of each binary package in any given release by design. For reproduceable build, it is required to manage previous versions of Debian packages. The designed Debian package mirror is to guarantee that all the versions of Debian packages used by sonic-imagebuild exist in the mirror. + +The current design of the Debian package mirror is managed by [Aptly](https://www.aptly.info/). The [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website) is used to publish the mirror. The [Azure Content Delivery Network (CDN)](https://docs.microsoft.com/en-us/azure/cdn/cdn-overview) integrated with Azure Storage Account is enabled to cache content from Azure Storage. To improve availability and performance, [Azure Traffic Manager](https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-overview) that supports failover across multiple Azure CDN endpoints, is exposed to users. + +![](images/azure-debian-mirror.png) + + + +## Debian Package Mirror managed by Aptly + +Steps to publish a Debian package mirror by Aptly: + +1. Create a mirror pool and update the pool, example commands as below: + + _aptly mirror create debian-buster-main http://deb.debian.org/debian buster main_ + + _aptly mirror update debian-buster-main_ + +2. Create a repository and import the packages from mirror pool, example commands as below: + + _aptly repo create repo-debian-buster-main_ + + _aptly repo import debian-buster-main repo-debian-buster-main_ + +3. Publish a repository + + _aptly publish repo -distribution=buster repo-debian-buster_ + +Steps to update a Debian package mirror by Aptly: + +1. Update the mirror pool + + _aptly mirror update debian-buster-main_ + +2. Import the repository + + _aptly repo import debian-buster-main repo-debian-buster-main_ + +3. Update the publish + + _aptly publish update buster_ + +To Publish the mirror to Azure Storage, [Azure Blob Fuse](https://github.com/Azure/azure-storage-fuse) supports to mount the Azure Blob Container to the filesystem, it makes easy to publish the Debian package mirror. The Azure Storage Account, storage container name and the Storage Account Assess Key or Storage SAS Token should be provided. The publish settings of aptly are as below: + +_"FileSystemPublishEndpoints": {_ + +_"web": {_ + +_"rootDir": " __/data/aptly/debian/web__",_ + +_"linkMethod": "copy",_ + +_"verifyMethod":"md5"_ + +_},_ + +… + +_}_ + +The rootDir above can be a blob fuse mount point. It will publish to Azure Storage directly. + +It is not guaranteed that all the Debian packages are in the mirror, if the size of the mirror glows too fast, we will remove some of the old versions of the packages which are not used by SONiC. + +# Pypi Package Mirror + +The Pypi official mirror supports multiple versions of each binary package. It is not required to have an additional mirror for the reproduceable build. Not like Debian package mirror, the designed Pypi mirror only publishes the packages used by SONiC image build, the other packages will not be published. Because the size of the full Pypi package mirror is up to 6 trillion bytes now, it is really hard to manage. + +The design of the Pypi package mirror is only for monitoring and alerting. When a specified version of the package is used in SONiC, we can expect the package should never be changed. If it was changed (the same package name and version, but different binary), we will be notified. + +If a package used by SONiC is removed from the Pypi official mirror accidently, the SONiC build will be failed. One of the solution is to add the extra URLs for pip command, see [here](https://pip.pypa.io/en/stable/reference/pip_install/#install-extra-index-url). But the risk should be very low, we keep to update the version to the latest one by design. It should not be the high priority task to do it. + +To publish only the packages used by SONiC, the used packages and versions should be collected by SONiC build. The input file should be like this as below: + +_Automat==0.6.0_ + +_Babel==2.3.4_ + +_Babel==2.6.0_ + +_Flask==1.1.2_ + +_Jinja2==2.10_ + +_Jinja2==2.11.2_ + +_Jinja2==2.7.3_ + +For one package, it may contain several versions to publish. The input file schema is the same as the Pypi version control file. + +The [python-pypi-mirror](https://pypi.org/project/python-pypi-mirror/) is used to publish the Pypi mirror to [Azure Storage Static Website](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website), likes Debian package mirror, it supports CDN and traffic manager. + +# Docker Base Images + +In the original design, the base images of the SONiC Dockerfile are based on image tag, for instance, in the Dockerfile sonic-slave-buster/Dockerfile, the base image is debian:buster, we can find "From debian:buster" in the file. When a new debian buster image is pushed in the docker registry, the debian:buster will refer to the new one, it is not reproduceable. To make it support reproduceable, we can change to refer to the digest hash value of the image. The docker base images that are based on the image tag will be changed to base on the image digest to support reproduceable build. The Dockerfile as below is based on the image tag: + +_From \:\_ + +Example: + +_From debian:buster_ + +Change to: + +_From \:\_ + +Example: + +_From debian@sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ + +The docker file will be changed to use the right digest during the build based on a version control file in file/build/versions/versions-docker-base, the content of the file is as below: + +debian:buster==_sha256:439a6bae1ef351ba9308fc9a5e69ff7754c14516f6be8ca26975fb564cb7fb76_ + +debian:stretch==_sha256:335ecf9e8d9b2206c2e9e7f8b09547faa9f868e694f7c5be14c38be15ea8a7cf_ + +debian:jessie==_sha256:8fc7649643ca1acd3940706613ea7b170762cfce6e7955a6afb387aa40e9f9ea_ + +# Git Repository + +The SONiC build will download source code from remote repository during the build. For instance, when building sflow, it will download source code by command: git clone [https://github.com/sflow/sflowtool](https://github.com/sflow/sflowtool). It will download the latest source code, not reproduceable. + +A version control file is introduced to control which commit of the git repository will be download when building SONiC image. The schema of the version control file is same as wget/curl. The content of the control file likes as below: + +[https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121](https://github.com/sflow/sflowtool==1e42bc69fab8a8bc58b04a0ff9ad2c366eef6121) + +It will run "git reset --hard" after the "git clone" command. + +We do not manage the git repositories, if a git commit used by SONiC is removed, then the build will be broken. + +# Q&A + +1. Is the reproduceable build feature will be enabled in all branches? + + It will be only enabled on the release branches, master branch will not be enabled in the short term. + +2. How a developer adds version configuration for new packages in a docker image? + + The developer should build the docker image or any other targets, the referred packages will be collected into target folder automatically during the build, the developer can run a command "make freeze" to freeze the version configuration without manual effort to manage it, then checks in the version changes by "git add files/build/versions" together with the code change. + +3. How a developer upgrades the packages to latest versions? + + The developer can upgrade the packages for one target or multiple targets, and can upgrades only one component or multiple components. There are two steps to do it, the first step is to make the targets with version version control option "make SONIC_VERSION_CONTROL_COMPONENTS=none \", the second step is to freeze the version by command "make freeze" the same as mentioned above. See more options in "rule/config". + +4. How a developer merges code across branches? How to resolve the version conflict? + + The process to merge version files is similar to merge code, we need to resolve the version conflict. To use the higher version or lower version depended on the usage scenarios, we would like to use to latest version better than the old one, but for some release branches, keep older version may be better. + +5. How to migrate from one debian distribution to anther one? + + In the version configuration files, different version files are used in different distributions if the version files are relative the distribution. For example, the version file name versions-deb-buster is used for debian packages in buster distribution. But for python packages, we use version-py2 and version-py3 by default, it is not relative to distribution. + + When migrating from stretch to buster, we may need to add additional version configuration for debian packages, but not require to add version files for python packages. See Q&A #2 for adding additional version configuration and #4 for version conflict. + +