[build]: Parallel building of sonic dockers using native dockerd(dood) #6086

Kalimuthu-Velappan · 2020-12-02T07:39:23Z

Currently, the build dockers are created as a user dockers(docker-base-stretch-, etc) that are
specific to each user. But the sonic dockers (docker-database, docker-swss, etc) are
created with a fixed docker name and common to all the users.

docker-database:latest
docker-swss:latest

When multiple builds are triggered on the same build server that creates parallel building issue because
all the build jobs are trying to create the same docker with latest tag.
This happens only when sonic dockers are built using native host dockerd for sonic docker image creation.

This patch creates all sonic dockers as user sonic dockers and then, while
saving and loading the user sonic dockers, it rename the user sonic
dockers into correct sonic dockers with tag as latest.

The user sonic docker names are derived from '_LOAD_DOCKER' make
variable and using Jinja template, it replaces the FROM docker name with correct user sonic docker name for
loading and saving the docker image.

The template processing covers only for common dockers, Broadcom and VS platform dockers. For other vendor specific dockers, respective vendors need to add the support.

- Why I did it
To enable the parallel build using native dockerd for sonic dockers.

- How I did it
Created user dockers for each of the sonic dockers.
docker-swss:latest <=SAVE/LOAD=> docker-swss-<user>:latest

- How to verify it
Parallel build on the same build server

- Which release branch to backport (provide reason below if selected)

201811
201911
202006

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

dockers/docker-config-engine-stretch/Dockerfile.j2

liushilongbuaa · 2020-12-10T06:39:13Z

retest this please

Kalimuthu-Velappan · 2020-12-16T10:42:10Z

retest this please

xumia · 2020-12-18T13:57:27Z

retest vs please

ben-gale · 2020-12-22T21:34:57Z

@xumia - what's the next step? Can we advance towards merge pls?

ben-gale · 2020-12-31T17:12:37Z

@lguohan, @xumia - how do we proceed to approve/merge? Is anything else needed from Broadcom?

Thx.

ben-gale · 2021-01-04T23:04:23Z

@lguohan, @xumia

@lguohan, @xumia - please advise - let's get this merged pls.

Makefile.work

slave.mk

rules/config

Makefile.work

lguohan · 2021-01-06T15:13:25Z

@ben-gale , i have posted my comments, we have some questions.

lguohan · 2021-01-06T15:15:43Z

@xumia , does this have any interference with reproducible build as they rename the docker from name. not sure if you have some hardcoded rule to tack this.

ben-gale · 2021-01-06T20:09:28Z

@ben-gale , i have posted my comments, we have some questions.

Thx

rules/functions

Makefile.work

Currently, the build dockers are created as a user dockers(docker-base-stretch-<user>, etc) that are specific to each user. But the sonic dockers (docker-database, docker-swss, etc) are created with a fixed docker name and common to all the users. docker-database:latest docker-swss:latest When multiple builds are triggered on the same build server that creates parallel building issue because all the build jobs are trying to create the same docker with latest tag. This happens only when sonic dockers are built using native host dockerd for sonic docker image creation. This patch creates all sonic dockers as user sonic dockers and then, while saving and loading the user sonic dockers, it rename the user sonic dockers into correct sonic dockers with tag as latest. The user sonic docker names are derived from '_LOAD_DOCKER' make variable and using Jinja template, it replaces the FROM docker name with correct user sonic docker name for loading and saving the docker image.

ben-gale · 2021-01-20T21:29:29Z

retest vsimage please

lguohan · 2021-01-26T09:18:59Z

i tried this pr and armhf build, but it failed.

https://dev.azure.com/sonicswitch/be1b070f-be15-4154-aade-b1d3bfb17054/_apis/build/builds/1873/logs/13

lguohan · 2021-01-26T09:21:22Z

if I am doing only the armhf build without this pr, then it works. https://dev.azure.com/sonicswitch/build/_build/results?buildId=1817&view=logs&j=83516c17-6666-5250-abde-63983ce72a49

i am not sure what went wrong here.

lguohan · 2021-01-26T09:29:56Z

dockers/docker-config-engine-buster/Dockerfile.j2

@@ -1,5 +1,5 @@
 {% from "dockers/dockerfile-macros.j2" import install_debian_packages, install_python_wheels, copy_files %}
-FROM docker-base-buster
+FROM {{ docker_config_engine_buster_load_image }}


i am not able to figure out what went wrong for this pr for armhf build, but can you split this pr into following pr so that the problem can be isolated.

use the load_image from the .mk file.

everything else.

lguohan · 2021-01-26T09:30:59Z

slave.mk

@@ -278,6 +288,69 @@ endif
 export kernel_procure_method=$(KERNEL_PROCURE_METHOD)
 export vs_build_prepare_mem=$(VS_PREPARE_MEM)

+###############################################################################
+## Canned sequences


maybe we should move this to Makefile.docker? all instructions related to docker should go there.

I will make the change as you suggested and will get back to you.

Kalimuthu-Velappan · 2021-02-24T15:44:58Z

It is conflicting with the reproducible build PR and requires some rework. We will fix and submit the changes in a separate PR and will close this PR.

Kalimuthu-Velappan force-pushed the DOOD_SHARED branch 4 times, most recently from 34d1680 to 2d8c2ec Compare December 3, 2020 18:47

lguohan added the brcm-upstream label Dec 8, 2020

lguohan requested a review from xumia December 9, 2020 01:23

xumia previously approved these changes Dec 9, 2020

View reviewed changes

liushilongbuaa reviewed Dec 10, 2020

View reviewed changes

dockers/docker-config-engine-stretch/Dockerfile.j2 Show resolved Hide resolved

Kalimuthu-Velappan dismissed xumia’s stale review via 83d09b1 December 16, 2020 10:41

Kalimuthu-Velappan force-pushed the DOOD_SHARED branch from 2d8c2ec to 83d09b1 Compare December 16, 2020 10:41

Kalimuthu-Velappan force-pushed the DOOD_SHARED branch from 83d09b1 to 521c411 Compare December 18, 2020 11:43

Kalimuthu-Velappan force-pushed the DOOD_SHARED branch from 521c411 to 12c5b6a Compare December 23, 2020 10:07

lguohan force-pushed the master branch from 3690c1a to 512eb6b Compare December 25, 2020 18:34