Support for DPKG local caching #4117

Kalimuthu-Velappan · 2020-02-06T10:29:18Z

DPKG caching framework provides the infrastructure to cache the sonic module/target .deb files into a local cache by tracking the target dependency files.SONIC build infrastructure is designed as a plugin framework where any new source code can be easily integrated into sonic as a module and that generates output as a .deb file. The source code compilation of a module is completely independent of other modules compilation. Inter module dependency is resolved through build artifacts like header files, libraries, and binaries in the form of Debian packages. For example module A depends on module B. While module A is being built, it uses B's .deb file to install it in the build docker.

The DPKG caching framework provides an infrastructure that caches a module's deb package and restores it back to the build directory if its dependency files are not modified. When a module is compiled for the first time, the generated deb package is stored at the DPKG cache location. On the subsequent build, first, it checks the module dependency file modification. If none of the dependent files is changed, it copies the deb package from the cache location, otherwise, it goes for local compilation and generates the deb package. The modified files should be checked-in to get the newer cache deb package.

This provides a huge improvement in build time and also supports the true incremental build by tracking the dependency files.

- How I did it
It takes two global arguments to enable the DPKG caching, the first one indicates the caching method and the second one describes the location of the cache.
SONIC_DPKG_CACHE_METHOD=cache
SONIC_DPKG_CACHE_SOURCE=

    where  SONIC_DPKG_CACHE_METHOD - Default method is 'cache' for deb package caching
                            none:     no caching
                            cache:    cache from local directory

Dependency file tracking:
Dependency files are tracked for each target in two levels.
1. Common make infrastructure files - rules/config, rules/functions, slave.mk etc.
2. Per module files - files which are specific to modules, Makefile, debian/rules, patch files, etc.

    For example: dependency files for Linux Kernel - src/sonic-linux-kernel,

            SPATH       := $($(LINUX_HEADERS_COMMON)_SRC_PATH)
            DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/linux-kernel.mk rules/linux-kernel.dep
            DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
            SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files))

            DEP_FLAGS := $(SONIC_COMMON_FLAGS_LIST) \
                         $(KERNEL_PROCURE_METHOD) $(KERNEL_CACHE_PATH)

            $(LINUX_HEADERS_COMMON)_CACHE_MODE  := GIT_CONTENT_SHA
            $(LINUX_HEADERS_COMMON)_DEP_FLAGS   := $(DEP_FLAGS)
            $(LINUX_HEADERS_COMMON)_DEP_FILES   := $(DEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_FILES := $(SMDEP_FILES)
            $(LINUX_HEADERS_COMMON)_SMDEP_PATHS := $(SPATH)

Cache file tracking:
The Cache file is a compressed TAR ball of a module's target DEB file and its derived-target DEB files.
The cache filename is formed with the following format

    FORMAT:
            <module deb filename>.<24 byte of DEP SHA hash >-<24 byte of MOD SHA hash>.tgz
            Eg:
              linux-headers-4.9.0-9-2-common_4.9.168-1+deb9u3_all.deb-23658712fd21bb776fa16f47-c0b63ef593d4a32643bca228.tgz

            < 24-byte DEP SHA value > - the SHA value is derived from all the dependent packages.
            < 24-byte MOD SHA value > - the SHA value is derived from either of the following.
                    GIT_COMMIT_SHA  - SHA value of the last git commit ID if it is a submodule
                    GIT_CONTENT_SHA - SHA value is generated from the content of the target dependency files.

Target Specific rules:
Caching can be enabled/disabled on a global level and also on the per-target level.

            $(addprefix $(DEBS_PATH)/, $(SONIC_DPKG_DEBS)) : $(DEBS_PATH)/% : .platform $$(addsuffix -install,$$(addprefix $(DEBS_PATH)/,$$($$*_DEPENDS))) \
                    $(call dpkg_depend,$(DEBS_PATH)/%.dep )
            $(HEADER)


            # Load the target deb from DPKG cache
            $(call LOAD_CACHE,$*,$@)


            # Skip building the target if it is already loaded from cache
            if [ -z '$($*_CACHE_LOADED)' ] ; then

                  .....
                 # Rules for Generating the target DEB file.
                  .....

                  # Save the target deb into DPKG cache
                  $(call SAVE_CACHE,$*,$@)
            fi


            $(FOOTER)


    The make rule-'$(call dpkg_depend,$(DEBS_PATH)/%.dep )' checks for target dependency file modification. If it is newer than the target, it will go for re-generation of that target.

    Two main macros 'LOAD_CACHE' and 'SAVE_CACHE' are used for loading and storing the cache contents.
    The 'LOAD_CACHE' macro is used to load the cache file from cache storage and extracts them into the target folder. It is done only if target dependency files are not modified by checking the GIT file status, otherwise, cache loading is skipped and full compilation is performed.
    It also updates the target-specific variable to indicate the cache is loaded or not.
    The 'SAVE_CACHE' macro generates the compressed tarball of the cache file and saves them into cache storage. Saving into the cache storage is protected with a lock.

- How to verify it

    The caching functionality is verified by enabling it in Linux kernel submodule.
    It uses the cache directory as 'target/cache' where Linux cache file gets stored on the first-time build and it is picked from the cache location during the subsequent clean build.

- Description for the changelog
The DPKG caching framework provides the infrastructure to save the module-specific deb file to be cached by tracking the module's dependency files.
If the module's dependency files are not changed, it restores the module deb files from the cache storage.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

DOCUMENT PR:

           https://github.com/Azure/SONiC/pull/559

lguohan · 2020-02-08T04:10:10Z

@Kalimuthu-Velappan , can you check following messages?

[ DPKG ] Cache is not enabled for iptables_1.6.0+snapshot20161117-6_amd64.deb package
[ DPKG ] Cache is not enabled for sonic_version.yml package
[ DPKG ] Cache is not enabled for docker-nat.gz package
[ DPKG ] Cache is not enabled for docker-nat-dbg.gz package

lguohan · 2020-02-08T04:11:46Z

for a fresh build, it looks like /tmp/dpkg_cache is created with root privilidge only. can you fix this by explicitly create /tmp/dpkg_cache with 777 permission?

touch: cannot touch '/dpkg_cache/libnl-3-200_3.5.0-1_amd64.deb_cache_accss.lock': Permission denied
slave.mk:355: recipe for target 'target/debs/stretch/libnl-3-200_3.5.0-1_amd64.deb' failed
make: *** [target/debs/stretch/libnl-3-200_3.5.0-1_amd64.deb] Error 1
Makefile.work:214: recipe for target 'stretch' failed
make[1]: *** [stretch] Error 2
make[1]: Leaving directory '/data2/sonic/buster/sonic-buildimage'
Makefile:7: recipe for target 'target/sonic-vs.img.gz' failed
make: *** [target/sonic-vs.img.gz] Error 2

lguohan · 2020-02-08T04:32:25Z

rules/sonic-ztp.dep

+SPATH       := $($(SONIC_ZTP)_SRC_PATH)
+DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/sonic-ztp.mk rules/sonic-ztp.dep   
+DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
+SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files|grep -Ev "inband-ztp-ip|dhclient-exit-hooks.d/ztp"))


why do we need grep here?

Fixed and uploaded the patchset.

i can still see the grep, can you double check?

lguohan · 2020-02-08T04:35:00Z

rules/sonic-platform-common.dep

+SPATH       := $($(SONIC_PLATFORM_COMMON_PY2)_SRC_PATH)
+DEP_FILES   := $(SONIC_COMMON_FILES_LIST) rules/sonic-platform-common.mk rules/sonic-platform-common.dep   
+DEP_FILES   += $(SONIC_COMMON_BASE_FILES_LIST)
+SMDEP_FILES := $(addprefix $(SPATH)/,$(shell cd $(SPATH) && git ls-files|grep -Ev "sonic_sfp|sonic_eeprom"))


why is this special? why need grep here?

sonic_sfp and sonic_eeprom are created as soft links. The git hash-object command is not working with soft links. Hence these files are excluded.

@jleveque , why do we have soft links? can we remove them?

lguohan · 2020-02-08T04:36:24Z

rules/sairedis.dep

+
+SMDEP_PATHS := $(SPATH) $(SPATH)/SAI $(SPATH)/SAI/bm/behavioral-model $(SPATH)/SAI/test/ptf $(SPATH)/SAI/test/saithrift/ctypesgen
+$(foreach path, $(SMDEP_PATHS), $(eval $(path) :=$(filter-out $(SMDEP_PATHS),$(addprefix $(path)/, \
+					$(shell cd $(path)  && git ls-files | grep -v " ")))))


why do we have such complexity here? can you help to explain?

I do not really understand SMDEP has these $(SPATH)/SAI/bm/behavioral-model $(SPATH)/SAI/test/ptf $(SPATH)/SAI/test/saithrift/ctypesgen

and I see this error

fatal: Not a git repository: /data2/sonic/stretch/sonic-buildimage/.git/modules/sonic-sairedis/modules/SAI/modules/bm/behavioral-model

Fixed and uploaded the patchset.

This package has multiple Sub repo. The git hash-object command is not working for the repo directory name. Hence we need to exclude the directory name. We also need to exclude the files that has space on it.

Makefile.cache

platform/broadcom/rules.dep

lguohan · 2020-02-08T04:48:45Z

First of all, really appreciated the work here. i do not see dep module for other platforms like barefoot, marvell. do you expect them to come up with the dep moduels?

rules/sonic-ztp.dep

lguohan · 2020-02-09T02:58:20Z

for a fresh build, it looks like /tmp/dpkg_cache is created with root privilidge only. can you fix this by explicitly create /tmp/dpkg_cache with 777 permission?
touch: cannot touch '/dpkg_cache/libnl-3-200_3.5.0-1_amd64.deb_cache_accss.lock': Permission denied
slave.mk:355: recipe for target 'target/debs/stretch/libnl-3-200_3.5.0-1_amd64.deb' failed
make: *** [target/debs/stretch/libnl-3-200_3.5.0-1_amd64.deb] Error 1
Makefile.work:214: recipe for target 'stretch' failed
make[1]: *** [stretch] Error 2
make[1]: Leaving directory '/data2/sonic/buster/sonic-buildimage'
Makefile:7: recipe for target 'target/sonic-vs.img.gz' failed
make: *** [target/sonic-vs.img.gz] Error 2
The /tmp/dpkg_cache is being used just for code review. Please suggest a default folder for DPKG caching. This folder should be pre-created and set with right permission.

suggest /var/cache/sonic/artifacts/

lguohan · 2020-02-09T02:59:53Z

suggest not to do a force push, just do a incremental push, easier for the code review. we will do squash merge eventually so all your commits related to this pr will be merged as one commits.

slave.mk

Makefile.cache

slave.mk

lguohan · 2020-02-11T14:14:12Z

do you support a read-only cache mode? for example, we build the cache for all nightly master build, and then for PR build, it only read from the cache, but it does not write to the cache.

Kalimuthu-Velappan · 2020-02-11T14:21:30Z

Added support for readonly and writeonly cache support

…

On Tue, Feb 11, 2020 at 7:44 PM lguohan ***@***.***> wrote: do you support a read-only cache mode? for example, we build the cache for all nightly master build, and then for PR build, it only read from the cache, but it does not write to the cache. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4117>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2XPKKV2ESYHCNHGUKLRCKXDJANCNFSM4KQ2LMOQ> .

-- Thanks - Kals

Makefile.cache

platform/vs/docker-sonic-vs.dep

rules/functions

Makefile.cache

Kalimuthu-Velappan · 2020-02-13T13:20:47Z

Yes, we have tested on the NFS mount. I will check this issue.

…

On Thu, Feb 13, 2020 at 5:50 PM xumia ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In Makefile.cache <#4117 (comment)> : > +# +# Parameters: +# $(1) - target file name (without path) +# $(2) - lock file path (only) +# $(3) - designated lock file suffix +# $(4) - flock timeout (in seconds) +# +# $(call MOD_LOCK,file,path,suffix,timeout) +define MOD_LOCK + if [[ ! -f $(2)/$(1)_$(3).lock ]]; then + touch $(2)/$(1)_$(3).lock + chmod 777 $(2)/$(1)_$(3).lock; + fi + $(eval $(1)_lock_fd=$(subst ~,_,$(subst -,_,$(subst +,_,$(subst .,_,$(1)))))) + exec {$($(1)_lock_fd)}<"$(2)/$(1)_$(3).lock"; + if ! flock -x -w $(4) "$${$($(1)_lock_fd)}" ; then Does it support NFS? /mnt/nfs is the mounted nfs path. It has a "Bad file descriptor" issue, but the command "flock [options] | -c " works fine. $ flock -x /mnt/nfs/test_lock.lock -c "echo test" test $ exec {test}<"/mnt/nfs/test_lock.lock" $ flock -x -w 10 "${test}" *flock: 15: Bad file descriptor* $ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4117>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2VZIW7AYCP74YBYQ3LRCU3KPANCNFSM4KQ2LMOQ> .

-- Thanks - Kals

Makefile.cache

platform/vs/sonic-version.dep

Makefile.cache

platform/broadcom/docker-syncd-brcm-rpc.dep

platform/broadcom/libsaithrift-dev.dep

Makefile.cache

Kalimuthu-Velappan · 2020-03-03T17:30:32Z

gest /var/cache/sonic/artifacts/

Good one, I can add support for this along with review comments fix.
…
On Tue, Feb 11, 2020 at 7:44 PM lguohan @.***> wrote: do you support a read-only cache mode? for example, we build the cache for all nightly master build, and then for PR build, it only read from the cache, but it does not write to the cache. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4117?email_source=notifications&email_token=AM2UC2VPY37I4IPOJCCAYJTRCKXDJA5CNFSM4KQ2LMO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELMR5NI#issuecomment-584654517>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM2UC2XPKKV2ESYHCNHGUKLRCKXDJANCNFSM4KQ2LMOQ .
-- Thanks - Kals

Added support for readonly and writeonly cache

Makefile.cache

Makefile.work

qiluo-msft · 2020-04-13T22:00:40Z

platform/broadcom/rules.dep

+include $(PLATFORM_PATH)/raw-image.dep
+include $(PLATFORM_PATH)/one-aboot.dep
+include $(PLATFORM_PATH)/libsaithrift-dev.dep
+include $(PLATFORM_PATH)/docker-ptf-brcm.dep


~~Is $(SYNCD)_CACHE_MODE missing in this PR? I can only see $(SYNCD_VS)_CACHE_MODE~~

OK, I notice the file rules/sairedis.dep

Kalimuthu-Velappan marked this pull request as ready for review February 6, 2020 11:55

Kalimuthu-Velappan mentioned this pull request Feb 6, 2020

Support for DPKG local caching framework sonic-net/SONiC#559

Merged

Kalimuthu-Velappan requested a review from lguohan February 6, 2020 13:40

qiluo-msft self-requested a review February 6, 2020 23:14

lguohan reviewed Feb 8, 2020

View reviewed changes

Makefile.cache Outdated Show resolved Hide resolved

lguohan reviewed Feb 8, 2020

View reviewed changes

platform/broadcom/rules.dep Show resolved Hide resolved

lguohan reviewed Feb 8, 2020

View reviewed changes

rules/sonic-ztp.dep Show resolved Hide resolved

Kalimuthu-Velappan force-pushed the DPKG_CACHE branch from ea4172e to 7cabd23 Compare February 8, 2020 18:08

This comment has been minimized.

Sign in to view

lguohan reviewed Feb 9, 2020

View reviewed changes

slave.mk Show resolved Hide resolved

lguohan reviewed Feb 9, 2020

View reviewed changes

Makefile.cache Outdated Show resolved Hide resolved

qiluo-msft reviewed Feb 10, 2020

View reviewed changes

slave.mk Show resolved Hide resolved

xumia reviewed Feb 12, 2020

View reviewed changes