new: support driverkit multiarch driver builds #734

FedeDP · 2022-06-16T11:53:43Z

This PR adds support for driverkit configs $arch subfolders for newer driverversions, that we assume will be semversioned.

On top of that, it also adds support for driverkit 0.9.1 features like kernelurls and multiarch builds.

Biggest challenge is keeping backward compatibility with current driverkit directory tree.

Old directory tree:
config/$driverversion/*.yml
and expected url:
https://download.falco.org/?prefix=driver/$driverversion/*.{o,ko}
New one:
config/$driverversion(semver)/$arch/*.yml
and expected url:
https://download.falco.org/?prefix=driver/$driverversion/$arch/*.{o,ko}

Work is still in progress. Namely, backward compatibility must still be addressed, as new config files expect "/$arch/configs" paths, while the old had only x86_64 in the root. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

…rt to driverkit scrape_and_generate/generate utilities. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

…, utils/scrape_and_generate to be backward compatible. Moreover, fixed a couple of stupid bugs. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Note: even when filtering for TARGET_ARCH, `validate_old` and `specific_target_old` will always run. This is a small issue while we support both the old and the new format, and will eventually go away once we drop support for old format. It is not impactful since we do never filter for TARGET_ARCH (given that it is a newly introduced filter). Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

…upport the new $driverversion/$arch/ format. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2022-06-16T11:54:12Z

driverkit/Makefile

+# Recursive wildcard
+rwildcard=$(foreach d,$(wildcard $(1:=/*)),$(call rwildcard,$d,$2) $(filter $(subst *,%,$2),$d))
+
+CONFIGS := $(call rwildcard,config/*,*.yaml)


Find all configs under config and its subfolders (eventually, for supported $arch).

FedeDP · 2022-06-16T11:54:38Z

driverkit/Makefile

 S3_DRIVERS_BUCKET ?= "falco-distribution"
 S3_DRIVERS_KEY_PREFIX ?= "driver"
 SKIP_EXISTING ?= true

-validate: $(patsubst config_%,validate/%,$(subst /,_,$(wildcard config/${TARGET_VERSION}*/${TARGET_DISTRO}_${TARGET_KERNEL}-*)))
+validate: validate_old validate_new


Split in 2 sub targets to correctly manage new and old driverkit config tree.

I like it, even not elegant. When needed to cleanup it will be simple.

FedeDP · 2022-06-16T11:54:46Z

driverkit/Makefile


 all: $(patsubst config_%,%,$(subst /,_,$(CONFIGS)))

-specific_target: $(patsubst config_%,%,$(subst /,_,$(wildcard config/${TARGET_VERSION}*/${TARGET_DISTRO}_${TARGET_KERNEL}-*)))
+specific_target: specific_target_old specific_target_new


Split in 2 sub targets to correctly manage new and old driverkit config tree.

Same as above

FedeDP · 2022-06-16T11:55:15Z

driverkit/Makefile


 prepare: $(addprefix prepare_,$(VERSIONS))
 publish: $(addprefix publish_,$(VERSIONS))
 publish_s3: $(addprefix publish_s3_,$(VERSIONS))

 generate:
 	$(foreach VERSION,$(VERSIONS),\
-		utils/generate -k ${TARGET_KERNEL} -d ${TARGET_DISTRO} -v ${VERSION}; \
+		utils/generate -a $(TARGET_ARCH) -k ${TARGET_KERNEL} -d ${TARGET_DISTRO} -h ${TARGET_HEADERS} -v ${VERSION}; \


Generate target will now generate configs with proper architecture and kernelurls set.

See FedeDP#1 and review

FedeDP · 2022-06-16T11:55:37Z

driverkit/Makefile

@@ -40,36 +60,53 @@ cleanup_s3:
 	S3_DRIVERS_BUCKET=${S3_DRIVERS_BUCKET} S3_DRIVERS_KEY_PREFIX=${S3_DRIVERS_KEY_PREFIX} utils/cleanup_s3 $(addprefix -v ,$(VERSIONS))

 # $(1): pseudo-target name
-# $(2): driver version
-# $(3): config file path
+# $(2): config file path


Dropped unused driver version param.

FedeDP · 2022-06-16T11:56:11Z

driverkit/Makefile

 ifneq ("$(wildcard output/$(1)/*.ko)","")
 	@mkdir -p output/$(1)/kernel-module
 	@mv -f output/$(1)/*.ko output/$(1)/kernel-module
 endif
+ifneq ("$(wildcard output/$(1)/x86_64/*.ko)","")


Basically, we now search in both root folder (old directory tree) and new $arch subfolders.

FedeDP · 2022-06-16T11:57:02Z

driverkit/utils/build

@@ -9,6 +9,15 @@ input="$1"
 IFS=/ read -a input_parts <<< ${input}
 driver_version="${input_parts[1]}"

+# If there is an arch subdir -> use it!
+# TODO: rm once old format support is dropped; just use arch="${input_parts[2]}"
+if [[ ${#input_parts[@]} -eq 4 ]]; then


This workaround is needed to be able to check correct module path for old and new directory tree.

FedeDP · 2022-06-16T11:57:59Z

driverkit/utils/checkfiles

+    echo -n "${arch}"
+}
+
+arch="$(get_arch_from_path ${input})"


Fetch arch from config file path (if any); it is later used to check that if config file is under a $arch subfolder, its output module and probe must be under same $arch subfolder.

FedeDP · 2022-06-16T11:58:39Z

driverkit/utils/driverstats

-configs=(config/${TARGET_VERSION}*/${TARGET_DISTRO}_${TARGET_KERNEL}-*.yaml)
+# Only load old configs if x86_64 or * arch is requested
+# TODO: rm this once old format support is dropped
+if [ "${TARGET_ARCH}" = "*" ] || [ "${TARGET_ARCH}" = "x86_64" ]; then


If users filter for TARGET_ARCH, only load old config if either "*" (any) or "x86_64" is requested.

FedeDP · 2022-06-16T11:59:36Z

driverkit/utils/generate

+# in the TARGET_VERSION folder.
+# TODO: rm this once old format support is dropped, and always use ${TARGET_VERSION}/${TARGET_ARCH}
+function path_from_version() {
+  rx='^([0-9]+\.){0,2}(\*|[0-9]+)$'


We assume that new driverversions, that will use new directory tree, all have a semver.

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2022-06-16T12:06:14Z

/cc @maxgio92 @zuc @leogr

I think i will need multiple eyes on this one :)
Biggest problem is maintaining backward compatibility with current directory tree.

leogr · 2022-06-16T12:44:39Z

/cc @maxgio92 @zuc @leogr

I think i will need multiple eyes on this one :) Biggest problem is maintaining backward compatibility with current directory tree.

Hey @FedeDP

Could you describe the expected directory (and URL) layout in the PR description for a future reference? 🙏
It would be also helpful to make sure we are on the same page.

FedeDP · 2022-06-16T12:45:25Z

@leogr i will edit the OP!

driverkit/utils/generate

Signed-off-by: Massimiliano Giovagnoli <me@maxgio.it>

maxgio92 · 2022-06-22T18:07:33Z

/lgtm

poiana · 2022-06-22T18:07:38Z

LGTM label has been added.

Git tree hash: ee2bd9af134e184408ee12c06d8e2f0e15d134b3

maxgio92 · 2022-06-22T18:08:25Z

I think we'll then need a refactoring of the DBG considering arbitrary architecture, maybe after the removal of deprecated (configuration and distribution) paths.

FedeDP · 2022-06-23T07:09:42Z

I think we'll then need a refactoring of the DBG considering arbitrary architecture, maybe after the removal of deprecated (configuration and distribution) paths.

Yep; i think the work is in place indeed, we just need some improvements here and there (that will be much easier once we drop support for old tree paths)!

FedeDP · 2022-06-24T07:47:07Z

/hold
I'd wait until driverkit 0.9.2/0.10 is out, as it will bring fixes for qemu driver cross-compilation.

FedeDP · 2022-06-24T10:28:29Z

/unhold
Perhaps it is better to merge this and do a test build on a alpha driver git tag.
@leogr @maxgio92

maxgio92 · 2022-06-25T13:52:11Z

/cc @leogr @zuc
LGTM let's coordinate first :-)

leogr

Let's give it a try :)

poiana · 2022-06-27T12:03:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP, leogr, maxgio92

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [leogr,maxgio92]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

FedeDP added 8 commits June 14, 2022 11:03

update(config): updated driverkit to 0.9.0 and added kernelurls suppo…

ae7c20a

…rt to driverkit scrape_and_generate/generate utilities. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

update(driverkit): port utils/build, utils/checkfiles, utils/generate…

ac6663a

…, utils/scrape_and_generate to be backward compatible. Moreover, fixed a couple of stupid bugs. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

update(driverkit): port utils/validate to be backward compatible.

ddc7424

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

update(driverkit): port utils/driverstats to be backward compatible.

f11b7e4

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore(driverkit): added TODOs to remove useless checks once we only s…

3090270

…upport the new $driverversion/$arch/ format. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

update(images): updated driverkit to 0.9.1.

023f335

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

poiana added do-not-merge/work-in-progress dco-signoff: yes size/L labels Jun 16, 2022

poiana requested review from fntlnz and leodido June 16, 2022 11:54

FedeDP commented Jun 16, 2022

View reviewed changes

fix(driverkit): fixed semver check regex.

72bb7ce

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

poiana requested review from leogr, maxgio92 and zuc June 16, 2022 12:06

LucaGuerra mentioned this pull request Jun 17, 2022

Cut Falco 0.32.1 Release falcosecurity/falco#2070

Closed

15 tasks

FedeDP changed the title ~~wip: new: support driverkit multiarch driver builds~~ new: support driverkit multiarch driver builds Jun 20, 2022

poiana removed the do-not-merge/work-in-progress label Jun 20, 2022

maxgio92 reviewed Jun 21, 2022

View reviewed changes

driverkit/utils/generate Show resolved Hide resolved

maxgio92 reviewed Jun 21, 2022

View reviewed changes

driverkit/utils/generate Show resolved Hide resolved

fix(driverkit/makefile): pass right archs to generate targets

e87df75

Signed-off-by: Massimiliano Giovagnoli <me@maxgio.it>

FedeDP force-pushed the new/driverkit_multiarch branch from 00ebdaf to e87df75 Compare June 21, 2022 14:20

poiana assigned maxgio92 Jun 22, 2022

poiana added the lgtm label Jun 22, 2022

poiana added the approved label Jun 22, 2022

poiana added the do-not-merge/hold label Jun 24, 2022

poiana removed the do-not-merge/hold label Jun 24, 2022

leogr approved these changes Jun 27, 2022

View reviewed changes

poiana assigned leogr Jun 27, 2022

poiana merged commit 86b9375 into falcosecurity:master Jun 27, 2022

FedeDP deleted the new/driverkit_multiarch branch June 27, 2022 12:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new: support driverkit multiarch driver builds #734

new: support driverkit multiarch driver builds #734

FedeDP commented Jun 16, 2022 •

edited

Loading

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

maxgio92 Jun 21, 2022

FedeDP Jun 16, 2022

maxgio92 Jun 21, 2022

FedeDP Jun 16, 2022

maxgio92 Jun 21, 2022

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

FedeDP Jun 16, 2022

FedeDP commented Jun 16, 2022

leogr commented Jun 16, 2022

FedeDP commented Jun 16, 2022

maxgio92 commented Jun 22, 2022

poiana commented Jun 22, 2022

maxgio92 commented Jun 22, 2022

FedeDP commented Jun 23, 2022

FedeDP commented Jun 24, 2022

FedeDP commented Jun 24, 2022

maxgio92 commented Jun 25, 2022

leogr left a comment

poiana commented Jun 27, 2022

new: support driverkit multiarch driver builds #734

new: support driverkit multiarch driver builds #734

Conversation

FedeDP commented Jun 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FedeDP commented Jun 16, 2022

leogr commented Jun 16, 2022

FedeDP commented Jun 16, 2022

maxgio92 commented Jun 22, 2022

poiana commented Jun 22, 2022

maxgio92 commented Jun 22, 2022

FedeDP commented Jun 23, 2022

FedeDP commented Jun 24, 2022

FedeDP commented Jun 24, 2022

maxgio92 commented Jun 25, 2022

leogr left a comment

Choose a reason for hiding this comment

poiana commented Jun 27, 2022

FedeDP commented Jun 16, 2022 •

edited

Loading