Falco test cases are failing on ppc64le platform

[sunil1783]

What happened :
To support falco and sysdig with eBPF support on ppc64le platform, done the required code
changes. Following PR is created for these changes.
#932

After building the code ,when try to run the regression test cases, it gives following types of errors.
TestFail: Different counts for rule Change thread namespace: expected=2, actual=1
TestFail: Different counts for rule Launch Privileged Container: expected=3, actual=1
TestFail: Different counts for rule Launch Sensitive Mount Container: expected=3, actual=1
TestFail: Different counts for rule Read sensitive file untrusted: expected=1, actual=0
TestFail: Different counts for rule Write below binary dir: expected=4, actual=1
TestFail: Different counts for rule Change thread namespace: expected=2, actual=1
TestFail: Different counts for rule Open From Cat ($.*+?()[]{}|^): expected=8, actual=1
TestFail: Different counts for rule open_13: expected=1, actual=0
TestFail: Different counts for rule open_10: expected=1, actual=0
TestFail: Different counts for rule detect_open: expected=2, actual=0
TestFail: Could not find a line 'ERROR: ' in falco output
TestFail: Could not find a line 'WARNING: ' in falco output
TestFail: Stdout of falco process 'Warning: macro some macro not refered to by any rule/macro
TestFail: Stdout was not exactly Compilation error when compiling "foo": Undefined macro 'foo' used in filter.
TestFail: Stdout was not exactly Compilation error when compiling "evt.type=execve foo": 17: syntax error, unexpected
'foo', expecting 'or', 'and'
TestFail: Stdout of falco process '/home/user/src/falco/test/rules/invalid_base_rule.yaml: Ok
TestFail: Stdout was not exactly Undefined macro 'bar' used in filter.
TestFail: Stdout of falco process 'Ok
TestFail: Stdout of falco process 'Warning: list cat_binaries not refered to by any rule/macro/list
TestFail: Stderr of falco process did not contain content matching event drop detected: 9 occurrences

(1/2) /home/user/src/falco/test/falco_test.py:FalcoTest.test;docker_package-9934:  ERROR: Falco command "docker run --rm --name falco-test --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /root/.sysdig:/root/.sysdig:ro -v /usr:... (4.39 s)
 (2/2) /home/user/src/falco/test/falco_test.py:FalcoTest.test;centos_package-6886:  ERROR: fail() takes at most 2 arguments (4 given) (0.14 s)

ERROR| OSError: [Errno 8] Exec format error (/home/user/build/falcoctl convert psp --psp-path /home/user/src/falco/test/psps/privileged_name_with_spaces.yaml --rules-path /home/user/build/psp_rules.yaml)

How to reproduce it (as minimally and precisely as possible):

$ git clone https://github.com/sunil1783/falco
$ cd falco && git checkout falco_ppc64le && cd ..
$ git clone https://github.com/shirodkara/sysdig
$ mkdir falco/build
$ cd falco/build
$ cmake -DBUILD_BPF=True -DCMAKE_VERBOSE_MAKEFILE=On -DSYSDIG_DIR=/sysdig ..
$ ../test/run_regression_tests.sh /build

What you expected to happen:
Falco test cases should passed.

Anything else we need to know?:

Investigation/Queries :

1. After debugging it seems that while test execution all the events from ".scap" files are not
   written on output console.
   Ex. Only 1 event log appears in log file,whereas expected are 2 for rule Change thread
   namespace. ( Please check the attached log files.)

job-2019-12-18T02.21-c2df134.log
job-2019-12-18T02.24-125a3f9.log
job-2019-12-18T02.27-4ed2e27.log
job-2019-12-18T02.27-eaa9500.log
job-2019-12-18T02.29-0e58290.log

As the console output is used to compare the event count for rule, may be due to this these 
errors are are occurring.
Tried to update the value of variable duration_to_tot=50 in file falco/userspace/falco/falco.cpp 
before calling  do_inspect(). This gives the time to read the event data and update onto console, 
Due to this the test case expected and actual result are match.
But this timeout change is not appropriate , as its value can be different from platform to 
platform.

2. Test cases which gives errors "fail() takes at most 2 arguments" and "Exec format error" are
   depend on Intel x86_64specific docker image/containers/binaries. So need to port these docker
   image/containers for ppc64le platform.

   • Falco docker files (Ex. docker/local/Dockerfile), download packages which are already build
     using gcc 6 and gcc 5. As gcc 6 and 5 is no longer included in Debian unstable platform, So
     how to upgrade these packages for other platform.
   • Is there any plan to support falco on other architecture platform.?
     What should we do for power platform? Some of the falco test cases are dependent on these
     docker images/containers ,if we by pass these test cases what will be its impact?

Environment:

• Falco version:
  git clone https://github.com/sunil1783/falco
  git clone https://github.com/shirodkara/sysdig

• System info:
  {
  "machine": "ppc64le",
  "nodename": "pts00450-vm2",
  "release": "4.15.0-66-generic",
  "sysname": "Linux",
  "version": "Readme tweaks #75-Ubuntu SMP Tue Oct 1 05:24:20 UTC 2019"
  }

• OS:
  NAME="Ubuntu"
  VERSION="18.04.3 LTS (Bionic Beaver)"

• Kernel:
  4.15.0-66-generic

• Installation method:
  Build from source code

[sunil1783]

I have done the changes in falco.cpp::do_inspect() to fix the test cases failed issue.
#932.

Currently when I run the test cases, following test cases are failed and gives errors as follows :

Running: avocado run --mux-yaml /home/github/falco/test/falco_tests_package.yaml --job-results-dir /home/github/falco/test/job-results -- /home/github/falco/test/falco_test.py
JOB ID : 7e7c4f2ee17e7ef68fd409c1d5ad5dee3c25d48a
JOB LOG : /home/github/falco/test/job-results/job-2019-12-25T01.55-7e7c4f2/job.log
(1/2) /home/github/falco/test/falco_test.py:FalcoTest.test;docker_package-9934: ERROR: Falco command "docker run --rm --name falco-test --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /root/.sysdig:/root/.sysdig:ro -v /usr:... (3.05 s)
(2/2) /home/github/falco/test/falco_test.py:FalcoTest.test;centos_package-6886: FAIL: Package path /home/github/falco/build/falco*.rpm did not match exactly 1 file. Instead it matched: (0.12 s)
RESULTS : PASS 0 | ERROR 1 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 4.15 s
job.log

I have Install falco after building from Source code .

I have following queries related to docker images and deb/rpm packages.

1. How to generate falco deb and rpm packages?
   Will it be generated when we build the falco source code.? OR
   Should I need to create these packages using the build and what will be the structure/content
   of it.?

2. How to build the docker images (Ex. falcosecurity/falco:test), Which is used while running the test
   cases.
   Should we build docker images using docker file (Ex.'docker/tester/Dockerfile) '?

3. Some of the docker files ( Ex. docker/local/Dockfile) download the prebuilt gcc (v6 and 5)
   dependent packages.
   On ppc64le platform I need to rebuild these packages again as gcc(v6&5) is not available on
   debian:unstable. So how can I build these packages ? if we by pass these test cases what will be
   its impact?
   Is there any plan to support falco on other architecture platform.?

[sunil1783]

I have generated the deb/rpm packages using CMake -DCPACK_SOURCE_RPM=ON and
#make package.
Build the falcoctl (https://github.com/falcosecurity/falcoctl) binary on ppc64le platform using GO package setup.
https://tecadmin.net/install-go-on-ubuntu/

So currently all the falco test cases are passed, except following test case specific to docker image.
Because the respective "local/Dockerfile" contains the Intel specific prebuilt gcc dependent packages.
which can't install on ppc64le paltform.

(1/2) /home/github/falco/test/falco_test.py:FalcoTest.test;docker_package-9934: ERROR: Falco command "docker run --rm --name falco-test --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /root/.sysdig:/root/.sysdig:ro -v /usr:... (2.90 s)
stderr: "Unable to find image 'falcosecurity/falco:test' locally\ndocker: Error response from daemon: manifest for falcosecurity/falco:test not found.\nSee 'docker run --help'.\n"

Need help to port the following Dockerfiles for ppc64le , which download the prebuilt gcc (v6 and 5) dependent Intel packages.
dev/Dockerfile : FROM debian:unstable
minimal/Dockerfile : FROM ubuntu:18.04 as ubuntu
stable/Dockerfile : FROM debian:unstable
local/Dockerfile : FROM debian:unstable

How can we build these packages for ppc64le platform? I am thinking of using Advanced toolchain with following versions.
https://developer.ibm.com/linuxonpower/advance-toolchain/advtool-installation/#installUbuntu

• Advance toolchain version 10 - gcc 6 - Ubuntu 16.04
• Advance toolchain version 9 - gcc 5 - Ubuntu 14.10

But it will not work for ubuntu:18.04 specific docker files.

Please let me know if you have any ideas/suggestions to port these dockerfiles for ppc64le platform.

[sunil1783]

Env : Ubuntu 18.04 having kernel version 4.15.0-66-generic.
PR : #932

Directly On host :

• Build falco and created packages with eBPF enabled.
• All tests are passed, except docker container specific test which failed.

Tried to build the builder/Dockerfile image for ppc64le, but getting error as
'Cannot find a valid baseurl for repo: centos-sclo-sclo/ppc64le'.

On tester container :
While running regression tests.
- Build falcosecurity/falco-tester using tester/Dockerfile.
- falcosecurity/falco-tester container invokes entrypoint 'tester/root/usr/bin/entrypoint'.
- It tries to build 'falcosecurity/falco:test' image using local/Dockerfile.
- But the local/Dockerfile download prebuilt intel packages for gcc-6/5 and binutils, these are not
valid for ppc64le ,So 'falcosecurity/falco:test' image creation failed.

Need help/suggetion to port these types of dockerfiles for ppc64le.
- To change the dockerfiles to support the latest kernels (>= 4.14) with eBPF support.
- So need to update the OS type in dockerfiles, like centos:7 (builder/Dockerfile).
I checked for ppc64le,But 'centos:latest'(In builder/Dockerfile) image is not available.

How can we provide the support for latest gcc for falco on ppc64le?

[leodido]

Thanks @sunil1783 for reporting all this info.

I here see two main topics:

1. Falco team plans (and resources) to port it to other platforms
2. Falco (regression/integration) test suite

Regarding point 1.

We should discuss it all together.

Would you mind joining the today community call maybe? You can find the calendar/zoom invitation in the README.
It'd be very useful also to explain in person to all the participant the findings you described here in this issue.

Regarding point 2.

During past community calls we often discussed the test suite situation.
And we all ended up agreeing that it needs to be done with a different approach, probably from scratch.

Regarding the specific test case that is blocking you it would be acceptable to disable it for the ppc64le platform, imho. Unfortunately I do not know how to instruct Avocado (the current python testing framework Falco repo is using) to do so.

/triage support

/milestone 1.0.0

[sunil1783]

Thanks for the response.
I will join the next community call and discuss on the above issue which we are facing on ppc64le ,while using falco-builder/falco-tester containers to build/test falco code.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[leodido]

Any update on this? L.

…

On Sun, Mar 22, 2020 at 5:42 PM stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#982 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA5J43S4CZZJR54JI34KDDRIY5X5ANCNFSM4J4JB6PQ> .

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.