Unable to run Regression test on host after build from source.

[prankkelkar]

What happened:
Tried running Regression test using command ./test/run_regression_tests.sh $PWD/build.
The test case execution gave following error.

JOB LOG    : /root/falco/test/job-results/job-2019-08-27T10.37-c4333bd/job.log
Error running method "initialize" of plugin "yaml_to_mux": 'Namespace' object has no attribute 'get'
 (1/1) /root/falco/test/falco_test.py:FalcoTest.test: ERROR: [Errno 2] No such file or directory (/build/release/userspace/falco/falco -r /root/falco/test/../rules/falco_rules.yaml   -c /root/falco/test/../falco.yaml  -o json_output=False -o json_include_output_property=True -o priority=debug -v) (0.04 s)
RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 0.16 s

What you expected to happen:
Successful execution of tests.

How to reproduce it (as minimally and precisely as possible):
After Complete build. Follow steps from falco documentation to run test cases.( https://falco.org/docs/source/#run-the-tests )

Anything else we need to know?:
Looks like it is searching for specific folder structure (build/release/).
Observed that release directory did not get created after building falco.

Environment:

• Falco version (use falco --version):
  0.17.0

• OS (e.g: cat /etc/os-release):

VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a):
  Linux ************** 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4 22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools (e.g. in kubernetes, rpm, deb, from source):
  Build from Source.

[leodido]

Thanks for reporting this.

Moving the /build directory (built from docker-builder) into /build/release/ should work. But this is a manual workaround and we do not like it. So we agree that this flow for running regression tests locally must be improved.

/assign @fntlnz
/assign @leodido

[prankkelkar]

Yes I have tried this workaround but still facing issues while running test cases.

[prankkelkar]

I had few questions
1- Is it possible to run test cases after build from source on host without using falcon builder/tester images?
2- Can only tester image be used to run test cases for package built from source without using builder image?

[leodido]

I confirm that the bug exists. Would you like to try to submit a first PR to fix it?

This sounds like a good first issue to approach the project. :)

[prankkelkar]

Thanks @leodido for suggesting a fix to this bug.( #808 ).
Tried this but it has same outcome like copying the contents of build folder to required folder like it was done earlier manually. The error for missing directory is gone but other issue still persists Error running method "initialize" of plugin "yaml_to_mux": 'Namespace' object has no attribute 'get' .
One thing i would like to mention that i have followed steps from here https://falco.org/docs/source/ and haven't used faco-builder image to build the package.

[leodido]

/good-first-issue

[poiana]

@leodido:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[prankkelkar]

@leodido Thanks for considering this issue. It would be really helpful if we can get test cases running on host directly.

[fntlnz]

Since this is now open for new contributors, I'm unassigning myself and leo for now.

/unassign @fntlnz
/unassign @leodido

[mstemm]

I think I made this easier in 6e11e75#diff-e7e5a4947037bfbf38f675b1e40611a2. Try setting BUILD_DIR to the directory on the host where your build directory is before running the regression tests.

Also, you'll want to make sure you have the python package https://github.com/avocado-framework/avocado installed (our tester image uses 0.69.0) and also has the yaml-to-mux plugin installed. https://github.com/falcosecurity/falco/blob/dev/docker/tester/Dockerfile shows the installation steps to install it, from a fedora base image.

[prankkelkar]

Yes I have tried all these things but no luck with avocado framework. Anyways i managed to apply a hack wherein i used tester image mounting required directories and running steps in container bash. Could get it running this way . But on host issue persists

[leodido]

/area build

[sunil1783]

I have done the required changes to build falco on Ubuntu 18.04 ppc64le platform.
But while running the regression test cases get following error.

#./test/run_regression_tests.sh $PWD/build

Error running method "initialize" of plugin "yaml_to_mux": 'Namespace' object has no attribute 'get'
(1/1) /source/falco/test/falco_test.py:FalcoTest.test: ERROR: Falco command "/build/release/userspace/falco/falco -r /source/falco/test/../rules/falco_rules.yaml -c /source/falco/test/../falco.yaml -o json_output=False -o json_include_output_property=True -o priority=debug -v" exited with unexpected return value ... (0.16 s)

Currently the falco-tester container is for Intel platform. So I have done the required changes specific to Ubuntu ppc64le platform in falco-tester container docker file and build the docker image.
But while running the regression test cases using this docker image container getting the same error "'Namespace' object has no attribute 'get'" as mention above.

#docker run --user $(id -u):$(id -g) -v $HOME:$HOME:ro -v /boot:/boot:ro -v /var/run/docker.sock:/var/run/docker.sock -v /etc/passwd:/etc/passwd:ro -e FALCO_VERSION=${FALCO_VERSION} -v /home/user/src:/source -v /home/user/build/:/build falco-tester_power test

Please let me know if any pointers to solve this issue.

Environment :

• Falco Version : 0.18.0
• Kernel : 4.15.0-66-generic
• OS :
  root@pts00450-vm2:~# cat /etc/os-release
  NAME="Ubuntu"
  VERSION="18.04.3 LTS (Bionic Beaver)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 18.04.3 LTS"
  VERSION_ID="18.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=bionic
  UBUNTU_CODENAME=bionic

[leodido]

Thanks for looking into this @sunil1783 !

Anyway having a PR with the work you have done in order to support ppc64le platform would be helpful for the debugging :)

Could you open it and link back to this issue (it should have been a new different issue but ok) ?

[sunil1783]

On Ubuntu 18.04 ppc64le Default 'avocado-framework-plugin-varianter-yaml-to-mux' installed version is higher than expected(73).
So downgraded it to version 69 , same as avocado-framework version.
pip install avocado-framework==69 avocado-framework-plugin-varianter-yaml-to-mux==69

After this run the regression test cases, error Namespace' object has no attribute 'get' not occurred.
The test cases are executed , but some of the test are failing and giving following types of errors.
test_output.txt

After debugging it seems that while test execution all the events from ".scap" files are not written on console(Getting SCAP_TIMEOUT while reading the event in falco.cpp:do_inspect() )
As the console output is used to compare the event count for rule. So these errors are occurring.

Line 417: 00:30:07 ERROR| TestFail: Different counts for rule Change thread namespace: expected=2, actual=1
Line 583: 00:30:07 ERROR| TestFail: Different counts for rule Launch Privileged Container: expected=3, actual=1
Line 749: 00:30:08 ERROR| TestFail: Different counts for rule Launch Sensitive Mount Container: expected=3, actual=1
Line 1081: 00:30:12 ERROR| TestFail: Different counts for rule Read sensitive file untrusted: expected=1, actual=0
Line 1247: 00:30:15 ERROR| TestFail: Different counts for rule Write below binary dir: expected=4, actual=1
Line 1413: 00:31:23 ERROR| TestFail: Different counts for rule Change thread namespace: expected=2, actual=1
Line 2428: 00:32:32 ERROR| TestFail: Different counts for rule Open From Cat ($.*+?()[]{}|^): expected=8, actual=1
Line 3745: 00:32:56 ERROR| TestFail: Different counts for rule open_13: expected=1, actual=0
Line 4241: 00:32:59 ERROR| TestFail: Different counts for rule open_10: expected=1, actual=0
Line 4836: 00:33:08 ERROR| TestFail: Different counts for rule detect_open: expected=2, actual=0
Line 915: 00:30:09 ERROR| TestFail: Could not find a line 'ERROR: ' in falco output
Line 2518: 00:32:34 ERROR| TestFail: Could not find a line 'WARNING: ' in falco output
Line 2867: 00:32:48 ERROR| TestFail: Stdout of falco process 'Warning: macro some macro not refered to by any rule/macro
Line 2995: 00:32:48 ERROR| TestFail: Stdout was not exactly Compilation error when compiling "foo": Undefined macro 'foo' used in filter.
Line 3093: 00:32:49 ERROR| TestFail: Stdout was not exactly Compilation error when compiling "evt.type=execve foo": 17: syntax error, unexpected 'foo', expecting 'or', 'and'
Line 3195: 00:32:50 ERROR| TestFail: Stdout of falco process '/home/user/src/falco/test/rules/invalid_base_rule.yaml: Ok
Line 3351: 00:32:50 ERROR| TestFail: Stdout was not exactly Undefined macro 'bar' used in filter.
Line 3435: 00:32:50 ERROR| TestFail: Stdout of falco process 'Ok
Line 4995: 00:33:14 ERROR| TestFail: Stdout of falco process 'Warning: list cat_binaries not refered to by any rule/macro/list
Line 4914: 00:33:13 ERROR| TestFail: Stderr of falco process did not contain content matching event drop detected: 9 occurrences

I will upload the code changes in PR and link to this issue.

[leodido]

/milestone 1.0.0

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.