Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running as kubernetes daemonset failing #225

Closed
trmiller opened this issue Mar 21, 2017 · 5 comments · Fixed by #226
Closed

Running as kubernetes daemonset failing #225

trmiller opened this issue Mar 21, 2017 · 5 comments · Fixed by #226
Assignees

Comments

@trmiller
Copy link

trmiller commented Mar 21, 2017

Attempting to run falco as a daemonset on kubernetes v1.5.2

I'm using the example yaml file in this github repo, subtracting the post out to slack messenger.

The daemonset is failing with the following logs:

2017-03-21T19:22:54.356123597Z Tue Mar 21 19:22:54 2017: Falco initialized with configuration file /etc/falco.yaml

2017-03-21T19:22:54.478623894Z Tue Mar 21 19:22:54 2017: Parsed rules from file /etc/falco_rules.yaml

2017-03-21T19:22:54.491812317Z Tue Mar 21 19:22:54 2017: Unable to load the driver. Exiting.

2017-03-21T19:22:54.492146169Z Tue Mar 21 19:22:54 2017: Runtime error: error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.. Exiting.

I have the sysdig cloud agent running as a daemonset already. Anything I'm doing wrong?

mstemm added a commit that referenced this issue Mar 21, 2017
If a daemonset specifies a command, this overrides the entrypoint. In
falco's case, the entrypoint handles the details of loading the kernel
driver, so specifying a command accidently prevents the driver from
being loaded.

This happens to work if you had a previously loaded sysdig_probe driver
lying around.

The fix is to specify args instead. In this case, the driver will be
loaded via the entrypoint.

This fixes #225.
@mstemm mstemm self-assigned this Mar 21, 2017
@mstemm
Copy link
Contributor

mstemm commented Mar 21, 2017

Thanks. I found the problem--I was using command to start the falco pod when I should have been using args. This will be fixed in #226.

@trmiller
Copy link
Author

trmiller commented Mar 21, 2017

I tried your fix on my own, and still seeing issues, though it does get further. Looks like the permissions on that bucket may be too locked down?

2017-03-21T21:29:32.783056198Z * Setting up /usr/src links from host
2017-03-21T21:29:32.795356137Z ls: cannot access '/host/usr/src': No such file or directory
2017-03-21T21:29:32.804122756Z * Unloading sysdig-probe, if present
2017-03-21T21:29:32.804993188Z rmmod: ERROR: Module sysdig_probe is not currently loaded
2017-03-21T21:29:32.825740328Z * Running dkms autoinstall
2017-03-21T21:29:33.511926152Z 
2017-03-21T21:29:33.511957532Z Kernel preparation unnecessary for this kernel.  Skipping...
2017-03-21T21:29:33.526017613Z 
2017-03-21T21:29:33.526038760Z Building module:
2017-03-21T21:29:34.886356613Z cleaning build area...
2017-03-21T21:29:37.572046446Z make -j1 KERNELRELEASE=4.9.9-coreos-r1 -C /lib/modules/4.9.9-coreos-r1/build M=/var/lib/dkms/sysdig/0.13.0/build...(bad exit status: 2)
2017-03-21T21:29:37.572682162Z Error! Bad return status for module build on kernel: 4.9.9-coreos-r1 (x86_64)
2017-03-21T21:29:37.572696108Z Consult /var/lib/dkms/sysdig/0.13.0/build/make.log for more information.
2017-03-21T21:29:37.573686618Z * Trying to load a system sysdig-probe, if present
2017-03-21T21:29:37.574815532Z * Trying to load a dkms sysdig-probe, if present
2017-03-21T21:29:37.575703912Z * Trying to find precompiled sysdig-probe for 4.9.9-coreos-r1
2017-03-21T21:29:37.584066041Z Found kernel config at /proc/config.gz
2017-03-21T21:29:37.590296259Z * Trying to download precompiled module from https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.13.0-x86_64-4.9.9-coreos-r1-b5d86d6036ba49f0f5b91abb7d2cab0e.ko
2017-03-21T21:29:58.150105250Z Download failed, consider compiling your own sysdig-probe and loading it or getting in touch with the sysdig community
2017-03-21T21:29:58.155564402Z Tue Mar 21 21:29:58 2017: Falco initialized with configuration file /etc/falco.yaml
2017-03-21T21:29:58.277443481Z Tue Mar 21 21:29:58 2017: Parsed rules from file /etc/falco_rules.yaml
2017-03-21T21:29:58.288493971Z Tue Mar 21 21:29:58 2017: Unable to load the driver. Exiting.
2017-03-21T21:29:58.288844268Z Tue Mar 21 21:29:58 2017: Runtime error: error opening device /host/dev/sysdig0. Make sure you have root credentials and that the sysdig-probe module is loaded.. Exiting.

When trying to just curl that location I get an Access Denied error.

curl https://s3.amazonaws.com/download.draios.com/stable/sysdig-probe-binaries/sysdig-probe-0.13.0-x86_64-4.9.9-coreos-r1-b5d86d6036ba49f0f5b91abb7d2cab0e.ko

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>removed</RequestId><HostId>removed</HostId></Error>

@mstemm
Copy link
Contributor

mstemm commented Mar 21, 2017

What's happening now is that when the falco container runs, it can't use dkms to build the kernel driver on the host. That's normal, because you're using coreos. As a backup, we provide pre-built versions of the kernel module for a variety of combinations of (sysdig version + coreos version). There's a longer description of this at the end of this blog post: https://sysdig.com/blog/coreos-sysdig-part-1-digging-into-coreos-environments/.

However, we don't have a pre-built copy of that particular version because the sysdig driver version is older than the coreos version. The coreos version using kernel 4.9.9 came out after we migrated from sysdig 0.13.0 to sysdig 0.14.0.

Normally, that wouldn't be a problem and the solution would be to use a newer version of the driver. However, falco itself doesn't currently have its own standalone driver. Instead, it borrows the sysdig kernel module, and since we haven't done a falco release lately, it depends on a too-old sysdig kernel module.

I'm actually in the middle of making changes so falco has its own kernel module, so we don't have these kinds of kernel module dependencies between sysdig and falco--#224.

In the meantime, if you downgrade coreos versions to an older one (something like 1262.0.0 should work), there will be a pre-built driver available.

@trmiller
Copy link
Author

Ah ok, thanks. I'll try downgrading and see if that works for now.

@trmiller
Copy link
Author

Confirmed that downgrading to an older CoreOS version resolved this.
Thanks!

leogr pushed a commit to falcosecurity/rules that referenced this issue Dec 21, 2022
If a daemonset specifies a command, this overrides the entrypoint. In
falco's case, the entrypoint handles the details of loading the kernel
driver, so specifying a command accidently prevents the driver from
being loaded.

This happens to work if you had a previously loaded sysdig_probe driver
lying around.

The fix is to specify args instead. In this case, the driver will be
loaded via the entrypoint.

This fixes falcosecurity/falco#225.
leogr pushed a commit to falcosecurity/rules that referenced this issue Dec 21, 2022
If a daemonset specifies a command, this overrides the entrypoint. In
falco's case, the entrypoint handles the details of loading the kernel
driver, so specifying a command accidently prevents the driver from
being loaded.

This happens to work if you had a previously loaded sysdig_probe driver
lying around.

The fix is to specify args instead. In this case, the driver will be
loaded via the entrypoint.

This fixes falcosecurity/falco#225.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants