http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

exiett · 2022-10-26T13:38:43Z

Describe the bug
After installing Falco in a Kubernetes cluster using the official Helm chart, whenever I try to output the results in a JSON format to an HTTP endpoint, Falco gives me a SSL error. Isn't this ca-certificate package supposed to come with the default installation?

How to reproduce it

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm install falco falcosecurity/falco
exec into one of the pods running Falco.
falco -o json_ouput=true http_output.enabled=true http_output.url=<REQUESTBIN_ADDRESS>
exec into another pod (or even the same pod) to generate a Falco event.

Expected behaviour
Falco sends an HTTP call to the endpoint containing the JSON of the syscall it just registered.

Screenshots

Output when running curl <MY_REQUEST_BIN> from inside a pod running Falco:

{"about":"Pipedream is the fastest way to connects APIs. Build and run workflows with code-level control when you need it — and no code when you don't.","event_id":"2GfreizFr8I0hJjji77gNQEoP8Y","workflow_id":"p_LQCDnkR","owner_id":"u_KDh7vnG","deployment_id":"d_zqs7eNxx","timestamp":"2022-10-26T13:22:56.700Z","inspect":"https://pipedream.com/@/p_LQCDnkR","quickstart":"https://pipedream.com/quickstart/"}

Output when Falco receives an event with json_output=true and http_output=true enabled to the same requestbin address:

Fri Oct 21 18:02:03 2022: libcurl error: Problem with the SSL CA cert (path? access rights?)

Command we are running from within the pod: falco -o json_ouput=true -o http_output.enabled=true -o http_output.url=<MY_REQUEST_BIN_ADDRESS>

Environment

Falco version: 0.33.0 (x86_64)
System info: "machine":"x86_64","nodename":"falco-2tcqs","release":"5.13.0-1031-aws","sysname":"Linux","version":"#35~20.04.1-Ubuntu SMP Mon Jun 13 22:30:30 UTC 2022"
Cloud provider or hardware configuration:
OS: Debian GNU/Linux 11 (bullseye)
Kernel: Linux falco-2tcqs 5.13.0-1031-aws #35~20.04.1-Ubuntu SMP Mon Jun 13 22:30:30 UTC 2022 x86_64 GNU/Linux
Installation method: Official Helm Chart

Additional context
TL;DR: We're trying to use the built-in functionality of HTTP Output that comes with Falco to send JSON output about syscalls that Falco catches inside our containers. The main idea is to understand if this works or Sidekick is the only alternative.

The text was updated successfully, but these errors were encountered:

alacuku · 2022-10-27T10:43:46Z

Hi @exiett, thanks for reporting the issue. The following PR was made to fix the bug: falcosecurity/libs#442 but it seems that the issue is still present. We are going to investigate further and fix the bug.

trallnag · 2023-01-18T20:52:59Z

The issue is still there btw.

Wed Jan 18 20:47:02 2023: Falco version: 0.33.1-105+c1985a7 (x86_64)
...
Wed Jan 18 20:47:58 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)
Wed Jan 18 20:48:18 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)

I am testing it by pointing Falco to a pipedream request bin like https://xxx.m.pipedream.net.

Issif · 2023-01-18T21:48:40Z

The certificates for CA were missing in the docker image, the PR #2355 fixes that for the upcoming release

jasondellaluce · 2023-01-19T11:36:36Z

/milestone 0.34.0

leogr · 2023-01-23T10:57:54Z

I am assuming #2355 has fixed this PR, so
/close

poiana · 2023-01-23T10:57:57Z

@leogr: Closing this issue.

In response to this:

I am assuming #2355 has fixed this PR, so
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

movd · 2023-01-24T08:19:52Z

Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS.

$ falco --version
Falco version: 0.33.1
Libs version:  0.9.2
Plugin API:    2.0.0
Driver:
  API version:    2.0.0
  Schema version: 2.0.0
  Default driver: 3.0.1+driver
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 11 (bullseye)
Release:	11
Codename:	bullseye

ca-certificates is installed

ca-certificates/stable,now 20210119 all [installed,automatic]

The certifcate is valid:

$ curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.

Error

$ sudo journalctl -fu falco
...
Jan 24 09:10:35 node01.example.com falco[890543]: libcurl error: Problem with the SSL CA cert (path? access rights?)
...

movd · 2023-01-24T08:20:03Z

/reopen

poiana · 2023-01-24T08:20:04Z

@movd: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

leogr · 2023-01-24T16:45:31Z

Quoting @movd :

Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS.

👍

/reopen

Note the issue may be different. Could you provide more detail about your OS/distro/environment? Or a detailed repro. Thanks in advance

cc @Issif @jasondellaluce @LucaGuerra

poiana · 2023-01-24T16:45:34Z

@leogr: Reopened this issue.

In response to this:

Quoting @movd :

Reopening this I'm experiencing the same problem but in normal package install on a Debian Linux. I want to talk to Falcosidekick via HTTPS.

👍

/reopen

Note the issue may be different. Could you provide more detail about your OS/distro/environment? Or a detailed repro. Thanks in advance

cc @Issif @jasondellaluce @LucaGuerra

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Issif · 2023-01-24T17:03:11Z

I'm pretty sure it's same issue. If it's ok in the next release 0.34 we'll be sure.

movd · 2023-01-25T10:13:24Z

Yes sure. I can update the issue once 0.34 is out. Our setup will automatically install it.

Describe the bug

When I set up http_output with an https URL like so:

http_output:
  enabled: true
  url: https://falcosidekick.apps.example.com

I'll receive the following errors:

Jan 24 09:10:35 node01.example.com falco[890543]: libcurl error: Problem with the SSL CA cert (path? access rights?)

As the URL states, I deploy a Falcosidekick inside a Kubernetes Cluster, but our Falco processes lives outside of the cluster and are installed via apt. When using unencrypted HTTP, it works just fine.

How to reproduce it

Enable http_output with a valid https endpoint (see above). Do something to trigger an Falco alert that should get forwarded to your https destination. E. g.:

Trigger Error File below /etc opened for writing rule:

root@node01:~# echo "bad file" > /etc/falco_test.sh

Falco can't forward the alert:

$ sudo journalctl -fu falco.service
...
Jan 25 09:44:05 node01.example.com falco[939076]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Jan 25 09:44:05 node01.example.com falco[939076]: {"hostname":"node01.example.com","output":"09:44:05.330918780: Error File below /etc opened for writing (user=<NA> user_loginuid=-1 command=prometheus-conf --listen-address=:8080 --reload-url=http://127.0.0.1:9093/-/reload --config-file=/etc/alertmanager/config/alertmanager.yaml.gz --config-envsubst-file=/etc/alertmanager/config_out/alertmanager.env.yaml pid=66005 parent=containerd-shim pcmdline=containerd-shim -namespace k8s.io -id 5c3c2ab4942c4dec45c48a32a06d56843706ba94748a4ed33d2e8cad9d6dae71 -address /run/containerd/containerd.sock file=/etc/alertmanager/config_out/alertmanager.env.yaml.tmp program=prometheus-conf gparent=systemd ggparent=<NA> gggparent=<NA> container_id=033b1aeea6a0 image=quay.io/prometheus-operator/prometheus-config-reloader)","priority":"Error","rule":"Write below etc","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2023-01-25T08:44:05.330918780Z", "output_fields": {"container.id":"033b1aeea6a0","container.image.repository":"quay.io/prometheus-operator/prometheus-config-reloader","evt.time":1674636245330918780,"fd.name":"/etc/alertmanager/config_out/alertmanager.env.yaml.tmp","proc.aname[2]":"systemd","proc.aname[3]":null,"proc.aname[4]":null,"proc.cmdline":"prometheus-conf --listen-address=:8080 --reload-url=http://127.0.0.1:9093/-/reload --config-file=/etc/alertmanager/config/alertmanager.yaml.gz --config-envsubst-file=/etc/alertmanager/config_out/alertmanager.env.yaml","proc.name":"prometheus-conf","proc.pcmdline":"containerd-shim -namespace k8s.io -id 5c3c2ab4942c4dec45c48a32a06d56843706ba94748a4ed33d2e8cad9d6dae71 -address /run/containerd/containerd.sock","proc.pid":66005,"proc.pname":Wed Jan 25 09:44:05 2023: libcurl error: Problem with the SSL CA cert (path? access rights?)

Expected behaviour

Event gets forwarded to the https destination

Environment

Falco version:

root@node01:~# falco --version
Falco version: 0.33.1
Libs version:  0.9.2
Plugin API:    2.0.0
Driver:
  API version:    2.0.0
  Schema version: 2.0.0
  Default driver: 3.0.1+driver

System info:

root@node01:~# falco --support | jq .system_info
Wed Jan 25 09:46:01 2023: Falco version: 0.33.1 (x86_64)
Wed Jan 25 09:46:01 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed Jan 25 09:46:01 2023: Loading rules from file /etc/falco/falco_rules.yaml
Wed Jan 25 09:46:01 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
{
  "machine": "x86_64",
  "nodename": "node01.example.com",
  "release": "5.10.0-21-amd64",
  "sysname": "Linux",
  "version": "#1 SMP Debian 5.10.162-1 (2023-01-21)"
}

Cloud provider or hardware configuration:

KVM based virtual Machine (on premise)

OS:

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Kernel:

root@node01:~# uname -a
Linux node01.example.com 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux

Installation method:
Via offical deb straight like described in the offical docs: https://falco.org/docs/getting-started/installation. I added a Drop-In unit so that Falco uses the eBPF driver:

# systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
     Loaded: loaded (/lib/systemd/system/falco.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/falco.service.d
             └─ebpf.conf
     Active: active (running) since Wed 2023-01-25 09:43:16 CET; 3min 36s ago
       Docs: https://falco.org/docs/
   Main PID: 939076 (falco)
      Tasks: 18 (limit: 19165)
     Memory: 62.7M
        CPU: 7.204s
     CGroup: /system.slice/falco.service
             └─939076 /usr/bin/falco --pidfile=/var/run/falco.pid
...

root@node01:~# cat /etc/systemd/system/falco.service.d/ebpf.conf
# Ansible managed
# Ref: https://falco.org/docs/event-sources/drivers/#ebpf-probe
# Empty FALCO_BPF_PROBE environment variable enables eBPF support
[Service]
Environment=FALCO_BPF_PROBE=""
ExecStartPre=
ExecStopPost=

Additional context

The certificate is valid and works fine with plain curl (from same host to the same destination).

$ curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.

Issif · 2023-02-08T16:16:21Z

Sadly, even with 0.34 we face SSL issues #2395 (comment)

movd · 2023-02-09T07:56:01Z

Can confirm I'm still sadly facing the same issue.

falco.yaml

http_output:
  enabled: true
  url: https://falcosidekick.apps.example.com
  user_agent: "falcosecurity/falco"

unit journal:

root@node04:~# journalctl -fu falco-bpf.service
-- Journal begins at Wed 2023-02-08 06:28:02 CET. --
Feb 09 08:51:02 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:07 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:12 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)
Feb 09 08:51:17 node04.k8s-dgi-dev.dg-i.net falco[850382]: libcurl error: Problem with the SSL CA cert (path? access rights?)

I'm using a valid certificate:

root@node04:~# curl -vL 'https://falcosidekick.apps.example.com' 2>&1 | grep -A6 'Server certificate:'
* Server certificate:
*  subject: CN=*.apps.example.com
*  start date: May  3 00:00:00 2022 GMT
*  expire date: May  3 23:59:59 2023 GMT
*  subjectAltName: host "falcosidekick.apps.example.com" matched cert's "*.apps.example.com"
*  issuer: C=US; O=DigiCert Inc; CN=RapidSSL TLS DV RSA Mixed SHA256 2020 CA-1
*  SSL certificate verify ok.

Environment:

root@node04:~# falco --support | jq .system_info
Thu Feb  9 08:54:52 2023: Falco version: 0.34.0 (x86_64)
Thu Feb  9 08:54:52 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Thu Feb  9 08:54:52 2023: Loading rules from file /etc/falco/falco_rules.yaml
Thu Feb  9 08:54:52 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
{
  "machine": "x86_64",
  "nodename": "node04.example.com",
  "release": "5.10.0-21-amd64",
  "sysname": "Linux",
  "version": "#1 SMP Debian 5.10.162-1 (2023-01-21)"
}

Let me know If I can be of any help :)

alacuku · 2023-02-09T08:16:41Z

I'm working on that: #2412 . I'm gonna run some tests to make sure that solves the problem!

FedeDP · 2023-02-09T09:02:57Z

Moving to next milestone for now; possibly a 0.34.1 patch-release will be released though!
/milestone 0.35.0

alacuku · 2023-02-20T16:53:59Z

Falco 0.34.1 is out. The patch release should solve this issue. Could you please check it out?
cc @movd @exiett

PaulFurtado · 2023-03-21T00:06:37Z

Even after those fixes, this is still not fixed for RHEL-based distros, I left more details in a comment here: https://github.com/falcosecurity/libs/pull/871/files#r1142781849

It would also be great if the HTTP feature allowed users to specify a custom path to a CA bundle (ex: for using internal purpose-specific CAs). If you added an option ca_file to the yaml confi, you could pass it to curl with CURLOPT_CAINFO. Having this option would also allow users to workaround this bug without recompiling.

leogr · 2023-03-21T07:19:52Z

cc @alacuku

alacuku · 2023-03-21T07:37:00Z

Hi @PaulFurtado, thanks for testing and reporting this. We definitely need to make the CA/certs path configurable.

exiett added the kind/bug label Oct 26, 2022

poiana added this to the 0.34.0 milestone Jan 19, 2023

poiana closed this as completed Jan 23, 2023

poiana reopened this Jan 24, 2023

poiana modified the milestones: 0.34.0, 0.35.0 Feb 9, 2023

This was referenced Feb 9, 2023

fix(dockerfile/no-driver): install ca-certificates #2412

Merged

fix(libcurl): set path to ca-certificates at build time for libcurl falcosecurity/libs#871

Merged

Cut Falco 0.34.1 release #2421

Closed

This was referenced Mar 21, 2023

Make CA/certs path configurable for http_output #2448

Closed

new(falco/config): add new configurations for http_output #2458

Merged

poiana closed this as completed in #2458 Mar 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

exiett commented Oct 26, 2022

alacuku commented Oct 27, 2022

trallnag commented Jan 18, 2023

Issif commented Jan 18, 2023

jasondellaluce commented Jan 19, 2023

leogr commented Jan 23, 2023

poiana commented Jan 23, 2023

movd commented Jan 24, 2023 •

edited

Loading

movd commented Jan 24, 2023

poiana commented Jan 24, 2023

leogr commented Jan 24, 2023

poiana commented Jan 24, 2023

Issif commented Jan 24, 2023

movd commented Jan 25, 2023

Issif commented Feb 8, 2023

movd commented Feb 9, 2023

alacuku commented Feb 9, 2023

FedeDP commented Feb 9, 2023

alacuku commented Feb 20, 2023

PaulFurtado commented Mar 21, 2023

leogr commented Mar 21, 2023

alacuku commented Mar 21, 2023

http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

http_output not working, libcurl error: Problem with the SSL CA cert (path? access rights?) #2274

Comments

exiett commented Oct 26, 2022

alacuku commented Oct 27, 2022

trallnag commented Jan 18, 2023

Issif commented Jan 18, 2023

jasondellaluce commented Jan 19, 2023

leogr commented Jan 23, 2023

poiana commented Jan 23, 2023

movd commented Jan 24, 2023 • edited Loading

movd commented Jan 24, 2023

poiana commented Jan 24, 2023

leogr commented Jan 24, 2023

poiana commented Jan 24, 2023

Issif commented Jan 24, 2023

movd commented Jan 25, 2023

Issif commented Feb 8, 2023

movd commented Feb 9, 2023

alacuku commented Feb 9, 2023

FedeDP commented Feb 9, 2023

alacuku commented Feb 20, 2023

PaulFurtado commented Mar 21, 2023

leogr commented Mar 21, 2023

alacuku commented Mar 21, 2023

movd commented Jan 24, 2023 •

edited

Loading