Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanup(metrics): improve prometheus and plugin metrics info #1328

Merged
merged 3 commits into from
Jun 4, 2024
Merged

cleanup(metrics): improve prometheus and plugin metrics info #1328

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e963f30
cleanup(metrics): improve prometheus and plugin metrics info
incertum Jun 1, 2024
e6a2f72
fix(metrics): sha256 metric namings
incertum Jun 1, 2024
35e08c5
cleanup(metrics): simplify plugins metrics limitations
incertum Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 66 additions & 44 deletions content/en/docs/metrics/falco-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,12 +80,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -163,12 +163,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -259,12 +259,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -388,12 +388,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -551,12 +551,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -754,12 +754,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -972,12 +972,12 @@ falcosecurity_falco_kernel_release_info{raw_name="kernel_release",kernel_release
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1
# HELP falcosecurity_falco_falco.sha256_rules_file.falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_rules_file.falco_rules_info gauge
falcosecurity_falco_falco.sha256_rules_file.falco_rules_info{raw_name="falco.sha256_rules_file.falco_rules",falco.sha256_rules_file.falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco.sha256_config_file.falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco.sha256_config_file.falco_info gauge
falcosecurity_falco_falco.sha256_config_file.falco_info{raw_name="falco.sha256_config_file.falco",falco.sha256_config_file.falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_falco_sha256_rules_file_falco_rules_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_file_falco_rules_info gauge
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules",falco_sha256_rules_file_falco_rules="f176455ad6a1f39cf32065af14d33042e092b30489d255cbb1eff0dc03e67c5d"} 1
# HELP falcosecurity_falco_falco_sha256_config_file_falco_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_file_falco_info gauge
falcosecurity_falco_falco_sha256_config_file_falco_info{raw_name="falco_sha256_config_file_falco",falco_sha256_config_file_falco="c78b5de8e841917eb2c7a8257f37995e1c9594cffb71ea1e7aefa932172cac3d"} 1
# HELP falcosecurity_falco_evt_source_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_evt_source_info gauge
falcosecurity_falco_evt_source_info{raw_name="evt_source",evt_source="syscall"} 1
Expand Down Expand Up @@ -1238,19 +1238,39 @@ webserver:

This endpoint will allow observation of the internal state of Falco providing the same data as configured for the metrics outputs. It will be served on the same port as the health endpoint.

{{% pageinfo color=info %}}
Following the Prometheus recommendations, there might be some slight differences with regard to the other outputs. Typically calculated fields will not be returned as Prometheus provides the facilities to compute them as part of their queries. Expand the example outputs dropdowns above (for both JSON and Prometheus formats) to explore all supported metrics fields, including their naming conventions and units.
### Limitations and Additional Information

Expand the example outputs dropdowns above (for both JSON and Prometheus formats) to explore all supported metrics fields, including their naming conventions and units.

The Prometheus text format documentation can be found [here][3].

The OpenMetrics specification can be found [here][4].
{{% pageinfo color=info %}}
The `num_evts` wrapper / base field is currently not available for Prometheus metrics; otherwise, there is 1:1 support across all output channels.

However, following the Prometheus recommendations, there might be some slight differences with regard to some metrics fields. Typically calculated fields will not be returned as Prometheus provides the facilities to compute them as part of their queries (e.g. event or drop rates can be calculated in Prometheus).
{{% /pageinfo %}}

[1]: https://github.com/falcosecurity/falco/blob/master/falco.yaml
[2]: https://prometheus.io
[3]: https://prometheus.io/docs/instrumenting/exposition_formats/
[4]: https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md

## Plugin Metrics

The ability to add custom plugin metrics is currently under development (targeting Falco 0.39.0). However, plugin metrics can also include regular metrics when running Falco with either just a plugin source or both the primary syscalls event source and a plugin. This section will inform you about a few current limitations:

- When running Falco with a plugin only on macOS or Windows, there is currently no metrics support.
- Most of the available metrics are only relevant for the primary syscalls event source (e.g., `state_counters_enabled`, `kernel_event_counters_enabled`, and `libbpf_stats_enabled`), not for a plugin with a non-syscalls event source.
- When running Falco with a plugin only on Linux (without using the syscalls event source), it currently doesn't work well due to some issues in Falco's capture initialization phase. We are working on resolving remaining issues by Falco 0.39.0 (see this [issue](https://github.com/falcosecurity/falco/issues/3194#issuecomment-2111009270)). Therefore, the following fields are not available when running Falco with a plugin only on Linux:
- `falcosecurity_falco_kernel_release_info`
- `falcosecurity_evt_hostname_info`
- `falcosecurity_falco_start_timestamp_nanoseconds`
- `falcosecurity_falco_host_boot_timestamp_nanoseconds`
- `falcosecurity_falco_host_num_cpus_total`
- `falcosecurity_falco_duration_seconds_total`
- `falcosecurity_falco_cpu_usage_ratio` (broken given we don't initialize some of the above info)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explanation is interesting but for now I would simply indicate that, as of 0.38.0, it is not possible to run plugins without syscalls and use Prometheus metrics. You can link the issue if people need more details.


## Breaking Changes

{{% pageinfo color=info %}}
Expand All @@ -1268,3 +1288,5 @@ To ensure long-term consistency and validity, we have renamed the following metr
- `memory_vsz` -> `memory_vsz_kb` (or `memory_vsz_mb`)
- `container_memory_used` -> `container_memory_used_bytes` (or `container_memory_used_mb`)
{{% /pageinfo %}}

Near-term improvements are tracked in the following [issue](https://github.com/falcosecurity/falco/issues/3194#issuecomment-2111009270).