could not set systemd collector for Node Exporter Metrics plugin

[1puls2is3]

Bug Report

Describe the bug
OS: Debian GNU/Linux 12.5 x86_64
K8S: 1.30.1
containerd v1.6.32
runc version 1.1.12

fluent-bit: 3.12 or 3.13 or 2.2.3

helm install -f fluent-bit.yaml fluent-bit fluent/fluent-bit

cat fluent-bit.yaml

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Tag log_containers
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

    [INPUT]
        name            node_exporter_metrics
        tag             metrics_node
        scrape_interval 2

  filters: |
    [FILTER]
        Name kubernetes
        Match .*kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

  outputs: |
    [OUTPUT]
        name                   loki
        host                   loki-gateway.loki.svc.cluster.local
        uri                    /loki/api/v1/push
        port                   80
        tls                    off
        tls.verify             off
        match                  log_.*
        labels                 job=fluentbit
        auto_kubernetes_labels on

    [OUTPUT]
        name        prometheus_remote_write
        host        mimir-nginx.mimir.svc
        match       metrics_.*
        uri         /api/v1/push
        port        80
        tls         off
        tls.verify  on

• The node_exporter_metrics plugin keeps reporting errors :

[ info] [input:node_exporter_metrics:node_exporter_metrics.1] initializing
[ info] [input:node_exporter_metrics:node_exporter_metrics.1] storage_strategy='memory' (memory only)
[ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.procfs = /proc
[ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.sysfs  = /sys
[error] [input:node_exporter_metrics:node_exporter_metrics.1] systemd init failed
[error] [input:node_exporter_metrics:node_exporter_metrics.1] could not set systemd collector for Node Exporter Metrics plugin
[ info] [input:node_exporter_metrics:node_exporter_metrics.1] thread instance initialized

Running fluent-bit 3.1.3 directly on Debian 12.5 x86_64 with the same configuration file, node_exporter_metrics does not report an error, but it does have this message:

[debug] [task] created task=0x7ff653a36c80 id=0 without routes, dropping.
[debug] [task] destroy task=0x7ff653a36c80 (task_id=0)
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] CPU is missing core_throttle_count: /sys/devices/system/cpu/cpu0
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] CPU is missing package_throttle_count: /sys/devices/system/cpu/cpu0
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] CPU is missing core_throttle_count: /sys/devices/system/cpu/cpu1
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] cannot read info from: /class/thermal/thermal_zone[0-9]*
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: sda1
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: sda2
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop0
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop1
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop2
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop3
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop4
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop5
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop6
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] skip device: loop7
[debug] [input:node_exporter_metrics:node_exporter_metrics.1] NVMe storage is not mounted
[debug] [task] created task=0x7ff653a36d20 id=0 without routes, dropping.
[debug] [task] destroy task=0x7ff653a36d20 (task_id=0)

[1puls2is3]

https://docs.fluentbit.io/manual/pipeline/inputs/node-exporter-metrics#fluent-bit--prometheus--grafana
Following the instructions in the link above, I checked the fluent-bit container in my docker environment and got the same error:

[ info] [fluent bit] version=3.1.3, commit=12a9de521c, pid=1
[ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[ info] [cmetrics] version=0.9.1
[ info] [ctraces ] version=0.5.2
[ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /host/proc
[ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /host/sys
[error] [input:node_exporter_metrics:node_exporter_metrics.0] systemd init failed
[error] [input:node_exporter_metrics:node_exporter_metrics.0] could not set systemd collector for Node Exporter Metrics plugin
[ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[ info] [output:prometheus_exporter:prometheus_exporter.0] listening iface=0.0.0.0 tcp_port=2021

I'm guessing it's a problem with the node_exporter_metrics plugin or the mirrors.

Metrics can be collected with this error.

I'm not sure if having this error is causing any other problems.

[Athishpranav2003]

@1puls2is3 I checked this message in code NVMe storage is not mounted . Seems like it could permission issue. Could you please check if Fluentbit has sufficient permissions to access these metrics