Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exporter may freeze if network file system is unavailable #3058

Open
vitoyucepi opened this issue Jun 21, 2024 · 4 comments
Open

Exporter may freeze if network file system is unavailable #3058

vitoyucepi opened this issue Jun 21, 2024 · 4 comments

Comments

@vitoyucepi
Copy link

Host operating system: output of uname -a

Linux vm-ubuntu-22 5.15.0-1060-kvm #65-Ubuntu SMP Tue May 21 09:31:15 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.1 (branch: HEAD, revision: 400c3979931613db930ea035f39ce7b377cdbb5b)
  build user:       root@7afbff271a3f
  build date:       20240521-18:36:22
  go version:       go1.22.3
  platform:         linux/amd64
  tags:             unknown

node_exporter command line flags

--collector.filesystem.mount-timeout=1s
--log.level=debug

node_exporter log output

Log 
ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c3979931613db930ea035f39ce7b377cdbb5b)"
ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/amd64, user=root@7afbff271a3f, date=20240521-18:36:22, tags=unknown)"
ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:196 level=warn msg="Node Exporter is running as root user. This exporter is designed to run as unprivileged user, root is not required."
ts=2024-06-21T20:33:15.058Z caller=node_exporter.go:199 level=debug msg="Go MAXPROCS" procs=1
ts=2024-06-21T20:33:15.059Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-06-21T20:33:15.059Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-06-21T20:33:15.059Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-06-21T20:33:15.059Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=os
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=time
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-06-21T20:33:15.060Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-06-21T20:33:15.060Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-06-21T20:33:15.060Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100
ts=2024-06-21T20:33:19.760Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type"
ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.77
ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.37
ts=2024-06-21T20:33:19.761Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.14
ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=8.0849e-05
ts=2024-06-21T20:33:19.761Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded"
ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=2.6859e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=8.222e-05
ts=2024-06-21T20:33:19.761Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping"
ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=3.0973e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.761Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system"
ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=3.3635e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.761Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping"
ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=1.9921e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.761Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system"
ts=2024-06-21T20:33:19.761Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=1.4036e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu
ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io
ts=2024-06-21T20:33:19.761Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory
ts=2024-06-21T20:33:19.761Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.000140935
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.000456171
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=0.000107346
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=9.826e-05
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=9.995e-06
ts=2024-06-21T20:33:19.762Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping"
ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=2.6714e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.762Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory"
ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=3.9654e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000176827
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=8.7765e-05
ts=2024-06-21T20:33:19.762Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=1.7034e-05
ts=2024-06-21T20:33:19.762Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory"
ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=3.8249e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.762Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory"
ts=2024-06-21T20:33:19.762Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=1.9554e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.763Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.7190019997630088e+09
ts=2024-06-21T20:33:19.763Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC
ts=2024-06-21T20:33:19.763Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]"
ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.000139335
ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=3.3409e-05
ts=2024-06-21T20:33:19.763Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=1.7618e-05
ts=2024-06-21T20:33:19.763Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist"
ts=2024-06-21T20:33:19.763Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=2.2134e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.001059162
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=1.0888e-05
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=7.236e-06
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=3.8592e-05
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.00013394
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=2.6597e-05
ts=2024-06-21T20:33:19.764Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping"
ts=2024-06-21T20:33:19.764Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=1.7543e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=4.906e-05
ts=2024-06-21T20:33:19.764Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000158008
ts=2024-06-21T20:33:19.765Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs
ts=2024-06-21T20:33:19.765Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
ts=2024-06-21T20:33:19.765Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=3.2581e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.765Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.000605296
ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:19.765Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:19.766Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.000673482
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=8.0562e-05
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=7.1122e-05
ts=2024-06-21T20:33:19.766Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc
ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=3.6268e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=7.5467e-05
ts=2024-06-21T20:33:19.766Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory"
ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=2.4558e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.766Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type"
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.000131379
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=6.625e-06
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=2.6535e-05
ts=2024-06-21T20:33:19.766Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net
ts=2024-06-21T20:33:19.766Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=3.2812e-05 err="collector returned no data"
ts=2024-06-21T20:33:19.766Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=9.32e-06
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config
ts=2024-06-21T20:33:19.767Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service
ts=2024-06-21T20:33:19.768Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=0.004034153
ts=2024-06-21T20:33:21.538Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type"
ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.77
ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.37
ts=2024-06-21T20:33:21.540Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.14
ts=2024-06-21T20:33:21.540Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=0.000301563
ts=2024-06-21T20:33:21.540Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system"
ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=0.000116747 err="collector returned no data"
ts=2024-06-21T20:33:21.541Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping"
ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=9.6302e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.541Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system"
ts=2024-06-21T20:33:21.541Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=7.0611e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu
ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io
ts=2024-06-21T20:33:21.541Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory
ts=2024-06-21T20:33:21.542Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.000722049
ts=2024-06-21T20:33:21.543Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.001284556
ts=2024-06-21T20:33:21.543Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded"
ts=2024-06-21T20:33:21.543Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=9.2716e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.543Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=0.000272085
ts=2024-06-21T20:33:21.543Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping"
ts=2024-06-21T20:33:21.544Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=8.9869e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.544Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping"
ts=2024-06-21T20:33:21.544Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=0.000108483 err="collector returned no data"
ts=2024-06-21T20:33:21.544Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=0.000421419
ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=0.000415263
ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=3.7784e-05
ts=2024-06-21T20:33:21.545Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=8.8071e-05
ts=2024-06-21T20:33:21.545Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory"
ts=2024-06-21T20:33:21.545Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=0.000139228 err="collector returned no data"
ts=2024-06-21T20:33:21.545Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory"
ts=2024-06-21T20:33:21.545Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=8.7256e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000542391
ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=0.000276705
ts=2024-06-21T20:33:21.546Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=3.9723e-05
ts=2024-06-21T20:33:21.546Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist"
ts=2024-06-21T20:33:21.546Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=5.7399e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.549Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.003016928
ts=2024-06-21T20:33:21.549Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=2.9824e-05
ts=2024-06-21T20:33:21.550Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory"
ts=2024-06-21T20:33:21.550Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=6.325e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.550Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.719002001550096e+09
ts=2024-06-21T20:33:21.550Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC
ts=2024-06-21T20:33:21.550Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]"
ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.00028476
ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=2.035e-06
ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.000345937
ts=2024-06-21T20:33:21.550Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=7.756e-05
ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=3.8736e-05
ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=8.0736e-05
ts=2024-06-21T20:33:21.551Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs
ts=2024-06-21T20:33:21.551Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
ts=2024-06-21T20:33:21.551Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=6.4907e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.551Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping"
ts=2024-06-21T20:33:21.551Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=3.1584e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=0.00010089
ts=2024-06-21T20:33:21.551Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000339368
ts=2024-06-21T20:33:21.552Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=0.00015717
ts=2024-06-21T20:33:21.553Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.001230999
ts=2024-06-21T20:33:21.553Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:21.553Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:21.554Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:21.554Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=1.5636e-05
ts=2024-06-21T20:33:21.554Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net
ts=2024-06-21T20:33:21.554Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=5.8544e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.554Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=1.7544e-05
ts=2024-06-21T20:33:21.555Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc
ts=2024-06-21T20:33:21.555Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory"
ts=2024-06-21T20:33:21.555Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=5.0574e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:21.555Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:21.555Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.002288325
ts=2024-06-21T20:33:21.556Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=0.001697304
ts=2024-06-21T20:33:21.556Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type"
ts=2024-06-21T20:33:21.556Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.001886767
ts=2024-06-21T20:33:21.558Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=0.004203471
ts=2024-06-21T20:33:21.559Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=7.3392e-05 err="collector returned no data"
ts=2024-06-21T20:33:21.559Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=0.004094869
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config
ts=2024-06-21T20:33:21.559Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service
ts=2024-06-21T20:33:21.560Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=0.009668599
ts=2024-06-21T20:33:34.805Z caller=node_exporter.go:79 level=debug msg="collect query:" filters="unsupported value type"
ts=2024-06-21T20:33:34.806Z caller=collector.go:173 level=debug msg="collector succeeded" name=filefd duration_seconds=3.5575e-05
ts=2024-06-21T20:33:34.806Z caller=tapestats_linux.go:129 level=debug collector=tapestats msg="scsi_tape stats not found, skipping"
ts=2024-06-21T20:33:34.806Z caller=collector.go:167 level=debug msg="collector returned no data" name=tapestats duration_seconds=0.000175007 err="collector returned no data"
ts=2024-06-21T20:33:34.806Z caller=collector.go:173 level=debug msg="collector succeeded" name=vmstat duration_seconds=6.5867e-05
ts=2024-06-21T20:33:34.807Z caller=collector.go:173 level=debug msg="collector succeeded" name=diskstats duration_seconds=0.000210497
ts=2024-06-21T20:33:34.807Z caller=zfs_linux.go:53 level=debug collector=zfs msg="Cannot open file for reading" path=/proc/spl/kstat/zfs
ts=2024-06-21T20:33:34.807Z caller=zfs.go:73 level=debug collector=zfs err="ZFS / ZFS statistics are not available"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=zfs duration_seconds=0.003866769 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=netstat duration_seconds=0.000544537
ts=2024-06-21T20:33:34.808Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu0
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu1
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu2
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu3
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu4
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:295 level=debug collector=cpu msg="CPU is missing core_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:34.811Z caller=cpu_linux.go:305 level=debug collector=cpu msg="CPU is missing package_throttle_count" cpu=/sys/devices/system/cpu/cpu5
ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpu duration_seconds=0.003408076
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=cpufreq duration_seconds=5.085e-05
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=softnet duration_seconds=4.6219e-05
ts=2024-06-21T20:33:34.808Z caller=mdadm_linux.go:115 level=debug collector=mdadm msg="Not collecting mdstat, file does not exist" file=/proc
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=mdadm duration_seconds=0.003146689 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=entropy duration_seconds=5.3531e-05
ts=2024-06-21T20:33:34.808Z caller=nfs_linux.go:103 level=debug collector=nfs msg="Not collecting NFS metrics" err="open /proc/net/rpc/nfs: no such file or directory"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfs duration_seconds=0.003082385 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=meminfo.go:54 level=debug collector=meminfo msg="Set node_mem" memInfo="unsupported value type"
ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=meminfo duration_seconds=0.003121106
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=uname duration_seconds=3.816e-06
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=powersupplyclass duration_seconds=2.3564e-05
ts=2024-06-21T20:33:34.808Z caller=bonding_linux.go:64 level=debug collector=bonding msg="Not collecting bonding, file does not exist" file=/sys/class/net
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=bonding duration_seconds=0.003065661 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=dmi duration_seconds=3.749e-06
ts=2024-06-21T20:33:34.808Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=0 load=0.68
ts=2024-06-21T20:33:34.811Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=1 load=0.36
ts=2024-06-21T20:33:34.811Z caller=loadavg.go:55 level=debug collector=loadavg msg="return load" index=2 load=0.15
ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=loadavg duration_seconds=0.003108427
ts=2024-06-21T20:33:34.808Z caller=conntrack_linux.go:157 level=debug collector=conntrack msg="conntrack probably not loaded"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=conntrack duration_seconds=0.003101571 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=collector.go:173 level=debug msg="collector succeeded" name=udp_queues duration_seconds=5.2847e-05
ts=2024-06-21T20:33:34.808Z caller=nvme_linux.go:56 level=debug collector=nvme msg="nvme statistics not found, skipping"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=nvme duration_seconds=0.003043684 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=ipvs_linux.go:146 level=debug collector=ipvs msg="ipvs collector metrics are not available for this system"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=ipvs duration_seconds=0.003044284 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=fibrechannel_linux.go:101 level=debug collector=fibrechannel msg="fibrechannel statistics not found, skipping"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=fibrechannel duration_seconds=0.003022354 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=hwmon_linux.go:440 level=debug collector=hwmon msg="hwmon collector metrics are not available for this system"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=hwmon duration_seconds=0.003022138 err="collector returned no data"
ts=2024-06-21T20:33:34.808Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=cpu
ts=2024-06-21T20:33:34.811Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=io
ts=2024-06-21T20:33:34.811Z caller=pressure_linux.go:92 level=debug collector=pressure msg="collecting statistics for resource" resource=memory
ts=2024-06-21T20:33:34.811Z caller=collector.go:173 level=debug msg="collector succeeded" name=pressure duration_seconds=0.003136247
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=thermal_zone duration_seconds=0.000272793
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=stat duration_seconds=9.795e-05
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=sockstat duration_seconds=8.6614e-05
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=bcache duration_seconds=6.062e-06
ts=2024-06-21T20:33:34.809Z caller=infiniband_linux.go:119 level=debug collector=infiniband msg="infiniband statistics not found, skipping"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=infiniband duration_seconds=0.00269225 err="collector returned no data"
ts=2024-06-21T20:33:34.809Z caller=rapl_linux.go:77 level=debug collector=rapl msg="Platform doesn't have powercap files present" err="unable to read class/powercap: open /sys/class/powercap: no such file or directory"
ts=2024-06-21T20:33:34.811Z caller=collector.go:167 level=debug msg="collector returned no data" name=rapl duration_seconds=0.002685124 err="collector returned no data"
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=netdev duration_seconds=0.000134776
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=btrfs duration_seconds=0.000104789
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=timex duration_seconds=1.2511e-05
ts=2024-06-21T20:33:34.809Z caller=nfsd_linux.go:69 level=debug collector=nfsd msg="Not collecting NFSd metrics" err="open /proc/net/rpc/nfsd: no such file or directory"
ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=nfsd duration_seconds=0.002529992 err="collector returned no data"
ts=2024-06-21T20:33:34.809Z caller=watchdog.go:106 level=debug collector=watchdog msg="Could not read watchdog stats" err="failed to list watchdog devices at \"/sys/class/watchdog\": open /sys/class/watchdog: no such file or directory"
ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=watchdog duration_seconds=0.002531731 err="collector returned no data"
ts=2024-06-21T20:33:34.809Z caller=time.go:73 level=debug collector=time msg="Return time" now=1.719002014809634e+09
ts=2024-06-21T20:33:34.812Z caller=time.go:75 level=debug collector=time msg="Zone offset" offset=0 time_zone=UTC
ts=2024-06-21T20:33:34.812Z caller=time_linux.go:38 level=debug collector=time msg="in Update" clocksources="[{0 [kvm-clock tsc] kvm-clock}]"
ts=2024-06-21T20:33:34.812Z caller=collector.go:173 level=debug msg="collector succeeded" name=time duration_seconds=0.002609163
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=selinux duration_seconds=9.66e-07
ts=2024-06-21T20:33:34.809Z caller=collector.go:173 level=debug msg="collector succeeded" name=textfile duration_seconds=1.2051e-05
ts=2024-06-21T20:33:34.809Z caller=schedstat_linux.go:78 level=debug collector=schedstat msg="schedstat file does not exist"
ts=2024-06-21T20:33:34.812Z caller=collector.go:167 level=debug msg="collector returned no data" name=schedstat duration_seconds=0.002609169 err="collector returned no data"
ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=netclass duration_seconds=0.000960856
ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=xfs duration_seconds=4.928e-06
ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=edac duration_seconds=3.48e-06
ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=os duration_seconds=3.6102e-05
ts=2024-06-21T20:33:34.810Z caller=collector.go:173 level=debug msg="collector succeeded" name=arp duration_seconds=0.000113626
ts=2024-06-21T20:33:34.811Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/proc
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/security
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/shm
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/pts
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/cgroup
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/bpf
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/hugepages
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/dev/mqueue
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/debug
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/tracing
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/fs/fuse/connections
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/sys/kernel/config
ts=2024-06-21T20:33:34.812Z caller=filesystem_linux.go:78 level=debug collector=filesystem msg="Ignoring mount point" mountpoint=/run/credentials/systemd-sysusers.service
ts=2024-06-21T20:33:35.814Z caller=filesystem_linux.go:173 level=debug collector=filesystem msg="Mount point timed out, it is being labeled as stuck and will not be monitored" mountpoint=/media/test1
ts=2024-06-21T20:33:45.033Z caller=filesystem_linux.go:131 level=debug collector=filesystem msg="Mount point has recovered, monitoring will resume" mountpoint=/media/test1
ts=2024-06-21T20:33:45.033Z caller=filesystem_linux.go:138 level=debug collector=filesystem msg="Error on statfs() system call" rootfs=/media/test1 err="host is down"
ts=2024-06-21T20:33:45.033Z caller=collector.go:173 level=debug msg="collector succeeded" name=filesystem duration_seconds=10.223095789

Are you running node_exporter in Docker?

No.

What did you do that produced an error?

  1. Set up two VMs. The first one will mount the file system of the second one.
  2. Configure samba on the second VM.
  3. Configure fstab on the first VM. 
    //192.168.0.1/debian	/media/test1	cifs	defaults,_netdev,username=debian,password=debian	0 0
  4. mkdir /media/test1 on the first VM.
  5. mount /media/test1 on the first VM.
  6. Start node exporter on the first VM.
  7. Stop the second VM.
  8. Run time curl -s http://localhost:9100/metrics | grep node_scrape_collector_duration_seconds.

If you don't like cifs, replace it with sshfs.

What did you expect to see?

The exporter should respond in a about 1 second.
node_scrape_collector_duration_seconds{collector="filesystem"} 1.20955266

What did you see instead?

The exporter may freeze for an unknown amount of time, but may respond in 10 seconds.
@chengjoey
Copy link
Contributor

chengjoey commented Jun 27, 2024 •
edited
Loading

look issue #2903, this seems to be the expected behavior

chengjoey added a commit to chengjoey/node_exporter that referenced this issue Jun 27, 2024
@chengjoey
fix filesystem mountTimeout not working, it also ensures that stuckMo…
761aff2 
…unt can be deleted after it is successfully executed in goroutine. prometheus#2903 prometheus#3058

Signed-off-by: joey <zchengjoey@gmail.com>
@chengjoey chengjoey mentioned this issue Jun 27, 2024
Open
chengjoey added a commit to chengjoey/node_exporter that referenced this issue Jun 27, 2024
@chengjoey
fix filesystem mountTimeout not working, it also ensures that stuckMo…
8bc1566 
…unt can be deleted after it is successfully executed in goroutine. prometheus#2903 prometheus#3058

Signed-off-by: joey <zchengjoey@gmail.com>
chengjoey added a commit to chengjoey/node_exporter that referenced this issue Jun 27, 2024
@chengjoey
fix filesystem mountTimeout not working, it also ensures that stuckMo…
f7b3413 
…unt can be deleted after it is successfully executed in goroutine. prometheus#2903 prometheus#3058

Signed-off-by: joey <zchengjoey@gmail.com>
@vitoyucepi
Copy link
Author

I'm not sure if this behavior is expected, because in my test case the result can vary from 10.2s to an unknown amount of time, which in both cases is longer than the default timeout for Prometheus.

I discovered this problem after another network outage on my VM. During the outage, local Prometheus didn't collect any node exporter metrics because the scrape_timeout is 10s by default.

@chengjoey
Copy link
Contributor

I'm not sure if this behavior is expected, because in my test case the result can vary from 10.2s to an unknown amount of time, which in both cases is longer than the default timeout for Prometheus.

I discovered this problem after another network outage on my VM. During the outage, local Prometheus didn't collect any node exporter metrics because the scrape_timeout is 10s by default.

see #3063, I tried to fix the timeout not working issue, and #2903 was reverted because it was necessary to ensure that it could be removed from stuckMounts after successful execution.

@vitoyucepi
Copy link
Author

f7b3413 works for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chengjoey @vitoyucepi