Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bug: random fatal error: concurrent map read and map write with ethtool collector #2288

Closed
jmenan opened this issue Feb 11, 2022 · 2 comments
Closed

bug: random fatal error: concurrent map read and map write with ethtool collector #2288

jmenan opened this issue Feb 11, 2022 · 2 comments

Comments

@jmenan
Copy link

jmenan commented Feb 11, 2022 •
edited
Loading

Host operating system: output of uname -a

K8S cluster (1.18)
Host :

Linux ip-xx.xx.xx.xx 5.11.0-1028-aws #31~20.04.1-Ubuntu SMP Fri Jan 14 14:37:50 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

"Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"

node_exporter command line flags

    Args:
      --path.procfs=/host/proc
      --path.sysfs=/host/sys
      --path.rootfs=/host/root
      --web.listen-address=$(HOST_IP):9100
      --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
      --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
      --collector.processes
      --collector.ethtool

Are you running node_exporter in Docker?

yes

What did you do that produced an error?

restart pod node-exporter

What did you expect to see?

pod start without error

What did you see instead?

pods crash randomly wit this error.
maybe the ethtool collector have the same bug that the netdev collector describe here ?

2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.921Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.921Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)"
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.921Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.921Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=arp
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=bcache
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=bonding
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=btrfs
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=conntrack
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=cpu
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=cpufreq
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=diskstats
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=dmi
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=edac
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=entropy
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=ethtool
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=fibrechannel
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=filefd
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=filesystem
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=hwmon
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=infiniband
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=ipvs
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=loadavg
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=mdadm
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=meminfo
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=netclass
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=netdev
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=netstat
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=nfs
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=nfsd
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=nvme
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=os
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=powersupplyclass
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=pressure
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=processes
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=rapl
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=schedstat
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=sockstat
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=softnet
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=stat
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=tapestats
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=textfile
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=thermal_zone
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=time
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=timex
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=udp_queues
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=uname
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=vmstat
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=xfs
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:115 level=info collector=zfs
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=node_exporter.go:199 level=info msg="Listening on" address=0.0.0.0:9100
2022-02-11T11:06:09+01:00	ts=2022-02-11T10:06:09.922Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.917Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)"
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.917Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)"
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:108 level=info msg="Enabled collectors"
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=arp
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=bcache
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=bonding
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=btrfs
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=conntrack
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=cpu
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=cpufreq
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=diskstats
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=dmi
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=edac
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=entropy
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=ethtool
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=fibrechannel
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=filefd
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=filesystem
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=hwmon
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=infiniband
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=ipvs
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=loadavg
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=mdadm
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=meminfo
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=netclass
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=netdev
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=netstat
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=nfs
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=nfsd
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=nvme
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=os
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=powersupplyclass
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=pressure
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=processes
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=rapl
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=schedstat
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=sockstat
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=softnet
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=stat
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=tapestats
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=textfile
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=thermal_zone
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=time
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=timex
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=udp_queues
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=uname
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=vmstat
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=xfs
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.918Z caller=node_exporter.go:115 level=info collector=zfs
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.919Z caller=node_exporter.go:199 level=info msg="Listening on" address=0.0.0.0:9100
2022-02-11T11:05:29+01:00	ts=2022-02-11T10:05:29.919Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
2022-02-11T11:05:40+01:00	fatal error: concurrent map read and map write
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 150 [running]:
2022-02-11T11:05:40+01:00	runtime.throw({0xb2cbbf, 0xc000761201})
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000cff698 sp=0xc000cff668 pc=0x435511
2022-02-11T11:05:40+01:00	runtime.mapaccess2_faststr(0xa5a440, 0xc0003a1500, {0xc0006acf78, 0x14})
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/map_faststr.go:116 +0x3d4 fp=0xc000cff700 sp=0xc000cff698 pc=0x412a14
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*ethtoolCollector).Update(0xc00024cf60, 0xc0003a8180)
2022-02-11T11:05:40+01:00		/app/collector/ethtool_linux.go:423 +0x2158 fp=0xc000cffe30 sp=0xc000cff700 pc=0x97d4d8
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.execute({0xb13568, 0x0}, {0xbfa800, 0xc00024cf60}, 0x0, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:161 +0x9c fp=0xc000cfff60 sp=0xc000cffe30 pc=0x969cfc
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb13568, 0x0}, {0xbfa800, 0xc00024cf60})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:152 +0x3d fp=0xc000cfffb0 sp=0xc000cfff60 pc=0x969bfd
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect·dwrap·3()
2022-02-11T11:05:40+01:00		/app/collector/collector.go:154 +0x3a fp=0xc000cfffe0 sp=0xc000cfffb0 pc=0x969b9a
2022-02-11T11:05:40+01:00	runtime.goexit()
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc000cfffe8 sp=0xc000cfffe0 pc=0x467b21
2022-02-11T11:05:40+01:00	created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2022-02-11T11:05:40+01:00		/app/collector/collector.go:151 +0xd5
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 1 [IO wait]:
2022-02-11T11:05:40+01:00	internal/poll.runtime_pollWait(0x7f55f9891198, 0x72)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/netpoll.go:234 +0x89
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).wait(0xc00035cc80, 0x416f26, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).waitRead(...)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2022-02-11T11:05:40+01:00	internal/poll.(*FD).Accept(0xc00035cc80)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_unix.go:402 +0x22c
2022-02-11T11:05:40+01:00	net.(*netFD).accept(0xc00035cc80)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/fd_unix.go:173 +0x35
2022-02-11T11:05:40+01:00	net.(*TCPListener).accept(0xc00034de18)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/tcpsock_posix.go:140 +0x28
2022-02-11T11:05:40+01:00	net.(*TCPListener).Accept(0xc00034de18)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/tcpsock.go:262 +0x3d
2022-02-11T11:05:40+01:00	net/http.(*Server).Serve(0xc000368000, {0xc06e90, 0xc00034de18})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:3001 +0x394
2022-02-11T11:05:40+01:00	github.com/prometheus/exporter-toolkit/web.Serve({0xc06e90, 0xc00034de18}, 0xc000368000, {0x0, 0xe0}, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.7.0/web/tls_config.go:196 +0x17f
2022-02-11T11:05:40+01:00	github.com/prometheus/exporter-toolkit/web.ListenAndServe(0xc000368000, {0x0, 0x0}, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/exporter-toolkit@v0.7.0/web/tls_config.go:188 +0xf9
2022-02-11T11:05:40+01:00	main.main()
2022-02-11T11:05:40+01:00		/app/node_exporter.go:201 +0x137b
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 55 [runnable]:
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00026fe50)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:513 +0x945
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc000343c80, 0x2, 0x400})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:719 +0x217
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1({0x7f55f9840280, 0xc000134190}, 0xc00013c000)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/http.go:126 +0x108
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0x9e5188, {0x7f55f9840280, 0xc000134190}, 0xa75c80)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f55f9840280, 0xc000134190}, 0xc000134190)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:40 +0xd4
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0xc070a0, {0x7f55f9840280, 0xc000134190}, 0x40)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xc070a0, 0xc000148000}, 0xc00013c000)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101 +0x92
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0xc000134140, {0xc070a0, 0xc000148000}, 0x2)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	main.(*handler).ServeHTTP(0xc0000b7d40, {0xc070a0, 0xc000148000}, 0xc00013c000)
2022-02-11T11:05:40+01:00		/app/node_exporter.go:80 +0x3f2
2022-02-11T11:05:40+01:00	net/http.(*ServeMux).ServeHTTP(0x0, {0xc070a0, 0xc000148000}, 0xc00013c000)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2424 +0x149
2022-02-11T11:05:40+01:00	net/http.serverHandler.ServeHTTP({0xc000122060}, {0xc070a0, 0xc000148000}, 0xc00013c000)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2878 +0x43b
2022-02-11T11:05:40+01:00	net/http.(*conn).serve(0xc000233ae0, {0xc0bfb8, 0xc0003609c0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:1929 +0xb08
2022-02-11T11:05:40+01:00	created by net/http.(*Server).Serve
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:3033 +0x4e8
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 36 [semacquire]:
2022-02-11T11:05:40+01:00	sync.runtime_Semacquire(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/sema.go:56 +0x25
2022-02-11T11:05:40+01:00	sync.(*WaitGroup).Wait(0xc000072e90)
2022-02-11T11:05:40+01:00		/usr/local/go/src/sync/waitgroup.go:130 +0x71
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc000234cc0, {0xbfa3c0, 0xc0000b7c00}}, 0xc00024db60)
2022-02-11T11:05:40+01:00		/app/collector/collector.go:156 +0x254
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:446 +0x102
2022-02-11T11:05:40+01:00	created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:538 +0xb4d
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 20 [IO wait]:
2022-02-11T11:05:40+01:00	internal/poll.runtime_pollWait(0x7f55f98910b0, 0x72)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/netpoll.go:234 +0x89
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).wait(0xc00035ce80, 0xc000122071, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).waitRead(...)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2022-02-11T11:05:40+01:00	internal/poll.(*FD).Read(0xc00035ce80, {0xc000122071, 0x1, 0x1})
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
2022-02-11T11:05:40+01:00	net.(*netFD).Read(0xc00035ce80, {0xc000122071, 0x0, 0x0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/fd_posix.go:56 +0x29
2022-02-11T11:05:40+01:00	net.(*conn).Read(0xc00000eba8, {0xc000122071, 0x0, 0x0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/net.go:183 +0x45
2022-02-11T11:05:40+01:00	net/http.(*connReader).backgroundRead(0xc000122060)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:672 +0x3f
2022-02-11T11:05:40+01:00	created by net/http.(*connReader).startBackgroundRead
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:668 +0xcf
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 35 [semacquire]:
2022-02-11T11:05:40+01:00	sync.runtime_Semacquire(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/sema.go:56 +0x25
2022-02-11T11:05:40+01:00	sync.(*WaitGroup).Wait(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/sync/waitgroup.go:130 +0x71
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2()
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:463 +0x2f
2022-02-11T11:05:40+01:00	created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:462 +0x56f
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 57 [select]:
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc00026fe50)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:513 +0x945
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.Gatherers.Gather({0xc000343c80, 0x2, 0x2})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:719 +0x217
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1({0x7f55f9840280, 0xc0002861e0}, 0xc00013c100)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/http.go:126 +0x108
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0xbfbe60, {0x7f55f9840280, 0xc0002861e0}, 0xc000414150)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f55f9840280, 0xc0002861e0}, 0xc0002861e0)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:40 +0xd4
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0xc070a0, {0x7f55f9840280, 0xc0002861e0}, 0x40)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0xc070a0, 0xc0003682a0}, 0xc00013c100)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/promhttp/instrument_server.go:101 +0x92
2022-02-11T11:05:40+01:00	net/http.HandlerFunc.ServeHTTP(0xc000286190, {0xc070a0, 0xc0003682a0}, 0x2)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2046 +0x2f
2022-02-11T11:05:40+01:00	main.(*handler).ServeHTTP(0xc0000b7d40, {0xc070a0, 0xc0003682a0}, 0xc00013c100)
2022-02-11T11:05:40+01:00		/app/node_exporter.go:80 +0x3f2
2022-02-11T11:05:40+01:00	net/http.(*ServeMux).ServeHTTP(0x0, {0xc070a0, 0xc0003682a0}, 0xc00013c100)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2424 +0x149
2022-02-11T11:05:40+01:00	net/http.serverHandler.ServeHTTP({0xc000414030}, {0xc070a0, 0xc0003682a0}, 0xc00013c100)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:2878 +0x43b
2022-02-11T11:05:40+01:00	net/http.(*conn).serve(0xc000233b80, {0xc0bfb8, 0xc0003609c0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:1929 +0xb08
2022-02-11T11:05:40+01:00	created by net/http.(*Server).Serve
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:3033 +0x4e8
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 65 [IO wait]:
2022-02-11T11:05:40+01:00	internal/poll.runtime_pollWait(0x7f55f9890fc8, 0x72)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/netpoll.go:234 +0x89
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).wait(0xc00035cf00, 0xc000414041, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
2022-02-11T11:05:40+01:00	internal/poll.(*pollDesc).waitRead(...)
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
2022-02-11T11:05:40+01:00	internal/poll.(*FD).Read(0xc00035cf00, {0xc000414041, 0x1, 0x1})
2022-02-11T11:05:40+01:00		/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
2022-02-11T11:05:40+01:00	net.(*netFD).Read(0xc00035cf00, {0xc000414041, 0x0, 0x0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/fd_posix.go:56 +0x29
2022-02-11T11:05:40+01:00	net.(*conn).Read(0xc00000ebc0, {0xc000414041, 0x0, 0x0})
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/net.go:183 +0x45
2022-02-11T11:05:40+01:00	net/http.(*connReader).backgroundRead(0xc000414030)
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:672 +0x3f
2022-02-11T11:05:40+01:00	created by net/http.(*connReader).startBackgroundRead
2022-02-11T11:05:40+01:00		/usr/local/go/src/net/http/server.go:668 +0xcf
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 26 [semacquire]:
2022-02-11T11:05:40+01:00	sync.runtime_Semacquire(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/sema.go:56 +0x25
2022-02-11T11:05:40+01:00	sync.(*WaitGroup).Wait(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/sync/waitgroup.go:130 +0x71
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2()
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:463 +0x2f
2022-02-11T11:05:40+01:00	created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:462 +0x56f
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 27 [semacquire]:
2022-02-11T11:05:40+01:00	sync.runtime_Semacquire(0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/runtime/sema.go:56 +0x25
2022-02-11T11:05:40+01:00	sync.(*WaitGroup).Wait(0xc00016be90)
2022-02-11T11:05:40+01:00		/usr/local/go/src/sync/waitgroup.go:130 +0x71
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect({0xc000234cc0, {0xbfa3c0, 0xc0000b7c00}}, 0xc0003a8180)
2022-02-11T11:05:40+01:00		/app/collector/collector.go:156 +0x254
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:446 +0x102
2022-02-11T11:05:40+01:00	created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/registry.go:538 +0xb4d
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 38 [runnable]:
2022-02-11T11:05:40+01:00	strconv.ParseInt({0xc0004b35a0, 0x5}, 0xa, 0x40)
2022-02-11T11:05:40+01:00		/usr/local/go/src/strconv/atoi.go:184 +0x405
2022-02-11T11:05:40+01:00	github.com/prometheus/procfs.FS.AllProcs({{0xc0005d8840, 0x15}})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/proc.go:113 +0x37e
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).getThreadStates(_, _, {0xadf5, {0xc0004b3510, 0xf}, {0x1004a78, 0x1}, 0x998, 0xadf5, 0x998, ...}, ...)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:161 +0x546
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).getAllocatedThreads(0xc00026e820)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:142 +0x479
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).Update(0xc00026e820, 0xa6ab00)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:88 +0x3e
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.execute({0xb15b90, 0xc000132240}, {0xbfab80, 0xc00026e820}, 0xc0001322a0, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:161 +0x9c
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb15b90, 0xc00006d790}, {0xbfab80, 0xc00026e820})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:152 +0x3d
2022-02-11T11:05:40+01:00	created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2022-02-11T11:05:40+01:00		/app/collector/collector.go:151 +0xd5
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 100 [runnable]:
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.NewConstMetric(0xc00032ce00, 0x3, 0x0, {0xc0003124e0, 0x1, 0x1})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/value.go:87 +0x171
2022-02-11T11:05:40+01:00	github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/value.go:105
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*ethtoolCollector).Update(0xc00024cf60, 0xc00024db60)
2022-02-11T11:05:40+01:00		/app/collector/ethtool_linux.go:432 +0x2334
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.execute({0xb13568, 0x0}, {0xbfa800, 0xc00024cf60}, 0x0, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:161 +0x9c
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb13568, 0x0}, {0xbfa800, 0xc00024cf60})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:152 +0x3d
2022-02-11T11:05:40+01:00	created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2022-02-11T11:05:40+01:00		/app/collector/collector.go:151 +0xd5
2022-02-11T11:05:40+01:00
2022-02-11T11:05:40+01:00	goroutine 166 [runnable]:
2022-02-11T11:05:40+01:00	syscall.Syscall6(0x101, 0xffffffffffffff9c, 0xc0002a8060, 0x80000, 0x0, 0x0, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/syscall/asm_linux_amd64.s:43 +0x5
2022-02-11T11:05:40+01:00	syscall.openat(0x0, {0xc000523cc0, 0x20}, 0xc000523cc0, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/syscall/zsyscall_linux_amd64.go:69 +0x105
2022-02-11T11:05:40+01:00	syscall.Open(...)
2022-02-11T11:05:40+01:00		/usr/local/go/src/syscall/syscall_linux.go:155
2022-02-11T11:05:40+01:00	os.openFileNolog({0xc000523cc0, 0xbf1868}, 0x0, 0x0)
2022-02-11T11:05:40+01:00		/usr/local/go/src/os/file_unix.go:217 +0x9b
2022-02-11T11:05:40+01:00	os.OpenFile({0xc000523cc0, 0x20}, 0x0, 0x2)
2022-02-11T11:05:40+01:00		/usr/local/go/src/os/file.go:338 +0x45
2022-02-11T11:05:40+01:00	os.Open(...)
2022-02-11T11:05:40+01:00		/usr/local/go/src/os/file.go:318
2022-02-11T11:05:40+01:00	github.com/prometheus/procfs/internal/util.ReadFileNoStat({0xc000523cc0, 0xc000726918})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/internal/util/readfile.go:30 +0x50
2022-02-11T11:05:40+01:00	github.com/prometheus/procfs.Proc.Stat({_, {_, _}})
2022-02-11T11:05:40+01:00		/go/pkg/mod/github.com/prometheus/procfs@v0.7.4-0.20211011103944-1a7a2bd3279f/proc_stat.go:125 +0xb1
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).getThreadStates(_, _, {0xadf5, {0xc000794800, 0xf}, {0x1004a78, 0x1}, 0x998, 0xadf5, 0x998, ...}, ...)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:175 +0x8a6
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).getAllocatedThreads(0xc00026e820)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:142 +0x479
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.(*processCollector).Update(0xc00026e820, 0x0)
2022-02-11T11:05:40+01:00		/app/collector/processes_linux.go:88 +0x3e
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.execute({0xb15b90, 0x0}, {0xbfab80, 0xc00026e820}, 0x0, {0xbfa3c0, 0xc0000b7c00})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:161 +0x9c
2022-02-11T11:05:40+01:00	github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1({0xb15b90, 0x0}, {0xbfab80, 0xc00026e820})
2022-02-11T11:05:40+01:00		/app/collector/collector.go:152 +0x3d
2022-02-11T11:05:40+01:00	created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
2022-02-11T11:05:40+01:00		/app/collector/collector.go:151 +0xd5
@tanguyfalconnet tanguyfalconnet mentioned this issue Feb 11, 2022
Merged
@hbokh
Copy link
Contributor

hbokh commented Jul 7, 2022

Any update on this bug? Running into the same issue.

@ventifus
Copy link
Contributor

ventifus commented Jul 7, 2022

The fix has been merged, now we need to tag a new release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@discordianfish @hbokh @ventifus @jmenan