hostmetrics receiver logs spurious errors when it races with process termination

[ringerc]

Describe the bug

If a process terminates midway through a hostmetrics process scrape, the scraper will log a spurious error like

{"level":"error","ts":1704887224.2511945,"caller":"scraperhelper/scrapercontroller.go:200","msg":"Error scraping metrics","kind":"receiver","name":"hostmetrics","data_type":"metrics","error":"error reading cpu times for process \"postgres\" (pid 1965300): open /host/proc/1965300/stat: no such file or directory; error reading memory info for process \"postgres\" (pid 1965300): open /host/proc/1965300/statm: no such file or directory","scraper":"process","stacktrace":"go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport\n\tgo.opentelemetry.io/collector/receiver@v0.90.1/scraperhelper/scrapercontroller.go:200\ngo.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1\n\tgo.opentelemetry.io/collector/receiver@v0.90.1/scraperhelper/scrapercontroller.go:176"}

because /proc/1965300 existed when it listed /proc, but had vanished by the time it tried to read its contents.

This error is expected and can be safely silenced. It'd make sense to stat the directory on I/O error, and if it's ENOENT, suppress the error. Or just ignore ENOENT for subdirs entirely since that's the only likely cause for this error.

Steps to reproduce

Run a workload that creates and terminates lots of processes, while running hostmetrics process receiver.

What did you expect to see?

No error level logs.

What did you see instead?

error level logs about failure to read procfs entries for processes that vanished during the scrape.

What version did you use?

v0.91.0

What config did you use?

A generic sample config with the hostmetrics receiver enabled.

Environment

Generic k8s (kind)

[github-actions]

Pinging code owners for receiver/hostmetrics: @dmitryax @braydonk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

Thanks for the report @ringerc - would you like to try and offer a fix for this issue?

[ringerc]

@atoulme I'd be happy to give it a go though it'll be a while before I can queue it up. In the meantime hopefully others will see this and at least know what the errors are from / why they appear.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/hostmetrics: @dmitryax @braydonk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.