vCenter failed to receive VMs metrics panic runtime error invalid memory address or nil pointer dereference

[shacharSirotkin]

Component(s)

vCenter receiver

What happened?

Description

I tried to config a vcenterreciver and got a stack trace of runtime error saying:
panic: runtime error: invalid memory address or nil pointer dereference.

Steps to Reproduce

Install otel-collector-contrib on your PC and use a config.yaml to receive metrics from vCenter in version 7.0.3

Expected Result

normal metrics of vcenter

Actual Result

panic: runtime error: invalid memory address or nil pointer dereference.

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).buildVMMetrics()
.....

Collector version

0.105.0

Environment information

Environment

Windows 10 Enterprise

OpenTelemetry Collector configuration

receivers:
    vecenter/vc1:
        endpoint: https://dev-vc1.my.domain
        username: vm_mgr_svc
        password: **************
        collection_interval: 1m
        tls:
            insecure: false
            insecure_skip_verify: true

exporters:
     file/myfile:
         path: metrics

service:
     pipelines:
          metrics:
               receivers:
                    - vcenter/vc1
               exporters:
                   - file/myfile
      telemetry:
          metrics:
                address: 0.0.0.0:8993

Log output

panic: runtime error: invalid memory address or nil pointer dereference.

github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).buildVMMetrics()
.....

Additional context

No response

[github-actions]

Pinging code owners for receiver/vcenter: @djaglowski @schmikei @StefanKurek. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Thanks for filing @shacharSirotkin, is that the full stack trace from the panic? It would be helpful to have more specific information about where the panic occurred in buildVMMetrics.

[BominRahmani]

@shacharSirotkin Hey! I'd love to take a look at this, if you could please supply the full stack trace from the panic you see.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/vcenter: @djaglowski @schmikei @StefanKurek

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[bmhkb4]

hey ya'll (@BominRahmani), I am not the OP, but I am experiencing the same issue, oddly only in one of my vcenters (assuming that something is unique about this vcenter or the objects located within), here is my full stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0x41d24de]

goroutine 316 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).buildVMMetrics(0xc003df49c0, 0x1809294d4432a316, 0xc002059608, 0xc00697c028, 0xc00659fa60?)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:310 +0x1de
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).processVMs(0xc003df49c0, 0x1809294d4432a316, 0xc002059608, 0xc00593ba70, 0xc00659f850, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:228 +0xfb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).processDatacenterData(0xc003df49c0, 0xc002059608, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:41 +0x25d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).scrapeAndProcessAllMetrics(0xc003df49c0, {0xc0b1150, 0xc003e53260}, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/scraper.go:133 +0x413
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).scrape(0xc003df49c0, {0xc0b1150, 0xc003e53260})
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/scraper.go:112 +0x185
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape(...)
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport(0xc00315bcc0)
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:202 +0x16b
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1()
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:177 +0xd1
created by go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping in goroutine 96
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:163 +0x4f

Let me know what else I can provide, thanks!

[BominRahmani]

hey ya'll (@BominRahmani), I am not the OP, but I am experiencing the same issue, oddly only in one of my vcenters (assuming that something is unique about this vcenter or the objects located within), here is my full stack trace:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0x41d24de]

goroutine 316 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).buildVMMetrics(0xc003df49c0, 0x1809294d4432a316, 0xc002059608, 0xc00697c028, 0xc00659fa60?)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:310 +0x1de
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).processVMs(0xc003df49c0, 0x1809294d4432a316, 0xc002059608, 0xc00593ba70, 0xc00659f850, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:228 +0xfb
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).processDatacenterData(0xc003df49c0, 0xc002059608, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/processors.go:41 +0x25d
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).scrapeAndProcessAllMetrics(0xc003df49c0, {0xc0b1150, 0xc003e53260}, 0xc00659fcd0)
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/scraper.go:133 +0x413
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver.(*vcenterMetricScraper).scrape(0xc003df49c0, {0xc0b1150, 0xc003e53260})
	/go/pkg/mod/github.com/open-telemetry/opentelemetry-collector-contrib/receiver/vcenterreceiver@v0.112.0/scraper.go:112 +0x185
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape(...)
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scraper.go:20
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport(0xc00315bcc0)
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:202 +0x16b
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1()
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:177 +0xd1
created by go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping in goroutine 96
	/go/pkg/mod/go.opentelemetry.io/collector/receiver@v0.112.0/scraperhelper/scrapercontroller.go:163 +0x4f

Let me know what else I can provide, thanks!

@bmhkb4
Im happy to take a look at this for you! Could I ask you to send otel config, specifically the vcenter receiver block? You can redact or delete any of the credentials

[bmhkb4]

logging {
  level  = "debug"
  format = "logfmt"
}

otelcol.receiver.vcenter "tas_vcenter" {
  endpoint = "https://<vcenterurl>"
  username = "username"
  password = "password"
  tls {
    insecure = true
  }
  output {
    metrics = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.prometheus.default.input]
  }
}

otelcol.exporter.prometheus "default" {
  forward_to = [prometheus.remote_write.alloy.receiver]

  resource_to_telemetry_conversion = true
}

prometheus.remote_write "alloy" {
  endpoint {
    url = "https://<url>/api/v1/metrics/write"
  }
}

Through trial and error I was able to find the configuration in vcenter causing the issue:

These vms were orphaned, likely due to a ESX Host crash. After cleaning them up it works fine.

Thanks!