Scraping Windows service info crashes Alloy

[splitt3r]

What's wrong?

Grafana Alloy works very smooth over all. Thanks for your work!

I ran into the problems descirbed in https://community.grafana.com/t/windows-exporter-component-crash-when-scrapping-scheduled-tasks/126616 so i upgraded to the latest rc version of Grafana Alloy. In general it works great. But i have problems scraping the windows service information. I couldn't figure out what exactly the problem is. But without enabling the "service" collector everything works fine. If you have any more questions surrounding the setup i'm happy to answer them. Grafana Alloy runs as Windows SYSTEM user while some services run under different Active Directory Accounts. Maybe the problem is related to some access restrictions?

Steps to reproduce

See the configuration blocks below. Grafana Alloy crashes on the first windows integration scrape. So shortly after starting it (as a service / as a binary from cmd).

System information

Windows Server 2022 Standard 21H2

Software version

Grafana Alloy v1.4.0-rc.3

Configuration

The configuration was migrated form the static Grafana Agent configuration.

This works:

prometheus.exporter.windows "integrations_windows_exporter" {
	enabled_collectors = ["cpu","cs","logical_disk","memory","net","os","process","scheduled_task","system"]

	process {
		whitelist = "Test.*"
	}
}

This does not work:

prometheus.exporter.windows "integrations_windows_exporter" {
	enabled_collectors = ["cpu","cs","logical_disk","memory","net","os","process","scheduled_task","service","system"]

	process {
		whitelist = "Test.*"
	}

	service {
		where_clause = "Name='TestService'"
	}
}

I also tried this. This works but it generates a warning (see below):

prometheus.exporter.windows "integrations_windows_exporter" {
	enabled_collectors = ["cpu","cs","logical_disk","memory","net","os","process","scheduled_task","service","system"]

	process {
		whitelist = "Test.*"
	}

	service {
		where_clause          = "Name='TestService'"
		enable_v2_collector   = "true"
		use_api               = "true"
	}
}

Logs

Running Grafana Alloy with "service" collector enabled and only where_clause configured:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x7728d33]

goroutine 896 [running]:
github.com/prometheus-community/windows_exporter/pkg/collector/service.(*Collector).Collect(0xc001c11a90, 0x0?, 0x0?)
        /go/pkg/mod/github.com/prometheus-community/windows_exporter@v0.27.2/pkg/collector/service/service.go:152 +0x33
github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).execute(0xc00306f100, {0xa37368f, 0x7}, {0xba52528, 0xc001c11a90}, 0xc0000e6bb0, 0xc0049db980)
        /go/pkg/mod/github.com/prometheus-community/windows_exporter@v0.27.2/pkg/collector/prometheus.go:176 +0x8f
github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).Collect.func2({0xa37368f, 0x7}, {0xba52528?, 0xc001c11a90?})
        /go/pkg/mod/github.com/prometheus-community/windows_exporter@v0.27.2/pkg/collector/prometheus.go:117 +0xa5
created by github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).Collect in goroutine 880
        /go/pkg/mod/github.com/prometheus-community/windows_exporter@v0.27.2/pkg/collector/prometheus.go:115 +0x470

This works but generates a lot of warnings. Running Grafana Alloy with "service" collector enabled and all the properties configured:

ts=2024-09-25T08:30:14.9783748Z level=warn msg="API collection is enabled." component_path=/ component_id=prometheus.exporter.windows.integrations_windows_exporter collector=service
ts=2024-09-25T08:31:05.9236679Z level=warn msg="Get service McpManagementService config error:  0x3afc" component_path=/ component_id=prometheus.exporter.windows.integrations_windows_exporter collector=service
ts=2024-09-25T08:32:05.8807195Z level=warn msg="Get service McpManagementService config error:  0x3afc" component_path=/ component_id=prometheus.exporter.windows.integrations_windows_exporter collector=service
ts=2024-09-25T08:33:05.8822429Z level=warn msg="Get service McpManagementService config error:  0x3afc" component_path=/ component_id=prometheus.exporter.windows.integrations_windows_exporter collector=service

[jensjensjens]

We just started migrating from Agent Flow Mode 0.38 to Alloy 1.4.0 and this became a blocker for us. I believe it would be good to give this some attentions since many of us have to get to 1.4.0 get fix the security issues in the windows service in lower versions.

[erikbaranowski]

This is due to an upstream bug in windows_exporter. The version of windows_exporter was recently updated in alloy. I am digging into options.

Specifically, this code does not set v2

https://github.com/prometheus-community/windows_exporter/blob/4e460bc24c3668fbef641a187162073cd5da2ecb/pkg/collector/service/service.go#L58

I validated the fix locally by vendoring the package with alloy and a config that was able to reproduce the issue with the following code fix

func New(logger log.Logger, config *Config) *Collector {
	if config == nil {
		config = &ConfigDefaults
	}

	c := &Collector{
		serviceWhereClause: &config.ServiceWhereClause,
		useAPI:             &config.UseAPI,
		v2:                 &config.V2,
	}
	c.SetLogger(logger)

	return c
}

[erikbaranowski]

I've opened a bug issue with a code fix included to see if we can get it backported and quickly release a patch for alloy. Otherwise, we can back out the exporter update or jump to v0.29.* which looks like it may be working but requires a later go toolchain than alloy is using.

prometheus-community/windows_exporter#1645