Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add drive I/O metrics for Prometheus #955

Merged
merged 1 commit into from
Oct 15, 2024
Merged

Add drive I/O metrics for Prometheus #955

merged 1 commit into from
Oct 15, 2024

Conversation

balamurugana
Copy link
Member

@balamurugana balamurugana commented Oct 3, 2024 •
edited
Loading

Below metrics are exported:

  • directpv_stats_drive_ready
  • directpv_stats_drive_total_read_bytes
  • directpv_stats_drive_total_write_bytes
  • directpv_stats_drive_read_latency_seconds
  • directpv_stats_drive_write_latency_seconds
  • directpv_stats_drive_wait_time_seconds

Fixes #839

@balamurugana balamurugana force-pushed the Add-drive-I-O-metrics-for-Prometheus branch from 232229b to 0cd0bfc Compare October 3, 2024 16:15
Praveenrajmani
Praveenrajmani reviewed Oct 4, 2024
View reviewed changes
pkg/device/sysfs_linux.go Show resolved Hide resolved
pkg/metrics/collector.go Show resolved Hide resolved
@Praveenrajmani
Copy link
Collaborator

Praveenrajmani commented Oct 8, 2024 •
edited
Loading 

E1008 10:59:43.876992  132046 collector.go:162] "unable to read drive statistics" err="strconv.ParseUint: parsing \"\": invalid syntax"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1aae750]

goroutine 488 [running]:
github.com/minio/directpv/pkg/metrics.(*metricsCollector).publishDriveStats(0xc0009581e0?, 0xc000a82000, 0xc0004a4070)
	/home/praveen/go/src/github.com/minio/directpv/pkg/metrics/collector.go:181 +0x490
github.com/minio/directpv/pkg/metrics.(*metricsCollector).Collect(0xc0006107e0, 0xc0004a4070)
	/home/praveen/go/src/github.com/minio/directpv/pkg/metrics/collector.go:270 +0x54d
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
	/home/praveen/go/pkg/mod/github.com/prometheus/client_golang@v1.20.4/prometheus/registry.go:456 +0x105
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather in goroutine 484
	/home/praveen/go/pkg/mod/github.com/prometheus/client_golang@v1.20.4/prometheus/registry.go:548 +0xbab

I see a panic in node-server container when I try to scrape the metrics via prometheus @balamurugana

@balamurugana balamurugana force-pushed the Add-drive-I-O-metrics-for-Prometheus branch from 0cd0bfc to 925e2bd Compare October 9, 2024 01:19
@balamurugana
Copy link
Member Author 

E1008 10:59:43.876992  132046 collector.go:162] "unable to read drive statistics" err="strconv.ParseUint: parsing \"\": invalid syntax"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1aae750]

goroutine 488 [running]:
github.com/minio/directpv/pkg/metrics.(*metricsCollector).publishDriveStats(0xc0009581e0?, 0xc000a82000, 0xc0004a4070)
	/home/praveen/go/src/github.com/minio/directpv/pkg/metrics/collector.go:181 +0x490
github.com/minio/directpv/pkg/metrics.(*metricsCollector).Collect(0xc0006107e0, 0xc0004a4070)
	/home/praveen/go/src/github.com/minio/directpv/pkg/metrics/collector.go:270 +0x54d
github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1()
	/home/praveen/go/pkg/mod/github.com/prometheus/client_golang@v1.20.4/prometheus/registry.go:456 +0x105
created by github.com/prometheus/client_golang/prometheus.(*Registry).Gather in goroutine 484
	/home/praveen/go/pkg/mod/github.com/prometheus/client_golang@v1.20.4/prometheus/registry.go:548 +0xbab

I see a panic in node-server container when I try to scrape the metrics via prometheus @balamurugana

@Praveenrajmani fixed

@balamurugana balamurugana force-pushed the Add-drive-I-O-metrics-for-Prometheus branch 2 times, most recently from e8d0653 to 4121cae Compare October 9, 2024 10:24
@balamurugana balamurugana force-pushed the Add-drive-I-O-metrics-for-Prometheus branch 3 times, most recently from af76a90 to 77af8a2 Compare October 10, 2024 01:59
@balamurugana
Add drive I/O metrics for Prometheus
a9b4909 
Below metrics are exported:
* directpv_stats_drive_ready
* directpv_stats_drive_total_read_bytes
* directpv_stats_drive_total_write_bytes
* directpv_stats_drive_read_latency_seconds
* directpv_stats_drive_write_latency_seconds
* directpv_stats_drive_wait_time_seconds

Fixes minio#839

Signed-off-by: Bala.FA <bala@minio.io>
@balamurugana balamurugana force-pushed the Add-drive-I-O-metrics-for-Prometheus branch from 77af8a2 to a9b4909 Compare October 10, 2024 02:02
@Praveenrajmani Praveenrajmani merged commit 35aaf55 into minio:master Oct 15, 2024
27 checks passed
@Praveenrajmani Praveenrajmani mentioned this pull request Oct 15, 2024
Closed
@balamurugana balamurugana deleted the Add-drive-I-O-metrics-for-Prometheus branch October 17, 2024 00:59
balamurugana added a commit to balamurugana/minio-directpv that referenced this pull request Oct 17, 2024
@balamurugana
Add drive I/O metrics for Prometheus (minio#955)
67ce1ef 
Below metrics are exported:
* directpv_stats_drive_ready
* directpv_stats_drive_total_read_bytes
* directpv_stats_drive_total_write_bytes
* directpv_stats_drive_read_latency_seconds
* directpv_stats_drive_write_latency_seconds
* directpv_stats_drive_wait_time_seconds

Fixes minio#839

Signed-off-by: Bala.FA <bala@minio.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Praveenrajmani Praveenrajmani Praveenrajmani approved these changes

@r-scheele r-scheele Awaiting requested review from r-scheele

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Export IO metrics
2 participants
@balamurugana @Praveenrajmani