Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi partition support #32

Merged
merged 7 commits into from
Dec 12, 2024
Merged

Multi partition support #32

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8fe79f4
feat(math): Add RoundFloat function to work with primitive types
mertssmnoglu Dec 4, 2024
70afe66
feat(disk-metrics): Add MountPoint field to DiskData
mertssmnoglu Dec 4, 2024
a891420
feat(disk-metrics): Enhance CollectDiskMetrics to include MountPoint …
mertssmnoglu Dec 4, 2024
41e23e5
feat(disk-metrics): Rename MountPoint field to Device
mertssmnoglu Dec 6, 2024
32892f2
ref(disk-metrics): Refactor CollectDiskMetrics to use Device instead …
mertssmnoglu Dec 6, 2024
2efdc25
fix(disk-metrics): Set default Device value to "unknown" and improve …
mertssmnoglu Dec 6, 2024
636c7e7
fix(disk-metrics): Exclude '/dev/loop' devices in CollectDiskMetrics
mertssmnoglu Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 47 additions & 22 deletions internal/metric/disk.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
package metric

import (
disk2 "github.com/shirou/gopsutil/v4/disk"
"slices"
"strings"

"github.com/shirou/gopsutil/v4/disk"
)

func CollectDiskMetrics() (MetricsSlice, []CustomErr) {
defaultDiskData := []*DiskData{
{
Device: "unknown",
ReadSpeedBytes: nil,
WriteSpeedBytes: nil,
TotalBytes: nil,
Expand All @@ -15,35 +19,56 @@ func CollectDiskMetrics() (MetricsSlice, []CustomErr) {
},
}
var diskErrors []CustomErr
diskUsage, diskUsageErr := disk2.Usage("/")
var metricsSlice MetricsSlice
var checkedSlice []string // To keep track of checked partitions

// Set all flag to "false" to get only necessary partitions
// Avoiding unnecessary partitions like /run/user/1000, /run/credentials
partitions, partErr := disk.Partitions(false)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns a lot of partitions that we aren't interested in, I think we need to refine how we parse out which partitions we want information on.

Is it possible to filter out all loopbacks? That would clean this up a lot.

Example, my current lsblk output which matches what disk.Partitions returns:

loop0         7:0    0     4K  1 loop /snap/bare/5
loop1         7:1    0  55.4M  1 loop /snap/core18/2846
loop2         7:2    0   274M  1 loop /snap/firefox/5361
loop3         7:3    0   273M  1 loop /snap/firefox/5273
loop4         7:4    0 245.5M  1 loop /snap/gitkraken/247
loop5         7:5    0  91.7M  1 loop /snap/gtk-common-themes/1535
loop6         7:6    0 504.2M  1 loop /snap/gnome-42-2204/172
loop7         7:7    0  73.9M  1 loop /snap/core22/1663
loop8         7:8    0 505.1M  1 loop /snap/gnome-42-2204/176
loop9         7:9    0 248.9M  1 loop /snap/gitkraken/249
loop10        7:10   0  73.9M  1 loop /snap/core22/1722
loop11        7:11   0  12.9M  1 loop /snap/snap-store/1113
loop12        7:12   0  38.8M  1 loop /snap/snapd/21759
loop13        7:13   0  12.2M  1 loop /snap/snap-store/1216
loop14        7:14   0  44.3M  1 loop /snap/snapd/23258
loop15        7:15   0 150.3M  1 loop /snap/thunderbird/581
loop16        7:16   0 150.6M  1 loop /snap/thunderbird/585
nvme0n1     259:0    0   1.8T  0 disk 
├─nvme0n1p1 259:1    0   190M  0 part /boot/efi
├─nvme0n1p2 259:2    0   128M  0 part 
├─nvme0n1p3 259:3    0   962G  0 part

We're actually only interested innvme0n1 here, all the loops can be ignored

Copy link
Collaborator Author

@mertssmnoglu mertssmnoglu Dec 5, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now I'm thinking about reading only devices instead of mount points. This will show us

/dev/nvme0n1p2 => 0.80 usage
/dev/nvme0n1p1 => 0.30 usage
/dev/sda1 => 0.70 usage

I think this is more clear than the previous one. (home, boot, efi, dev ...)

Preview

{
    "device": "/dev/sda1",
    "read_speed_bytes": null,
    "write_speed_bytes": null,
    "total_bytes": 30713544704,
    "free_bytes": 6828425216,
    "usage_percent": 0.7777
}

Are you ok with that? @ajhollid

Copy link
Collaborator

@ajhollid ajhollid Dec 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mertssmnoglu I think that makes more sense! Sounds good to me 👍 Thanks for making the changes!

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current changes should work for it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mertssmnoglu sorry for the late reply, just having a chance to look at this now.

This is better, but I still have all the loop backs in my response:

{
	"data": [
		{
			"device": "/dev/nvme0n1p7",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 931258499072,
			"free_bytes": 719357485056,
			"usage_percent": 0.1862
		},
		{
			"device": "/dev/loop0",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 131072,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop1",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 58064896,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop2",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 286261248,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop3",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 77463552,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop4",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 260964352,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop5",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 528744448,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop6",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 77463552,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop7",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 157941760,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop13",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 157679616,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop14",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 46530560,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop8",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 529661952,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop10",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 13631488,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop15",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 257556480,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop16",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 287440896,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop11",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 40763392,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop9",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 96206848,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/loop12",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 12845056,
			"free_bytes": 0,
			"usage_percent": 1
		},
		{
			"device": "/dev/nvme0n1p1",
			"read_speed_bytes": null,
			"write_speed_bytes": null,
			"total_bytes": 195035136,
			"free_bytes": 66547712,
			"usage_percent": 0.6588
		}
	],
	"errors": null
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mat be related to this issue, unless I'm reading that discussion incorreclty it seems that psutil ignores the all flag for all systems due to how BSD behaves 🤔

This seems ot be the case, the output is the same whether I pass true or false for the all flag.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's interesting. I added a new condition to if statement in 636c7e7


if diskUsageErr != nil {
if partErr != nil {
diskErrors = append(diskErrors, CustomErr{
Metric: []string{"disk.usage_percent", "disk.total_bytes", "disk.free_bytes"},
Error: diskUsageErr.Error(),
Metric: []string{"disk.partitions"},
Error: partErr.Error(),
})
return MetricsSlice{defaultDiskData[0]}, diskErrors
}

// diskMetrics, diskErr := disk1.Get()
// if diskErr != nil {
// log.Fatalf("Unable to get disk metrics")
// }
for _, p := range partitions {
// Filter out partitions that are already checked or not a device
// Also, exclude '/dev/loop' devices to avoid unnecessary partitions
// * /dev/loop devices are used for mounting snap packages
if slices.Contains(checkedSlice, p.Device) || !strings.HasPrefix(p.Device, "/dev") || strings.HasPrefix(p.Device, "/dev/loop") {
continue
}

// for _, p := range diskMetrics {
// fmt.Println(p.Name, p.ReadsCompleted)
// }
diskUsage, diskUsageErr := disk.Usage(p.Mountpoint)
if diskUsageErr != nil {
diskErrors = append(diskErrors, CustomErr{
Metric: []string{"disk.usage_percent", "disk.total_bytes", "disk.free_bytes"},
Error: diskUsageErr.Error() + " " + p.Mountpoint,
})
continue
}

// var a uint64 = 2e+12
var metricsSlice MetricsSlice
checkedSlice = append(checkedSlice, p.Device)
metricsSlice = append(metricsSlice, &DiskData{
Device: p.Device,
ReadSpeedBytes: nil, // TODO: Implement
WriteSpeedBytes: nil, // TODO: Implement
TotalBytes: &diskUsage.Total,
FreeBytes: &diskUsage.Free,
UsagePercent: RoundFloatPtr(diskUsage.UsedPercent/100, 4),
})
}

if len(diskErrors) == 0 {
return metricsSlice, nil
}

if len(metricsSlice) == 0 {
return MetricsSlice{defaultDiskData[0]}, diskErrors
}

metricsSlice = append(metricsSlice, &DiskData{
ReadSpeedBytes: nil, // TODO: Implement
WriteSpeedBytes: nil, // TODO: Implement
TotalBytes: &diskUsage.Total,
FreeBytes: &diskUsage.Free,
UsagePercent: RoundFloatPtr(diskUsage.UsedPercent/100, 4),
})
return metricsSlice, diskErrors
}

Expand Down
7 changes: 4 additions & 3 deletions internal/metric/metric.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,12 @@ type MemoryData struct {
func (m MemoryData) isMetric() {}

type DiskData struct {
Device string `json:"device"` // Device
ReadSpeedBytes *uint64 `json:"read_speed_bytes"` // TODO: Implement
WriteSpeedBytes *uint64 `json:"write_speed_bytes"` // TODO: Implement
TotalBytes *uint64 `json:"total_bytes"` // Total space of "/" in bytes
FreeBytes *uint64 `json:"free_bytes"` // Free space of "/" in bytes
UsagePercent *float64 `json:"usage_percent"` // Usage Percent of "/"
TotalBytes *uint64 `json:"total_bytes"` // Total space of device in bytes
FreeBytes *uint64 `json:"free_bytes"` // Free space of device in bytes
UsagePercent *float64 `json:"usage_percent"` // Usage Percent of device
}

func (d DiskData) isMetric() {}
Expand Down
8 changes: 7 additions & 1 deletion internal/metric/metric_math.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,15 @@ import (

// Round a float to a given precision and return the pointer of the result
func RoundFloatPtr(val float64, precision uint) *float64 {
r := RoundFloat(val, precision)
return &r
}

// Round a float to a given precision and return the result
func RoundFloat(val float64, precision uint) float64 {
ratio := math.Pow(10, float64(precision))
prc := math.Round(val*ratio) / ratio
return &prc
return prc
}

func RandomIntPtr(max int64) *int {
Expand Down