adding support for a collector config file.

[AnalogJ]

This is still a WIP. But here's an example of what the collector config file will look like:

# Commented Scrutiny Configuration File
#
# The default location for this file is /scrutiny/config/collector.yaml.
# In some cases to improve clarity default values are specified,
# uncommented. Other example values are commented out.
#
# When this file is parsed by Scrutiny, all configuration file keys are
# lowercased automatically. As such, Configuration keys are case-insensitive,
# and should be lowercase in this file to be consistent with usage.


######################################################################
# Version
#
# version specifies the version of this configuration file schema, not
# the scrutiny binary. There is only 1 version available at the moment
version: 1

# This block allows you to override/customize the settings for devices detected by
# Scrutiny via `smartctl --scan`
# See the "--device=TYPE" section of https://linux.die.net/man/8/smartctl
# type can be a 'string' or a 'list'
devices:
  # example for forcing device type detection for a single disk
  - device: /dev/sda
    type: 'sat'

  # example to show how to ignore a specific disk/device.
  - device: /dev/sda
    ignore: true

  # examples showing how to force smartctl to detect disks inside a raid array/virtual disk
  - device: /dev/bus/0
    type:
      - megaraid,14
      - megaraid,15
      - megaraid,18
      - megaraid,19
      - megaraid,20
      - megaraid,21

  - device: /dev/twa0
    type:
      - 3ware,0
      - 3ware,1
      - 3ware,2
      - 3ware,3
      - 3ware,4
      - 3ware,5

You can test it out yourself by using the analogj/scrutiny:detect docker image. Then just create a collector.yaml file in the host directory you bind to /scrutiny/config:

docker run -it --rm -p 8080:8080 \
-v /run/udev:/run/udev:ro \
-v /tmp/scrutiny-config:/scrutiny/config \
--cap-add SYS_RAWIO \
--device=/dev/sda \
--device=/dev/sdb \
--name scrutiny \
analogj/scrutiny

---

Current Changelog

Adding ability to specify host identifier (label), that is updated on every collector run.
Can be specified by host-id CLI or COLLECTOR_HOST_ID env var.

Created a config class, interface and associated tests.

Created a "TransformDetectedDrives" function, that will allow users to insert drives not detected by Smarctl --scan, ignore drives that they dont want, and override smartctl device type.

Added Upsert functionality when registering devices.

Replaced "github.com/jinzhu/gorm" with "gorm.io/gorm" (ORM location moved, was using incorrect lib url)
Removed machineid library.

fixes #46
fixes #45
fixes #48
fixes #39
related #50 - adds a HostID to the database that is updated on every collector run. Not displayed in UI yet.

[codecov-io]

Codecov Report

Merging #88 into master will increase coverage by 7.19%.
The diff coverage is 72.22%.

@@            Coverage Diff             @@
##           master      #88      +/-   ##
==========================================
+ Coverage   32.86%   40.06%   +7.19%     
==========================================
  Files          12       14       +2     
  Lines         286      332      +46     
==========================================
+ Hits           94      133      +39     
- Misses        187      189       +2     
- Partials        5       10       +5     

Flag	Coverage Δ
#unittests	40.06% <72.22%> (+7.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
collector/pkg/detect/devices_linux.go	6.25% <0.00%> (ø)
webapp/backend/pkg/models/db/device.go	0.00% <ø> (ø)
webapp/backend/pkg/models/db/smart.go	100.00% <ø> (ø)
...ebapp/backend/pkg/models/db/smart_ata_attribute.go	0.00% <ø> (ø)
...bapp/backend/pkg/models/db/smart_nvme_attribute.go	0.00% <ø> (ø)
...app/backend/pkg/models/db/smart_scsci_attribute.go	0.00% <ø> (ø)
collector/pkg/config/factory.go	50.00% <50.00%> (ø)
collector/pkg/config/config.go	58.33% <58.33%> (ø)
collector/pkg/detect/detect.go	33.33% <95.83%> (+33.33%)	⬆️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32e7044...9fac3c6. Read the comment docs.

[PrplHaz4]

This config is working great for me for 2 SAS and 2 SATA drives on an H240 controller as HBA. I'm also running scrutiny on multiple hosts in docker with the standalone connector and not seeing any obvious issues yet. Nice work!

devices:
  - device: /dev/sdc
    type: 'sat'
  - device: /dev/sdd
    type: 'sat'

  - device: /dev/sg0
    type:
      - cciss,0
      - cciss,1

[mrtumnus]

Works for me! I'm using the 3ware type for 7 SATA drives, and it works great! Good to see all my drives are healthy :)

[AnalogJ]

Awesome! Thanks for the feedback :)
I'll merge this and make a new release.

[elliottback]

Thanks for the patch, confirm this is also working great for my Synology DS1513 + 2 expansion units.

[auricom]

Hello,

First of all, wonderful piece of software 👍

I run Scrutiny on a Kubernetes cluster, where it is not possible to passthrough devices to unprivileged containers. I'm running scrutiny with privileged: true. The scrutiny collector container see all the node devices, including the kubernetes volumes that are virtual devices (smartctl detect them as SCSI drives, as seen in smartctl --scan :
...
/dev/sdab -d scsi # /dev/sdab, SCSI device
/dev/sdad -d scsi # /dev/sdad, SCSI device
...

It seems that I should be using the "ignore: true" features on "devices" in collector.yaml , but I cannot know how many volumes will be present on each drives, so I cannot ignore specific devices.

I would like something like this to be working :

version: 1
devices:
  - type: 'nvme'
  - type: 'sat'
  - type: 'scsi'
    ignore: true

It seems that this is not working, as scrutiny-web still reports dozen of unwanted ISCI drives.
Do you know if this use case is manageable with the new feature you released ?

Thanks

[AnalogJ]

Hey @auricom

Unfortunately there's no way to block a device type completely, only individual devices.
You do bring up an interesing use-case for it though.
Can you open up a new feature request and copy-paste your comment? I think this would be useful for other users as well, and I don't want to lose it in a closed-issue.

[auricom]

hello,
thanks I'm glad you consider this use case to be interesting. I created a new feature request.

[FinalDoom]

Glad this is in here, hopefully makes it to the readme at some point. I got some nice metrics (woo, cool, great app) for my hp server running hpsa (HP Storage Array RAID) type--took a bit of digging into smartctl to find:

smartctl -a -d cciss,0 /dev/cciss/c0d0    (cciss driver under Linux)
smartctl -a -d cciss,0 /dev/sg2    (hpsa or hpahcisr drivers under Linux)

but putting the -d stuff in the config file worked a charm. Maybe it'll show up if anyone searches and runs into the same thing. I got here from the dell megaraid thread.

Also I'm dumb and didn't read the comments fully in the sample config. XD Woulda made it a little more obvious.