Tutorial: SMART Monitoring with Scrutiny across machines

[Tin-Joy59]

S.M.A.R.T. Monitoring with Scrutiny across machines

🤔 The problem:

Scrutiny offers a nice Docker package called "Omnibus" that can monitor HDDs attached to a Docker host with relative ease. Scrutiny can also be installed in a Hub-Spoke layout where Web interface, Database and Collector come in 3 separate packages. The official documentation assumes that the spokes in the "Hub-Spokes layout" run Docker, which is not always the case. The third approach is to install Scrutiny manually, entirely outside of Docker.

💡 The solution:

This tutorial provides a hybrid configuration where the Hub lives in a Docker instance while the spokes have only Scrutiny Collector installed manually. The Collector periodically send data to the Hub. It's not mind-boggling hard to understand but someone might struggle with the setup. This is for them.

🖥️ My setup:

I have a Proxmox cluster where one VM runs Docker and all monitoring services - Grafana, Prometheus, various exporters, InfluxDB and so forth. Another VM runs the NAS - OpenMediaVault v6, where all hard drives reside. The Scrutiny Collector is triggered every 30min to collect data on the drives. The data is sent to the Docker VM, running InfluxDB.

Setting up the Hub

The Hub consists of Scrutiny Web - a web interface for viewing the SMART data. And InfluxDB, where the smartmon data is stored.

🔗This is the official Hub-Spoke layout in docker-compose. We are going to reuse parts of it. The ENV variables provide the necessary configuration for the initial setup, both for InfluxDB and Scrutiny.

If you are working with and existing InfluxDB instance, you can forgo all the INIT variables as they already exist.

The official Scrutiny documentation has a sample scrutiny.yaml file that normally contains the connection and notification details but I always find it easier to configure as much as possible in the docker-compose.

version: "3.4"

networks:
  monitoring:       # A common network for all monitoring services to communicate into
    external: true
  notifications:    # To Gotify or another Notification service
    external: true

services:
  influxdb:
    container_name: influxdb
    image: influxdb:2.1-alpine
    ports:
      - 8086:8086
    volumes:
      - ${DIR_CONFIG}/influxdb2/db:/var/lib/influxdb2
      - ${DIR_CONFIG}/influxdb2/config:/etc/influxdb2
    environment:
      - DOCKER_INFLUXDB_INIT_MODE=setup
      - DOCKER_INFLUXDB_INIT_USERNAME=Admin
      - DOCKER_INFLUXDB_INIT_PASSWORD=${PASSWORD}
      - DOCKER_INFLUXDB_INIT_ORG=homelab
      - DOCKER_INFLUXDB_INIT_BUCKET=scrutiny
      - DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=your-very-secret-token
    restart: unless-stopped
    networks:
      - monitoring

  scrutiny:
    container_name: scrutiny
    image: ghcr.io/analogj/scrutiny:master-web
    ports:
      - 8080:8080
    volumes:
      - ${DIR_CONFIG}/scrutiny/config:/opt/scrutiny/config
    environment:
      - SCRUTINY_WEB_INFLUXDB_HOST=influxdb
      - SCRUTINY_WEB_INFLUXDB_PORT=8086
      - SCRUTINY_WEB_INFLUXDB_TOKEN=your-very-secret-token
      - SCRUTINY_WEB_INFLUXDB_ORG=homelab
      - SCRUTINY_WEB_INFLUXDB_BUCKET=scrutiny
      # Optional but highly recommended to notify you in case of a problem
      - SCRUTINY_WEB_NOTIFY_URLS=["http://gotify:80/message?token=a-gotify-token"]
    depends_on:
      - influxdb
    restart: unless-stopped
    networks:
      - notifications
      - monitoring

A freshly initialized Scrutiny instance can be accessed on port 8080, eg. 192.168.0.100:8080. The interface will be empty because no metrics have been collected yet.

Setting up a Spoke without Docker

A spoke consists of the Scrutiny Collector binary that is run on a set interval via crontab and sends the data to the Hub. The official documentation describes the manual setup of the Collector - dependencies and step by step commands. I have a shortened version that does the same thing but in one line of code.

# Installing dependencies
apt install smartmontools -y 

# 1. Create directory for the binary
# 2. Download the binary into that directory
# 3. Make it exacutable
# 4. List the contents of the library for confirmation
mkdir -p /opt/scrutiny/bin && \
curl -L https://github.com/AnalogJ/scrutiny/releases/download/v0.5.0/scrutiny-collector-metrics-linux-amd64 > /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 && \
chmod +x /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 && \
ls -lha /opt/scrutiny/bin

When downloading Github Release Assests, make sure that you have the correct version. The provided example is with Release v0.5.0. [The release list can be found here.](https://github.com/analogj/scrutiny/releases)

Once the Collector is installed, you can run it with the following command. Make sure to add the correct address and port of your Hub as --api-endpoint.

/opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://192.168.0.100:8080"

This will run the Collector once and populate the Web interface of your Scrutiny instance. In order to collect metrics for a time series, you need to run the command repeatedly. Here is an example for crontab, running the Collector every 15min.

# open crontab
crontab -e

# add a line for Scrutiny
*/15 * * * * /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://192.168.0.100:8080"

The Collector has its own independent config file that lives in /opt/scrutiny/config/collector.yaml but I did not find a need to modify it. A default collector.yaml can be found in the official documentation.

Setting up a Spoke with Docker

Setting up a remote Spoke in Docker requires you to split the official Hub-Spoke layout docker-compose.yml. In the following docker-compose you need to provide the ${API_ENDPOINT}, in my case http://192.168.0.100:8080. Also all drives that you wish to monitor need to be presented to the container under devices.

The image handles the periodic scanning of the drives.

version: "3.4"

services:

  collector:
    image: 'ghcr.io/analogj/scrutiny:master-collector'
    cap_add:
      - SYS_RAWIO
    volumes:
      - '/run/udev:/run/udev:ro'
    environment:
      COLLECTOR_API_ENDPOINT: ${API_ENDPOINT}
    devices:
      - "/dev/sda"
      - "/dev/sdb"

[AnalogJ]

Nice guide! I'll get it added to the docs/ folder

[MACscr]

Just to make sure I have this straight, this would allow me to to run the collector on different systems and send it to the central scrutiny docker so that i can monitor the health of other drives on my network?

[tenekev]

Yes. What was offred in the docs beforehand was a scenario where if you decided to use the hub and spoke setup instead of the single omnibus image, you would have to run both in docker. I run my hub in docker, alongside the rest of my monitoring tools. But the spoke is a NAS without docker. That's why I wrote this guide.

[itkfilelor]

Gonna leave this here for anyone who ends up having the same issue I did.
I am running my spokes across several proxmox nodes, directly on the host.
my cron command looks more like this:

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin && /opt/scrutiny/bin/scrutiny-collector-metrics-linux-amd64 run --api-endpoint "http://192.168.1.100:8080"

otherwise the collector log would fail stating the smartctl binary was missing.
I am sure there is a simpler more elegant method, but I am lazy and it works =)

[jp12git]

@Tin-Joy59 Thank you for the guide! I'm trying to follow this but not very familiar with Docker compose. I get the following error when trying to deploy your example config in Portainer stacks

failed to deploy a stack: time="2024-01-14T22:43:00Z" level=warning msg="The \"DIR_CONFIG\" variable is not set. Defaulting to a blank string." 

Any suggestions? Should I be setting the DIR_CONFIG variable in the YAML to something? but where?

[tenekev]

The ${DIR_CONFIG} variable is just the location where all the configs live. Imagine you have 50 other services and the configs for each live in /docker_configs. Instead of repeating this root directory multiple times, you can just set an environmental variable like this and avoid repetition.
${DIR_CONFIG}/app1/, ${DIR_CONFIG}/app2/, ${DIR_CONFIG}/app3/ instead of /docker_configs/app1, /docker_configs/app2, /docker_configs/app3.

Moreso, if you ever change the root config directory of your services, you will need to change only the value of the variable, not 50 different service definitions.

If you are using Portainer, there is an "Environmental variables" section under the stack editor. Create a DIR_CONFIG variable and set it to your config location. Or you can just replace the ${DIR_CONFIG} with the exact path of your config and proceed.

[tismofied]

Thank you for this guide. I was having a hard time getting data on the dashboard and couldn't figure out why. it truned out I needed this guide because of proxmox.

[tismofied]

@tenekev , Sorry to bother you after almost 2 years of making this guide but I am having a hard time setting up the notification.
do I remove the brackets from the notification url or keep it?
would the same structure that's in the Example scrutiny.yaml work in this compose file or is it a little different?

[AnalogJ]

@tismofied The square brackets in the examples are to specify that the username is optional. They should not be included in your configuration.

If you're having issues generating a notification url, you should look at the official Shoutrrr documentation for your service - https://containrrr.dev/shoutrrr/v0.8/services/bark/

Scrutiny just uses Shoutrrr under the hood.

[tismofied]

Fixed my issue thank you!