Trouble getting drives behind megaraid to show

[PriamX]

I have 10 drives behind an LSI/Avago/Broadcom MegaRAID 9361-16i adapter.

EID:Slt DID State DG       Size Intf Med SED PI SeSz Model                  Sp Type
------------------------------------------------------------------------------------
252:0    63 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:1    64 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:2    62 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:3    61 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:4    67 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:5    68 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:6    66 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:7    65 Onln   0   7.276 TB SATA HDD N   N  512B WDC WD8003FFBX-68B9AN0 U  -
252:8    70 Onln   1 185.781 GB SAS  SSD Y   N  512B H1SMR324 CLAR200       U  -
252:9    69 Onln   1 185.781 GB SAS  SSD Y   N  512B H1SMR324 CLAR200       U  -
------------------------------------------------------------------------------------

On the host, a smartclt --scan shows them just fine, the OS (Fedora 35) uses the RAID as /dev/sdc:

smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.19.4-100.fc35.x86_64] (local build)

[root@myserv ~]# smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/bus/13 -d megaraid,61 # /dev/bus/13 [megaraid_disk_61], SCSI device
/dev/bus/13 -d megaraid,62 # /dev/bus/13 [megaraid_disk_62], SCSI device
/dev/bus/13 -d megaraid,63 # /dev/bus/13 [megaraid_disk_63], SCSI device
/dev/bus/13 -d megaraid,64 # /dev/bus/13 [megaraid_disk_64], SCSI device
/dev/bus/13 -d megaraid,65 # /dev/bus/13 [megaraid_disk_65], SCSI device
/dev/bus/13 -d megaraid,66 # /dev/bus/13 [megaraid_disk_66], SCSI device
/dev/bus/13 -d megaraid,67 # /dev/bus/13 [megaraid_disk_67], SCSI device
/dev/bus/13 -d megaraid,68 # /dev/bus/13 [megaraid_disk_68], SCSI device
/dev/bus/13 -d megaraid,69 # /dev/bus/13 [megaraid_disk_69], SCSI device
/dev/bus/13 -d megaraid,70 # /dev/bus/13 [megaraid_disk_70], SCSI device
/dev/nvme0 -d nvme # /dev/nvme0, NVMe device
/dev/nvme1 -d nvme # /dev/nvme1, NVMe device
[root@myserv ~]#

In the docker container, they do not appear (I noticed the smartctl version is slightly older):

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.19.4-100.fc35.x86_64] (local build)

[root@myserv ~]# docker exec -it scrutiny /bin/bash
root@0521a150d7f2:/opt/scrutiny# smartctl --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdb -d scsi # /dev/sdb, SCSI device
/dev/sdc -d scsi # /dev/sdc, SCSI device
/dev/nvme0 -d nvme # /dev/nvme0, NVMe device
/dev/nvme1 -d nvme # /dev/nvme1, NVMe device
root@0521a150d7f2:/opt/scrutiny#

I have mapped /dev/bus through to the docker container, here's the docker command I'm using to deploy scrutiny:

docker run -d \
  --name=scrutiny \
  --cap-add=SYS_RAWIO \
  --cap-add=SYS_ADMIN `#optional` \
  -e TZ=America/Denver \
  -p 8084:8080 \
  -p 8086:8086 \
  --network="mynet" \
  -v /media/docker/appdata/scrutiny:/opt/scrutiny/config \
  -v /media/docker/appdata/influxdb2:/opt/scrutiny/influxdb \
  -v /run/udev:/run/udev:ro \
  --device /dev/sda:/dev/sda \
  --device /dev/sdb:/dev/sdb \
  --device /dev/sdc:/dev/sdc \
  --device /dev/sdd:/dev/sdd \
  --device /dev/nvme0:/dev/nvme0 \
  --device /dev/nvme1:/dev/nvme1 \
  --device /dev/sr0:/dev/sr0 \
  --device /dev/bus:/dev/bus \
  --restart unless-stopped \
  ghcr.io/analogj/scrutiny:master-omnibus

Here's the devices section from the collector.yaml I'm using:

devices:

   - device: /dev/sda
     type: 'ata'

   - device: /dev/sdb
     type: 'ata'

   - device: /dev/bus/13
     type:
       - megaraid,61
       - megaraid,62
       - megaraid,63
       - megaraid,64
       - megaraid,65
       - megaraid,66
       - megaraid,67
       - megaraid,68
       - megaraid,69
       - megaraid,70

   - device: /dev/sdd
     type:
       - jmb39x-q,0
       - jmb39x-q,1

   - device: /dev/nvme0
     type: 'nvme'

   - device: /dev/nvme1
     type: 'nvme'

   - device: /dev/sr0
     type: 'ata'

Here's the dashboard:

Has anyone got a MegaRAID to work with Scrutiny, if so, are there any tricks I'm missing?

Thanks!

[PriamX]

Okay, well, it works when I add the --privileged flag to the docker command.

So, apparently, it seems it's a permission issue.

I suspected running the container in privileged mode is not the correct way to solve this issue. Any alternative ideas?

Thanks!

[AnalogJ]

Just incase you havent seen it, here's my docs on megaraid3
Having said that, it looks like your configuration is similar to what's already documented there.

I don't personally use a megaraid controller, so this is a bit hard to troubleshoot for me. I'll leave this open for other users to chime in. You might also consider posting your question in the Self-Hosted Discord #storage channel

[PriamX]

I've been searching around the 'net regarding docker containers and /dev/bus, most of the examples are exposing a USB device through from the host to the container. If you look at my docker command above, you'll see that I pass the /dev/bus device through, but that alone didn't seem to work. I tried passing /dev/bus/13 through (which is the megaraid controller on the host), but the docker command fails with an error saying it doesn't exists. Since it works in "privileged" mode, it points heavily to a permission issue, so there may be some kind of option to get it working without running --privileged, but so far I haven't found it in my searches.

[PriamX]

Okay. I've been running this for about 20 days since my last post and thought I'd just leave a closing comment regarding running Scrutiny in a docker container with a MegaRAID (and perhaps other RAID controllers).

The cap_add parameters SYS_RAWIO is required and SYS_ADMIN (for NVMe) as the example shows, and work well for drives attached to the mobo. However, these capabilities don't get you down into to the RAID controller.

In the docker container you must also expose the RAID controller device to the container. For example, I had two RAIDs combined into one virtual drive (8 HDDs with 2 cache SSDs) to the Linux OS this virtual drive shows up as /dev/sdc.

And you can use smartctl to get the SMART information for each drive like so: smartctl -a -d megaraid,62 /dev/sdc but don't get into the habit of using /dev/sdX that can change depending on what USB devices you may have plugged in. For example, if you plug in a USB thumb drive, leave it in, and later reboot, that thumb drive may become /dev/sdc and your RAID is now /dev/sdd (for example).

Instead, when you expose the the RAID controller to the container, use the /dev/bus device. A smartctrl --scan should show that your MegaRAID device is a /dev/bus/XX something...for example, mine is /dev/bus/12. So in your docker command use --device /dev/bus or if you use docker compose add it under your devices: section.

Unfortunately, I've found that I can't use --device /dev/bus/12 and limit what's exposed to the container, the docker command just doesn't like it because it doesn't show up in the directory structure (if you do an ls -l /dev/bus you'll see that it's not visibly there).

Now, on top of that, for systemctl to work with the MegaRAID in the Scrutiny container (the -d flag) I had to also add the --privileged option or add privileged: true to my docker compose file for the container object. Although it's not a security issue for me, because the system is quite isolated, it may be an issue for some other folks.

In summary, what I've found is that to use Scrutiny as a docker container, if you're using a MegaRAID controller (perhaps others), you must run it in privileged mode. If that's a concern, then perhaps running Scrutiny on the host OS using the Manual Install method would be more appropriate for your situation.

[AnalogJ]

wow, thanks for the detailed troubleshooting notes @PriamX
Would you be willing to add it to the TROUBLESHOOTING_DEVICE_COLLECTOR.md doc? If not, I can add it myself.

Thanks again!

[AnalogJ]

closing this as I've referenced it in the docs. Thanks again for your detailed comment @PriamX !