Set fault beacon on drive failure

[behlendorf]

For JBOD style configurations it's desirable to set the device's fault beacon on drive failures. This can be done either through proc interface or through the sg_ses utilities. The relevant zed scripts (cmd/zed/zed.d/io-spare.sh), should be updated to take advantage of this functionality. The tricky bit is to map the device name to an enclosure and slot location.

# Set the fault beacon through proc
$ echo 1 >/sys/class/enclosure/6:0:9:0/000/fault

# Set the fault beacon with sg_ses
$ sg_ses --dev-slot-num=0 --set=ident /dev/sg3
$ sg_ses --dev-slot-num=0 --clear=ident /dev/sg3

[FransUrbo]

udevadm info -q all -p /sys/block/sda can be used to get information about the device, including /sys/device/.... paths. Don't know if it will give out enclosure/slot though.

Have a look at https://raw.githubusercontent.com/FransUrbo/scripts/master/GetDiskInfo.sh (it's huge and kludgy and I'm trying to rewrite it in perl which would make it a little cleaner, but ... :). I do a lot of stuff like that...

I'd love to help (should be able to whip something up quite quickly - I'm bored :), but I don't have something like a enclosure.

[behlendorf]

@FransUrbo That looks like a handy script!

I believe @nedbass figured out how to map the device to an enclosure and slot. Ned could you add the proceedure to this issue so we don't loose track of it. Then we can work it in to the scripts where appropriate.

[chrisrd]

Here's something we use to blink our lights, including the mapping of device to enclosure and slot:

#!/bin/bash
#
# Usage: disk-blink [--off] /dev/sd???
#
# ACHTUNG!
# ALLES TURISTEN UND NONTEKNISCHEN LOOKENPEEPERS!
# DAS KOMPUTERMASCHINE IST NICHT FÜR DER GEFINGERPOKEN UND MITTENGRABEN!
# ODERWISE IST EASY TO SCHNAPPEN DER SPRINGENWERK, BLOWENFUSEN UND POPPENCORKEN
# MIT SPITZENSPARKEN. IST NICHT FÜR GEWERKEN BEI DUMMKOPFEN. DER RUBBERNECKEN
# SIGHTSEEREN KEEPEN DAS COTTONPICKEN HÄNDER IN DAS POCKETS MUSS. ZO RELAXEN
# UND WATSCHEN DER BLINKENLICHTEN.
#
function usage
{
        cat <<END

Usage: $0 [--off] /dev/sd??

END
        exit 1
}

set -e -u

action=--set=locate

dev=$1
[ "${dev}" = --off ] && { action=--clear=locate; dev=$2; }
[ -b "${dev}" ] || { echo 1>&2 "${dev}: not a block device"; exit 1; }

sasaddr=$(
        lsscsi -tg | 
        sed -rn 's/.*sas:(0x[[:xdigit:]]+).*'"${dev//\//\\/}"'[[:space:]].*/\1/ p'
)       
[ "${sasaddr}" ] || { echo "${dev}: SAS address not found"; exit 1; }

#
# Scan all the enclosures for our SAS address
#
for encldev in $(lsscsi -tg | awk '$2 == "enclosu" { print $5 }')
do
        #
        # Note: we discard errors from sg_ses as, at version 1.64 20120118,
        # it prints an error on some enclosures like:
        #
        #  $ sg_ses -j /dev/sg45 > /dev/null
        #  join_work: oi=6, ei=255 (broken_ei=0) not in join_arr
        #
        # See Also: http://thread.gmane.org/gmane.linux.scsi/81514
        #
        slot=$(
                sg_ses -j "${encldev}" 2> /dev/null | 
                egrep "^Slot |^\s+SAS address:" | 
                grep -B1 ${sasaddr} | 
                awk '/^Slot/ { print $2 }'
        )       
        [ "${slot}" ] && break
done
[ "${slot}" ] || { echo 2>&1 "${dev}: enclosure/slot not found"; exit 1; }

#
# Light 'em up
#
sg_ses -D "Slot ${slot}" "${action}" "${encldev}"

exit 0

[behlendorf]

@chrisrd Nice! Thanks for posting this.

[dasjoe]

I've got some enclosures with (LSI) SAS expanders, which are visible in /sys/class/enclosure/.
This makes Slot 01's fault LED light up:
echo 1 > /sys/class/enclosure/1\:0\:21\:0/Slot\ 01/fault
"locate" makes it blink:
echo 1 > /sys/class/enclosure/1\:0\:21\:0/Slot\ 01/locate

[cvoltz]

I'm working on implementing this feature. identify_failed_drive.feature.txt provides a detailed feature description.

[rlaager]

@cvoltz Your description looks solid, except that I disagree that the UID light should be on. I would think only the fault light should be controlled. What is the advantage of adjusting both lights in lock-step? I think the UID should be left for administrator use.

[joehandzik]

@rlaager The UID could certainly be dropped, but there is potential value in a large configuration. With the UID + disk FAULT LED, customers know which chassis AND which disk a bit more easily.

[tonyhutter]

@cvoltz I'm working on pretty much the same thing at LLNL. Have you had any luck with using libstoragemgmt to blink the LEDs? Have you tried it for multipath devices as well?

[cvoltz]

I updated the feature description to remove the references to the UID lights on the drive.

[behlendorf]

For anyone following this issue you may want to checkout the latest master source which now has improved infrastructure for generically managing a devices SES LEDs, 1bbd877. The zedlet's environment will now contain a ZEVENT_VDEV_ENC_SYSFS_PATH variable when the SES sysfs path can be determined. This can be used to easily control the LEDs without any additional dependencies beyond the ses.ko kernel module. See the statechange-led.sh zedlet as an example.

This infrastructure is still being worked on but any feedback or testing on a wider range of configurations and hardware would be welcome.