Disable timeout on EBS volumes

[ssuryad-amzn]

Timeouts on EBS volumes can occur due to various reasons, such as
network outages, and can often be recovered after some time.

Timeouts are handled by Linux by remounting the underlying filesystem as
readonly which leads to a bad customer experience.

While the I/O timeout can be specified for the nvme driver via the
kernel commandline or modprobe.d, udev rules are the preferred way of
setting per device timeouts. This is required as local NVMe volumes must have
a finite timeout.

Before Change:

[ec2-user@ip-10-0-1-106 ~]$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     vol0f53eccc69bc1e40e Amazon Elastic Block Store               1           0.00   B /   8.59  GB    512   B +  0 B   1.0     
/dev/nvme1n1     AWSBA029A3CE993A5ED5 Amazon EC2 NVMe Instance Storage         1          75.00  GB /  75.00  GB    512   B +  0 B   0

[ec2-user@ip-10-0-1-106 ~]$ grep ^ /sys/block/nvme*/queue/io_timeout
/sys/block/nvme0n1/queue/io_timeout:4294966296
/sys/block/nvme1n1/queue/io_timeout:90000

[ec2-user@ip-10-0-1-106 ~]$ cat /sys/module/nvme_core/parameters/io_timeout 
4294967295

After Change:

[ec2-user@ip-10-0-1-106 ~]$ sudo nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     vol0f53eccc69bc1e40e Amazon Elastic Block Store               1           0.00   B /   8.59  GB    512   B +  0 B   1.0     
/dev/nvme1n1     AWSBA029A3CE993A5ED5 Amazon EC2 NVMe Instance Storage         1          75.00  GB /  75.00  GB    512   B +  0 B   0

[ec2-user@ip-10-0-1-106 ~]$ grep ^ /sys/block/nvme*/queue/io_timeout
/sys/block/nvme0n1/queue/io_timeout:4294967288
/sys/block/nvme1n1/queue/io_timeout:90000

[ec2-user@ip-10-0-1-106 ~]$ cat /sys/module/nvme_core/parameters/io_timeout 
30

As you can see, the nvme driver was initialized with the default 30s timeout. The udev rule for the EBS volume has been applied successfully as well as the already present rule for local NVMe storage.