Use io_queue_total_delay_sec counter instead of io_queue_delay

[xemul]

The io_queue_delay is the gauge that snapshots a random request delay in the queue. Commit scylladb/seastar@803e790 introduced a replacement for it. The io_queue_total_delay_sec is the increasing counter that accumulates total time requests spent in the queue so far. Applying a rate() on it should give more elaborated delay metrics.

Available since Scylla 4.5.0

[amnonh]

Hi @xemul congrats on your first scylla-monitoring issue,
when reporting an issue please make sure to add all relevant information, specifically, the dashbaord you are looking at and the panel you are looking at (the graph inside the dashbaord).

my guess is that you are talking about scylla_io_queue_delay that is in the detailed dashboard and the I/O queue delay panel

[xemul]

@amnonh , no, it's Advanced dashboard. It contains several plots groupped per-priority class, each class having "queue delay", "bandwidth" and "IOPS" graphics. The metrics I'm taking about is displayed on the "queue delay" ones. Maybe there's something with this metrics on the Detailed dashboard, but I'm unaware of it.