OSS: delaying writes in a more granular way

[isaac-io]

Objectives

• Slow down writes gradually to prevents stalls
• Stable performance over time

Product Requirements

Existing

The delayed write option in RocksDB is used to slow down writes when flush or compaction can’t keep up with the incoming write rate. This mechanism ensures the write rate will be aligned with the HW capabilities. This mechanism monitors three parameters to identify stalls (number of unflushed memtables, number of files in L0, and the urgency level of pending flushes). In case a threshold of one of them is exceeded, the delayed write limits the write rate to the constant delayed_write_rate value.

New

When delayed write is enabled and active, the write rate is decreased gradually to moderate the write stalls. It currently does not take memory limitation under consideration and does not stop writes when a memory limit is reached. This might cause OOM error, which can be fixed with a memory monitoring feature.

Tests

• High write workload vs. RocksDB
• See decreased performance instead of stalls
• We can test in on HDD on smaller tests
• Look at the memory consumption during the test

Tracked Tasks

• OSS: port the existing delayed write rate mechinsm (with gate flag) #20
• Delayed write: needs_flush_speedup doesn't actually work #21

[Yuval-Ariel]

needs_flush_speedup_ also needs to be fixed - SPDB-655

[isaac-io]

Updated to reflect needs_flush_speedup. I opened child issues and referenced them in the issue descriptions.