Intermittent Failed to acquire key Errors in Concurrent RateLimiter Usage with FakeRelativeClock

[NOBLES5E]

Test Code:

Repo here: https://github.com/NOBLES5E/governor-bug

fn main() {
    let clock = FakeRelativeClock::default();
    let quota = Quota::per_second(nonzero!(2500u32));
    let lim = Arc::new(RateLimiter::hashmap_with_clock(quota, clock));
    let ms = Duration::from_millis(1);
    let key = Bytes::from("conflict_key");

    crossbeam::scope(|scope| {
        for _ in 0..10 {
            let key = key.clone();
            let lim = lim.clone();
            scope.spawn(move |_| {
                for _ in 0..250 {
                    let ret = lim.check_key(&key).expect("Failed to acquire key");
                    lim.clock().advance(100 * ms);
                }
            });
        }
    })
    .unwrap();
}

Observed Behavior:
Occasionally, the test fails with an error similar to:

Failed to acquire key: NotUntil { state: StateSnapshot { t: Nanos(400µs), tau: Nanos(999.6ms), time_of_measurement: Nanos(60.5008s), tat: Nanos(60.5008s) }, start: Nanos(0ns) }

Check https://github.com/NOBLES5E/governor-bug/actions/runs/12728781775/job/35479772416 for example.

Expected Behavior:
Given the rate limiter is configured with a quota of 2500 requests per second and each thread performs 250 requests with a total simulated time advancement that should comfortably accommodate the quota, the check_key method should not fail.