Improve average performance of DefaultLongTaskTimer for out-of-order stopping

[fogninid]

The current implementation of DefaultLongTaskTimer optimizes for O(1) task starting, but performs poorly when stopping tasks that are not at the beginning of its internal queue (the oldest ones).
At the worst case, when calling stop immediately after starting, the stop call is currently expected to require O(N) operations.
Depending on the distribution of task lifetimes, the average case would be O(1) only for applications that stop tasks in exactly the same order as they were started; applications completing out-of-order and with unbiased lifetime would experience O(N) average.

Task stopping should not have any intrinsic difference to starting: both action are expected to be performed on application threads, and for a well-functioning application (that is not leaking of piling-up tasks) every call to start is matched by exactly one call to stop.

[pivotal-cla]

@fogninid Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-cla]

@fogninid Thank you for signing the Contributor License Agreement!

[shakuzen]

Thanks for the pull request. What you wrote makes sense. Still, I wanted to verify with some JMH benchmarks so we can put some numbers behind it and have them in place for checking any future changes that would affect performance around this. I made #5595 to add JMH benchmarks.

[shakuzen]

Sharing results from my MacBook Pro M1 with the benchmarks in the linked PR with 10,000 active samples and stopping a random sample. As expected, start is slower but the overall time to start and stop on average (with a random sample) is better.

Before

Benchmark                                   Mode  Cnt   Score   Error  Units
DefaultLongTaskTimerBenchmark.start           ss  200   0.495 ± 0.064  us/op
DefaultLongTaskTimerBenchmark.startAndStop    ss  200  15.351 ± 2.508  us/op
DefaultLongTaskTimerBenchmark.stopRandom      ss  200  14.784 ± 2.584  us/op

After

Benchmark                                   Mode  Cnt  Score   Error  Units
DefaultLongTaskTimerBenchmark.start           ss  200  1.002 ± 0.116  us/op
DefaultLongTaskTimerBenchmark.startAndStop    ss  200  6.338 ± 0.577  us/op
DefaultLongTaskTimerBenchmark.stopRandom      ss  200  5.154 ± 0.608  us/op

[jonatan-ivanov]

Thank you for the PR, I really like this.
Not necessarily in this PR but I' also wondering if we should add a JCStress tests on top of #5595.

[jonatan-ivanov]

Fyi: I polished the PR a bit and also resolved the comments above and rebased it to main (otherwise the build failed because of the Prometheus integration tests).

[jonatan-ivanov]

@fogninid Thank you for the contribution!