Move RealCall and RealConnection to loom safe locks

[yschimke]

Main fix for #8284

replace synchronized with ReentrantLock.withLock, and wait/notify with lock.newCondition.signal/await.

Relies on test results from BasicLoomTest

[swankjesse]

What are the best practices when synchronized is only used to guard state, and never used for a blocking or slow call? From our concurrency rules, a bunch of these might be unnecessary?

### Locks

We have 3 different things that we synchronize on.

#### Http2Connection

This lock guards internal state of each connection. This lock is never held for blocking operations. That means that we acquire the lock, read or write a few fields and release the lock. No I/O and no application-layer callbacks.

#### Http2Stream

This lock guards the internal state of each stream. As above, it is never held for blocking operations. When we need to hold an application thread to block a read, we use wait/notify on this lock. This works because the lock is released while `wait()` is waiting.

#### Http2Writer

Socket writes are guarded by the Http2Writer. Only one stream can write at a time so that messages are not interleaved. Writes are either made by application-layer threads or the do-stuff-later pool.

...

#### Per-Connection Locks

Each connection has its own lock. The connections in the pool are all in a `ConcurrentLinkedQueue`. Due to data races, iterators of this queue may return removed connections. Callers must check the connection's `noNewExchanges` property before using connections from the pool.

The connection lock is never held while doing I/O (even closing a socket) to prevent contention.

A lock-per-connection is used to maximize concurrency.

(We definitely need the Http2Writer part of this)

[swankjesse]

Maybe to frame it more precisely:

• what’s the runtime performance difference for locks vs. synchronized on physical threads?
• what’s the runtime performance difference for locks vs. synchronized on virtual threads?

Maybe we can find benchmarks on this.

[yschimke]

I'll create a tracking bug to benchmark. And review and rescope the locks.

I'll land this as the tests show it's a real problem unless we change the locks.

[swankjesse]

I don’t think we need to do any benchmarking work. For my own sake I just want an intuition on which locks to use where.

[yschimke]

@swankjesse Are we saying that the locks are not needed for RealCall, RealConnection, Dispatcher? That these should be reworked?

[swankjesse]

I think there’s two very different ways we use synchronized:

1. To guard in-memory state. For the accesses we acquire the lock, read or write some fields, and release the lock. There’s unlikely to be any contention for these, and if there is it’s likely to be very short-lived.
2. To guard I/O. For these accesses we acquire the lock, perform some potentially-slow I/O operation, and release the lock. These ones are likely to have contention.

I think we definitely want to change 2 to Loom-safe locks. I am unconvinced that there’s a benefit for changing 1. But there is a potential memory + performance cost.

[yschimke]

A dissenter https://medium.com/@apusingh1967/how-synchronized-beats-reentrantlock-simplified-for-a-long-suffering-java-programmer-c83d069a6fc0