add feature-flagged support for multi-threaded Tokio runtime

[hawkw]

This branch enables feature-flagged support for using Tokio's
multithreaded runtime when the proxy is given more than two CPU cores to
run on. The proxy's understanding of the system's number of CPUs
reflects cgroups limits, so docker CPU limits are taken into
consideration.

The multithreaded runtime offers significantly better performance under
load when multiple CPU cores are available. Here's benchmark results
from a run on my 24-thread Ryzen 3900X system, where the proxy, the load
generator, and the test server were all given 8 cores:

eliza on noctis in ~
:; nproc --all
24

Note that the locking in multiple metrics layers doesn't really seem
to have a big impact, as shown by the run with metrics disabled. There
might be some improvement in high tails from further optimizations to
the metrics code, though.

If only a single core is available to run a worker on, we will construct
Tokio's basic_scheduler, rather than a threaded_scheduler with a
single worker thread. This is because there is some overhead introduced
by the threaded scheduler, which results in worse performance when only
1 core is available:

I'd like to fix this upstream in Tokio, by having Runtime::new() and
the macros intelligently select the basic scheduler when only 1 core
is available, but this works for now.

Finally, a bug in the handle time metrics layer results in a deadlock
when used with the multithreaded runtime. Therefore, I've disabled this
metric for now --- my understanding is that nothing consumes it at the
moment. If it's needed, we can fix the bug and put it back.

[olix0r]

The PR description seems to indicate that this adds a feature flag but I don't think it does?

[olix0r]

I don't think we should dedicate an entire core to the admin thread... That's a lot of resources for something we shouldn't be doing much work on. The real goal of the separate admin runtime is to ensure we can still interact with the proxy if, for instance, the main runtime isn't healthy. I think we're best off keeping it simple and treating it as if the admin thread needs no dedicated resources.

[hawkw]

Mmm, yeah, that makes sense. My thought process was that if forwarding has more workers, load on the admin runtime for e.g. discovery & DNS lookups might increase. I think the long term solution might be to just move that work off of the admin runtime, and have it only run metrics/tap/trace endpoints/etc. Then we can probably feel confident not giving it a whole core.

Edit: whoops, looks like service discovery lookups are already run on the main thread, must have misremembered.

[olix0r]

Ah, nevermind I see that it is feature-flagged. We can merge and roll forward with tweaks.