VM CPU auto pinning causes slowdowns and stealtime #14133
Comments
|
We threw together a quick script to visualize that problem: Cores:
General rule with this system is:
This core placement puts very latency critical systems on the same (hyper)core, while leaving systems that have no real load on their own core. Even if this was a completely symmetrical CPU and even if all of those cores weren't hypercores, this would still waste resources when those VMs aren't equally loaded. |
|
Thanks for your detailed report! Yeah this was an area of concern originally:
But we are considering option 2 and 3 of your suggestions. |
|
Thank you for your quick response! Would it be possible to get a patch for the 6.1 series that would give us the option to turn this off? Otherwise we'd have to write tooling to repin the VMs based on things like stealtime or cpu pressure. We'd much rather just let the kernel handle it. |
The latest 5.21/stable LTS series does not have this feature (on purpose because it changes the default behaviour) so you could try that. Its more suitable for production purposes anyway as the latest/stable channel is the moving feature release channel and doesn't support downgrades. See https://documentation.ubuntu.com/lxd/en/latest/installing/#installing-release The 6.1 release won't get patches now, but will be replaced by 6.2, 6.3 etc. Hopefully we can land the new settings in one of those 2 releases, but 6.2 is imminent and might not make it in there. |
Required information
Issue description
The introduction of automatic core scheduling has led to significant decrease in performance in our infrastructure, with weird problems that make no sense if you are not aware of this issue, such as:
The current CPU scheduler doesn't seem to understand hardware topology, which is really surprising, considering that many new CPUs are now asymmetric and that on the kernel side much work is being done making sure that workload is put on "the best core for the job" with features like AMD Preferred Core and equivalents or new CPU schedulers like EEVDF.
It seems weird to put these placement decisions in LXD and turn them on by default, without offswitch when LXD does not consider that this might cause significant problems. LXD simply has not enough data, and static round robin placement is simply too simple.
From our perspective this is a significant design error for this feature, and we ask that this feature is either
Additionaly, snaps auto update mechanism has introduced this new feature to our infrastructure (which by itself is fine), and we'd ask you to consider that features like this are being continously applied to real workloads and while not being LTS, should still be at least not harmful.
Information to attach
Our current hardware topology has two L3 domains that have different sizes. Our VMs run typical web application workloads. The core load balancing has put multiple CPU bound cores on one physical core, leading to the weird stealtime above.
The text was updated successfully, but these errors were encountered: