[Docker Desktop] Investigate Mac CPU performance

[nebuk89]

Tell us about your request
CPU performance on Mac is a long standing issue, we have investigated this in the past but want to dedicate time to re-invetsigate this and try and find a fix

Which service(s) is this request for?
Desktop

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
We keep spiking to 100% CPU and this isn't good, no one likes their laptop taking off

Are you currently working around this issue?
N/A
Additional context
N/A
Attachments
docker/for-mac#1759
docker/for-mac#3499
docker/for-mac#2601

[ghost]

Ben, for purposes of documentation of pain points with Docker on Mac, I wanted to make sure there was a reference back to these complaints.

docker/for-mac#3499

Thanks -Alan

[cculianu]

Please add docker/for-mac#4392 to this list.

[nebuk89]

Thanks @aguedeney and @cculianu - when we move this into in progress I will link back the tickets we are using in for-mac to highlight where we will be discussing with everyone and also where we will be seeking fresh logs for issues :) please feel free to add other cases here as you think appropriate!

[schellj]

Please add docker/for-mac#4064.

[mc0]

The primary issue with CPU that I have seen relates to volume mounts, transfused, and heavy IO workloads. I have observed this a lot of times and inside the VM it is usually transfused using the CPU.

See this on why mounts use lots of CPU. Namely:

A Linux system call such as open or read that accesses bind-mounted files in a container must be.

- turned into a FUSE message in the Linux VFS
- proxied over a virtio socket by transfused
- forwarded onto a UNIX domain socket by HyperKit
- deserialized, dispatched and executed as a macOS system call by osxfs

The entire process then takes place in reverse to return the result of the macOS system call to the container.

[daveisfera]

See this on why mounts use lots of CPU.

This definitely makes sense for bind mounts because it's having to go in and out of the VM to interface with the host filesystem, but I'm still seeing high CPU usage after we switched to using named volumes and that should all be inside of the VM, but I'm still seeing high CPU usage when things are sitting idle, so is there any similar :cached/:delegated tags for named volumes to help with performance issues there?

[m8rge]

This issue is too broad. Some people may suffer from mounts CPU performance, but other have 100% cpu after prune command in this issue: docker/for-mac#4064

[nebuk89]

Thanks to all the responses!
We will start our investigation on this in the next ~2 weeks!
Watch out for incoming questions on some of the for-mac issues. We won't respond to posts in here with logs etc but will use this to share updates as we make progress :)

[alechirsch]

Hope this helps in some way.

I was running into CPU issues as well and noticed something weird happening with volumes.
I created three volumes in my compose file, one for the project directory, one to ignore node_modules, and another to mount a specific node_module.

It works as expected, a file from the specific node_module that I modified locally was copied to the container. Likewise I modified a file in another node_module and the change was not copied to the container. Everything works fine.

The issue came when I did a docker-compose restart, the other node_module that should have been ignore was copied into the container, and my laptop fans were taking off. Every file in node_modules after that point was copied over to the container even though I specified it not to.

I did investigating a few months ago so the reproduction steps might be off, but there was definitely an issue with volumes copying files that it was not supposed to.

 - volumes
    - ./:/src
    - /src/node_modules
    - ./node_modules/express:/src/node_modules/express

[andig]

I would like to highlight that docker/for-mac#3499 (comment) has a concrete suspicion of potentially bugged code that might be easy to investigate (with OCAML knowledge).

The primary issue with CPU that I have seen relates to volume mounts, transfused, and heavy IO workloads. I have observed this a lot of times and inside the VM it is usually transfused using the CPU.

I would also like to add that this is not the only cause. I do sometimes see docker spike to at remain at 100% without any containers running. Definitely no high IO workload in that case, pure "home" usage.

[schellj]

I've also had times where the Docker process was using 100% CPU indefinitely when no containers were running.

[ebriney]

@schellj , can you do a diagnose when it happens? (if the whale menu is responding of course).
I you cannot use the UI, you can still do it with cli command:
/Applications/Docker.app/Contents/MacOS/com.docker.diagnose gather -upload

[schellj]

@ebriney I haven't had any issues with Docker for Mac using a lot of CPU unexpectedly since early May - see docker/for-mac#4064 (comment), but I did a couple of diagnostics when I was - see docker/for-mac#4064 (comment) and docker/for-mac#4064 (comment) (I made some other relevant comments in that issue as well).

[haslo]

I installed the edge build (with spindump) and uploaded this diagnostic:
0D3FCEE0-663F-4A5E-8DA2-1B79B6746B54/20200626175354

The Kubernetes containers were running (with nothing to manage), nothing from my side, and Docker used 50% CPU.

[m8rge]

My Diag Id with spindump 80E081FA-7763-4245-BB3F-15814B31E0BB/20200627174556

[joerobot]

Another diag ID F2F1BBBF-4BF7-4158-BCFA-0C29469D1E33/20200630084358

[andig]

Here's another one with only 5-10% but it still seems like a lot with no containers running on a decent MBP: A913E1F9-1085-4C2B-A79D-4B6862C0EA1E/20200630095026

[xjrcode]

Another diag C39B37CE-DA64-46FE-B4ED-3A714CE9CC62/20200701075247 with spindump version.

Using Big Sur but I feel no differences from Catalina…

[mikepflu]

Another diag: 93B23EAA-7413-4ACB-8278-F008093AD412/20200701154406
I don't think I'm using spindump version, still reading up on this ticket, but my mac is about to take off and I've had a number of crashes lately. I suspect that they're related to docker, but that's just a guess on my part.

[vredeling]

The new edge build works pretty well for me. I can work for quite some time comfortably. CPU use stays at around 25% with a couple of containers running.

But after a while (2 hours or so) or when waking my machine from sleep, com.docker.hyperkit flares up to 200% - 250% and stays there until I stop all containers. With no containers running I'm at 5% cpu.

[JamesonNetworks]

Concur with @vredeling

Edge 2.3.6.0 is working much better for me. I think there is at least one dangling issue, however: Returning from sleep sends the hyperkit process spinning on all CPUs. Stopping the containers does not help. I am able to stop the bleeding be using Restart from the Docker tray icon. I have gRPC-FUSE enabled. Let me know if there is any information I can provide to help diagnose the issue.

[jsharpe]

With gRPC-FUSE enabled I'm getting some operation denied calls when a bazel build in the docker container attempts to create a symlink within a bind mounted directory.

[stephen-turner]

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

[djs55]

@jsharpe thanks for your feedback -- are you able to create and share a small repro case of the problem? If not, next time it manifests, could you upload a diagnostics report and quote the ID on a fresh ticket on https://github.com/docker/for-mac ? Thanks!

[jsharpe]

@djs55 Yes to repro:

git clone https://github.com/abseil/abseil-cpp.git
cd abseil-cpp
docker run \
  -e USER="$(id -u)" \
  -u="$(id -u)" \
  -v $PWD:/src/workspace \
  -w /src/workspace \
  l.gcr.io/google/bazel:2.2.0 \
  --output_user_root=/src/workspace/build_output \
  build //absl/...

Gives output when grpc-fuse is on:

Extracting Bazel installation...
FATAL: failed to create installation symlink '/src/workspace/build_output/a08c2e4811c846650b733c6fc815a920/install': (error: 1): Operation not permitted

When using osxfs you will see bazel build the software.

This is on edge 2.3.6.0

[JamesonNetworks]

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

@stephen-turner Alright, got an ID for the diag:
12F1947A-2753-47C1-A19E-A206946809E5/20200910125157

This is from version 2.3.6.0

Shutting down all of the containers seems to return the hyperkit process to normal. So this may be more a function of what I'm doing in containers (django services). However, I still do not believe returning from sleep should cause that kind of a meltdown, but at least the process does eventually calm down now.

After sitting with this issue for a couple of weeks, I'm noticing the high CPU calm down after about 30-40 mins. I assume this is actually expected behavior and not an issue. Wanted to come back here and update in case anyone was looking into it. I believe 2.3.7.0 solves all my CPU woes.

I appreciate all that the docker team is doing, thanks for the good work and awesome software everyone!

[j13k]

I just thought I'd report a positive outcome from upgrading to the 2.3.7.0 Edge release, following advice in the comments here.

For three days since upgrading from Stable 2.3.0.4 to 2.3.0.5, I had constant high CPU (and associated fan noise), despite virtually idle container activity.

Following the upgrade, I can report that hyperkit %CPU is sitting just under 20%:

[vredeling]

@vredeling @JamesonNetworks If you can reproduce this issue on waking the computer, please upload a diagnostics bundle and paste the GUID here. Thank you.

I've not encountered this issue any more since I moved to Docker Desktop 2.3.7.0

[djs55]

The latest CPU-related changes from the edge 2.3.7.0 have now made it to stable 2.4.0.0 for Mac and Windows. If you get a chance to try this, let us know how it goes. If there are still CPU problems please provide as much information as you can: ideally a repro example and a set of diagnostics.

Thanks for all your help so far!

[pkdetlefsen]

I just upgraded to 2.4 for Mac and CPU usage is a lot better for the app I'm running! The code is mounted inside the container during development and the app checks all relevant files (769 atm.) for changes every 5 seconds in order to auto reload. This usually caused a good spike in CPU usage up to around 50% CPU. Now there's a consistently low usage, mostly below 5%, when checking every second!

So I can confirm that IO performance and CPU usage have improved a lot in this update in my case.

Tested on: MacBook Pro (13-inch, 2017, 2,3 GHz Intel Core i5)

[andig]

I'm still getting 2.3.7.0 as latest release, both on stable and edge channel. What's needed to get 2.4 to install?

[rfay]

@andig 2.4.0.0 is latest in Stable. 2.3.7.0 was never in Stable and never will be. See the Stable Release Notes for how to download any stable version. Note that you will lose all containers and volumes switching from edge to stable.

[sdbruder]

I've upgraded to 2.4.0.0 and got a lot of ERROR: for <container-name> UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60) in our complex local dev stack, we instructed all devs to revert to the previous version which solved the issue (for now).

[aleks-rope]

I have updated Docker to 2.4.0 the CPU usage seems to be lower (now only around 10-12%) and everything has been running smooth so far on my MacBook Pro 2.8 GHz Quad-Core Intel Core i7.

[KhaledMohamedP]

Yes! this is a giant improvement!!!
Thanks everyone for working on this upgrade 🎊 🙏

[PhilETaylor]

And even works on MacOS Big Sur ;-)

[rpkoller]

hm the 2.4.0 build was behaving well so far without any fans running extensively all of a sudden without an apparent reason. but today after starting a project and leaving the computer for half an hour, just started up a project in DDEV right before, when i returned the fans were running on max and the cpu was spiking without a stop. :/

setup: macos 10.13.6 on a mbp 13" early 2011 8GB RAM with docker desktop 2.4.0 running a Drupal site with DDEV. i've sampled the .com.docker.backend process which was constantly at +/- 100% and com.docker.hyperkit which was around 10 to 30%... the samples:
samples.zip

p.s. and according to the deprecation notice for high sierra on the first startup of the minor 2.4.0 release it was the last release for macos 10.3 so no 2.4.x bug fix release anymore?

[nebuk89]

Hey all,

We have now shipped the changes from our investigation over the last 3 months as part of the 2.4 Desktop release.
This does not mean we have got all of the bugs potentially causing this (we fixed a new one yesterday! ) but it is the point where we are no longer going to be driving a significant investigation into multiple potential root causes to this.

We will do a blog post in a few weeks detailing the things we found/changes we made to give you some insight into what we did and why.

If you are still having issues, I would ask that you create new tickets on for-mac rather than comment on old tickets as we are going to close these to keep our backlog a bit tidier and not mix up the messages (and it is easier for us to get signal on the scale of issues this way).

Thanks again for your patience and an even bigger thank you to all of you who took the time to test builds & upload diagnostics!

[sashaaKr]

@nebuk89 can you share link to blog post if it was written?

[stephen-turner]

@sashaaKr Unfortunately we never got round to writing the blog post.