CPU Usage Still High on Docker 3.0.0

[pjjanak]

Seems like the issue described in #5044 is persisting even after the latest upgrade. I have removed unneeded shares (as mentioned in #3499) but that has not fixed the problem.

• [x ] I have tried with the latest version of my channel (Stable or Edge)
• [x ] I have uploaded Diagnostics
• Diagnostics ID: 5F7920E2-CFFC-44A8-9509-AE9C0DDF6CE2/20201210183007

Expected behavior

com.docker.hyperkit takes up a normal amount of CPU

Actual behavior

com.docker.hyperkit is constantly running at > 100% CPU

Information

Please, help us understand the problem. For instance:

• Is it reproducible? - Yes
• Is the problem new? - No (probably not given the amount of times this has been reported)
• Did the problem appear with an update? - Yes after the 3.0.0 update the problem resurfaced
• macOS Version: 10.15.7

Steps to reproduce the behavior

1. Start docker

[ricristian]

Now it's worst

OS: Catalina 10.15.7
Diagnostic ID: 152E874F-49F1-48A0-9A4E-30D9B88D56EA/20201210203925
File sharing is not working anymore if gRPC is enabled.

Settings:
Experimental: all disabled

docker-compose.yml

version: "2"

services:
  elasticsearch:
    container_name: elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.2.0
    volumes:
      - "./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro"
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk

  kibana:
    container_name: kibana
    image: docker.elastic.co/kibana/kibana:7.2.0
    volumes:
      - "./kibana.yml:/usr/share/kibana/config/kibana.yml:ro"
    ports:
      - "5601:5601"
    networks:
      - elk
    depends_on:
      - elasticsearch
  filebeat:
    container_name: filebeat
    hostname: filebeat
    user: root
    image: "docker.elastic.co/beats/filebeat:7.2.0"
    volumes:
      #Mount the filebeat configuration so users can make edit
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      #Mount the prospectors directory. Users can in turn add propspectors to this directory and they will be dynamically loaded
      - "../logs/:/usr/share/logs/"
    networks:
      - elk
    command: filebeat -e
    restart: on-failure

networks:
  elk:
    driver: bridge

EDIT: after 10-15 minutes of continuously "burning" CPU usage it seems that com.docker.hyperkit is eating between 8-30% CPU with 3 containers running ( kibana, Elasticsearch, filebeat )
That spike of 10-15 minutes of high CPU usage is triggered by docker-compose which creates 3 containers.

[stephen-turner]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

[ricristian]

@stephen-turner I can confirm that by default when all containers are stopped the load is small under 10%
When I'm starting containers with docker-compose ( in my case ) for 10-15 minutes the load increases to between 100-500% and after those 10-15 minutes the load decreases back to 10% which is really great.
So the high load appears only when I'm trying to start containers and after that load decreases to more than decent

EDIT:

I've made now more tests and I've discovered following:
UseCase 1. When I try to start some containers with images that are not already downloaded locally the entire time ( download + build + run ) (about 10-15 minutes) cpu load increases

After those 10-15 minutes everything is downloaded + started cpu decreases to 10%

UseCase 2. When I try to start same containers that are already downloaded cpu load increases for few seconds ( under 1 minute ) probably until containers finish loading some settings

After those few seconds cpu load decreases again to 10% and stays in 10-30%

All the tests were realised using docker compose mention earlier ( 3 containers - es - kibana - filbeat )

Maybe I was to hurry to report this but if we ignore the start-up. time of containers which generates high cpu load this release improves overall cpu usage.

[stephen-turner]

@ricristian Thanks, that's very interesting. I'm still surprised the images take quite so long to download: are they enormous?

[ricristian]

@stephen-turner it's highly probable that it is caused by my network ... sometimes bandwidth is not that high ...

[pjjanak]

@stephen-turner - I noticed this right after docker restarted after update, so no containers were running. I also don't use kubernetes on this machine, so don't have any running on here ever. I will say that things have quieted down a little since reporting, but my usage is still hovering around 70-80% with no containers running.

[osi]

I'm seeing the same observation as #5070 (comment) w/3.0.1 on Big Sur, 11.0.1 (20B50)

[pposca]

I have the same issue, but it only happens when there is a volume bound:

• Container with volume bounded: ~20% CPU usage
• Same container without volume bounded: ~4% CPU usage

[dawi]

After update to 3.0.0 the cpu usage is around 80 % without running any containers.
With running containers it's constantly over 200 %.

[alexcouret]

Having the same issue, upgraded to v3 and CPU got back to >200% usage.
Diagnostic ID: 74807466-79B8-4824-8035-608D77708609/20201214144155

Version 2.3 is the latest version that doesn't have this issue for me.

[Patrick-Remy]

HyperKit takes about 200% CPU (no K8s) on 3.0.1 without any containers running – exactly the same as at 2.5.0.

[Ky6uk]

Yep, it's back, but not all the time as it was on v2.5. Sometimes it's eating all attached CPUs, sometimes not. I have no idea about the condition, it looks random at the moment.

C1E3D580-6312-45F9-8E7F-B7D86211FCF9/20201215122501

[YotillaMarco]

I habe the same problem. ~170% CPU usage without any container running. Version 3.0.1 on macOS 10.15.7

[cyberatz]

Similar issue here.
I stopped all container but one - watchtower - which is set to run at 15min intervals.

CPU usage of docker:

CPU usage of the container:

These are the hyper kit process stats:

My gut feeling currently is that there's something running aggressively inside the docker created VM (~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw) that is eating up cycles - but this is just a hypothesis at the moment and I've no idea how to test that theory just yet.

Diagnostic ID:
55ADF900-D456-4893-BC2B-812E6F6622DD/20201218084210

EDIT:
Upon further troubleshooting (and after killing all Kubernetes instances as well. The thing that seems to be using the more CPU in the docker-desktop vm is etcd (etcd --advertise-client-urls=https://192.168.65.3:2379 --cert-file=/run/config/pki/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/etcd --initial-advertise-peer-urls=https://192.168.65.3:2380 --initial-cluster=docker-desktop=https://192.168.65.3:2380 --key-file=/run/config/pki/etcd/server.key --listen-cli)

so not sure if something can be tweaked here to see if it's the culprit (assuming my hypothesis of the VM being the issue). That said, it could also be something with HyperKit which could be causing the issues here.

EDIT:
Screw that. I blew away the existing docker-desktop VM and rebuilt it. With only one container running (watchtower) I'm down to ~3% CPU utilization at idle which I think is totally acceptable.
Perhaps just rebuilding the VM could help folks here too. (stopped Docker, deleted Docker.raw, restarted Docker).
Keep in mind that this could be super destructive - so I guess - only try this at home.

[n00bvn]

I've just updated to the latest version which is 3.0.2 but hyperkit cpu usage still at more than 200%.
I'm using Big Sur 11.1.

[NaeemSohrabi]

Hi,
i have the same problem of high cpu usage abut 320 to 350 %
OS : Big Sur 11.1.
Docker: 3.0.2

docker-compose.yml

version: "3.2"

services:
  node:
    build:
      dockerfile: node_watch.Dockerfile
      context: ./logger
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    volumes:
      - ./logger/public/:/var/www/logger/public/
      - ./logger/resources/:/var/www/logger/resources/
      # - ./logger/node_modules:/var/www/logger/node_modules
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    container_name: node
    networks:
      - mavara-network

  laravel_echo_server:
    build:
      dockerfile: Dockerfile
      context: ./echo
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    restart: unless-stopped
    container_name: laravel_echo_server
    volumes:
      - ./echo/laravel-echo-server.json:/opt/echo/laravel-echo-server.json
      - ./echo/certs:/opt/echo/certs/
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro

    ports:
      - 6001:6001
    depends_on:
      - php
      - redis
    networks:
      - mavara-network

  mysql:
    image: ${REGISTRY_URI}/mysql:5.7
    container_name: mysql
    restart: unless-stopped
    volumes:
      - ./mysql/init.sql:/docker-entrypoint-initdb.d/init.sql
      - ./mysql/data:/var/lib/mysql
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    env_file: ./mysql/.env
    networks:
      - mavara-network

  nginx:
    image: ${REGISTRY_URI}/nginx:1.19.2
    container_name: nginx
    restart: unless-stopped
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
      - ./logger/:/var/www/logger
      - ./nginx/cert:/etc/nginx/certs/
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    ports:
      - 80:80
      - 443:443
    depends_on:
      - php
    networks:
      - mavara-network

  redis:
    image: ${REGISTRY_URI}/redis:6.0.8
    container_name: redis
    restart: unless-stopped
    networks:
      - mavara-network
    volumes:
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro

  php:
    restart: unless-stopped
    build:
      dockerfile: php_dev.Dockerfile
      context: ./logger
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    container_name: php
    hostname: php
    environment:
      SOURCE_DIR: ${SOURCE_DIR}
      HOST_USER: ${HOST_USER}
      HOST_ADDR_NAME: ${HOST_ADDR_NAME}
    volumes:
      - ./logger/local.ini:/usr/local/etc/php/conf.d/local.ini
      - ./logger/:/var/www/logger/
      - ~/.ssh/:/root/.ssh/
      - /etc/sudoers:/var/www/logger/sudoers
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    env_file:
      - ./logger/.env.dev
    depends_on:
      - mysql
      - redis
    networks:
      - mavara-network

  snmp-server:
    build:
      dockerfile: Dockerfile
      context: ./snmp
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    restart: unless-stopped
    container_name: snmp-server
    volumes:
      - /proc:/proc
      - ./snmp/snmpd.conf:/etc/snmp/snmpd.conf
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/timezone:/etc/timezone:ro
      - /Users/naiemsohrabi/Documents/Project/Docker/Mavara/localtime:/etc/localtime:ro
    networks:
      - mavara-network
  reporter:
    restart: unless-stopped
    build:
      context: ./pdf-reporter
      args:
        REGISTRY_URI: ${REGISTRY_URI}
    container_name: reporter
    networks:
      - mavara-network
# Docker Networks
networks:
  mavara-network:
    driver: bridge

[ishmayeck]

Seeing the same with 3.0.3 on macOS 11.1. Just did a complete clean install of Docker Desktop, no containers running at all, and com.docker.hyperkit is hovering around 200% CPU usage.

Diagnostic ID: 0789f91c-d606-44fe-b1b9-c4363bbe7825/20201221191457

[fatroom]

Same problem. OSX 11.1 + docker 3.0.3 eats 130% CPU. Diag ID: 22478378-9FD7-4163-ACB5-21C629CD6CCC/20201228181903

[casprwang]

I suspect this is related to docker/roadmap#12

Also, rolling back to 2.3.0.5 worked for my Big Sur(20C5048l).

[richyen]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

I recently upgraded to 3.0.1 on Mac OS Catalina, and I am seeing hyperkit consume 86-100% of cpu even while there are no containers running (docker ps -a is blank)

[jasonlancaster]

Similar issues here. I upgraded from Catalina/3.0.1 -> Big Sur 11.1/3.0.3 and am seeing hyperkit CPU in constant load again ~30%. Disabling turbo boost seems to be the only method to control this.

[jeitnier]

Same here, Docker version 3.0.4 (51218) on Big Sur 11.2 Beta and hyperkit consistently ~206% CPU.

[fuxingloh]

TLDR: The same issue for me but figured external Docker daemon connection caused the high CPU usage.

Experienced the same issue when I upgraded to docker 3.0.4 from 3.0.3. Using macOS Catalina, com.docker.hyperkit was using 150%+ CPU. Stopped everything, restarted, pruned and downgraded to 2.5.1 all didn't help and CPU usage was still around 150%+. I was using IntelliJ with Docker plugin at that time with IntelliJ connected to Docker daemon (was connected during the upgrade as well) I assumed external connection caused the high CPU. Restarted IntelliJ and no issue after that.

You can easily reproduce this by running Docker through JetBrains products and have it attached during the restart or upgrade. The CPU usage will sit at around ~50-175%.

[richyen]

@pjjanak @ricristian Can you clarify whether you're also seeing high CPU load when no containers and no Kubernetes are running, or only when there are running containers? Those would likely have different causes. Thanks.

I recently upgraded to 3.0.1 on Mac OS Catalina, and I am seeing hyperkit consume 86-100% of cpu even while there are no containers running (docker ps -a is blank)

The worst part of this is that if I forget to quit Docker for Mac, and I walk away from my laptop, I'll come back to a laptop with 7% battery, even when it's plugged in the whole time.

[sidler]

TLDR: Same issue for me but figured external Docker daemon connection caused the high CPU usage.

Experienced the same issue when I upgraded to docker 3.0.4 from 3.0.3. Using macOS Catalina, hyperkit was using 150%+ CPU. Stopped everything, restarted, pruned and downgraded to 2.5.1 all didn't help and CPU usage was still around 150%+. I was using IntelliJ with Docker plugin at that time with IntelliJ connected to Docker daemon (was connected during the upgrade as well) I assumed external connection caused the high CPU. Restarted IntelliJ and no issue after that.

This. When restarting Docker for Mac (e.g. due to an update) a previously opened Docker-Service connection from IntelliJ (here: PHPStorm) tried to reconnect instantly, causing high cpu loads on hyperkit. Stopping the Docker-Service Plugin in PHPStorm stopped the cpu spike and everything went back to normal cpu load.

[ishmayeck]

Indeed, this was my issue as well! Disconnecting and reconnecting the Docker plugins in WebStorm does the trick, and brings they hyperkit process to a much more reasonable <10%. Good find!

[westonruter]

Disconnecting and reconnecting the Docker plugins in WebStorm does the trick

By disconnecting and reconnecting, does this refer to disabling and enabling the Docker plugin?

[kamok]

@hikerspath and @diurnalist can you explain what it means to "allow full disk access to Docker". Are you talking about the configuration dials on this config? https://docs.docker.com/docker-for-mac/space/

Edit: Sadly, did nothing for me. I'm running Kubernetes. Before "Allow Full Disk", was around 100% CPU Usage. Same for now. At least the fan does not turn on anymore at 100%, rather than the 500% it was before 3.0 upgrade.

[travisseitler]

@hikerspath and @diurnalist can you explain what it means to "allow full disk access to Docker".

@kamok they mean this setting in the MacOS System Preferences:

full-disk-access-Apple.Devices.HD.Most.Compatible.mp4

[dokudoki]

I have this issue after upgrading to Big Sur with > 200% CPU without any container running, resetting to default seemed to do the trick for me.

[felipe-tactile]

Another one with >200% CPU without running containers. This is unacceptable as a production release.

[robvanaarle]

I have had this issue for months, I had to restart the containers at least 5 times a day. Last week on Tuesday it got even worse: within 10 minutes after restarting containers, the CPU went to >300%. A co-worker also started experiencing this issue that day. We tried suggestions in this and other topics, but nothing seemed to work. Last Thursday, we both increased resources to 6 CPUs, 6GB memory and 2GB swap. Although we did not expect anything from this, we both did not have high CPU usage of com.docker.hyperkit since then.

[mikenac]

Confirmed with 3.5.1.7 in OS X 11.4 (Big Sur). Giving it less CPUs makes the problem somewhat better, but the fan still goes constantly and things heat up. if you give it more cores, it will eat up more CPU.

[CpuID]

Confirmed with MacOS Big Sur 11.4, using Docker Desktop 3.5.1 (66090) - includes Kubernetes v1.21.2

Resources (in Preferences) set to CPUs 8 Memory 24GB Swap 2GB Disk Image 128GB, all file sharing resources removed. the Docker Engine JSON blob looks like:

{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB"
    }
  },
  "features": {
    "buildkit": true
  },
  "experimental": false
}

If I have gRPC Fuse for file sharing disabled, and Kubernetes disabled I can get it as low as 3.8~% idle CPU.

If I enable Kubernetes, that goes up to 26-27~% idle CPU.

It doesn't look like there's any single process that's the culprit of all the extra CPU? it's a combination of lots of components using a little bit? I see a few items at 6-7~% CPU at various times.

$ docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

top snapshot:

Mem: 2545292K used, 22085460K free, 417160K shrd, 43248K buff, 1435052K cached
CPU:   0% usr   0% sys   0% nic  98% idle   0% io   0% irq   0% sirq
Load average: 0.34 0.16 0.11 1/1011 17680
  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 9004  8999 root     S    2309m   9%   2   0% kubelet --kubeconfig=/etc/kubernetes/kubelet.conf --config /etc/kubeadm/kubelet.yaml --bootstrap-kubeconfig=/
10298 10243 root     S    1069m   4%   6   0% kube-apiserver --advertise-address=192.168.65.4 --allow-privileged=true --authorization-mode=Node,RBAC --clie
16264 16242 root     S    10.1g  40%   2   0% etcd --advertise-client-urls=https://192.168.65.4:2379 --cert-file=/run/config/pki/etcd/server.crt --client-c
10309 10279 root     S     792m   3%   6   0% kube-controller-manager --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf --authorization-k
 1795  1790 root     S    2664m  10%   2   0% /usr/local/bin/dockerd --containerd /var/run/desktop-containerd/containerd.sock --pidfile /run/desktop/docker
12416 12391 root     S     731m   3%   5   0% /coredns -conf /etc/coredns/Corefile
10377 10324 root     S     730m   3%   6   0% kube-scheduler --authentication-kubeconfig=/etc/kubernetes/scheduler.conf --authorization-kubeconfig=/etc/kub
  393   387 root     S     701m   3%   7   0% /usr/bin/vpnkit-bridge --use-kernel-cmdline guest
  916     1 root     S     147m   1%   5   0% /usr/bin/containerd
 1447  1439 root     S    1744m   7%   2   0% /usr/local/bin/containerd --config /etc/containerd/containerd.toml
12479 12458 root     S     731m   3%   7   0% /coredns -conf /etc/coredns/Corefile
11814 11790 root     S     716m   3%   1   0% /storage-provisioner /var/lib/k8s-pvs
  378     1 root     S     104m   0%   5   0% /usr/bin/memlogd -fd-log 3 -fd-query 4 -max-lines 5000 -max-line-len 1024
 2341     2 root     IW       0   0%   1   0% [kworker/1:3-eve]
 8984     2 root     IW       0   0%   3   0% [kworker/u16:0-e]
...

I also tried the new Virtualization framework but idle CPU was higher (35-40%), turned that off and didn't test too much further there.

[stephen-turner]

Kubernetes is known to be a bit wasteful on CPU. We've turned on all the optimizations we can, reduced polling intervals etc., but it's not really designed for desktop use. Other desktop Kube distributions like minikube have the same problem.

[CpuID]

Kubernetes is known to be a bit wasteful on CPU. We've turned on all the optimizations we can, reduced polling intervals etc., but it's not really designed for desktop use. Other desktop Kube distributions like minikube have the same problem.

understood thanks @stephen-turner

curious, is it possible to tune parameters of the kibe-scheduler and kube-controller-manager on Docker Desktop? to try out ideas to see if they help? eg. suggestions like kubernetes/kubernetes#75565 (comment)

[tjmgregory]

MacOS: 11.5.1
Docker: 20.10.7, build f0df350
Docker Desktop: 3.5.2

With no containers running, docker's hyperkit would sit around 7-11% CPU usage.

However, I discovered after enabling the experimental Big Sur new virtualisation framework feature that this process has disappeared, and I think been replaced by a process of the MacOS Virtualisation framework - which with a small postgres container running is only at around 0-3% CPU usage 🎉

Would appreciate someone with greater knowhow to confirm this second process is the replacement of the first, but this is a huge win!

[tecbird]

Docker Desktop 3.6.0 - 200% cpu (2 cores a 100%) idle without starting any containers (on resetted to factory defaults)
-> After Downgrade to 3.5.2 everything looks normal

[leoncider]

Kubernetes is known to be a bit wasteful on CPU. We've turned on all the optimizations we can, reduced polling intervals etc., but it's not really designed for desktop use. Other desktop Kube distributions like minikube have the same problem.

My com.docker.hyperkit consumes >200% CPU without anything Kubernetes related (I'm only running a gitlab container right now)

Docker Desktop 3.6.0

[sandstrom]

I'm running Docker Desktop 3.6.0 (macOS) and have >200% CPU, even though the contains are only running at ~40%. Using the experimental virtualization technology.

Background details (config, output, etc)

[dgileadi]

I'm seeing the same behavior with 3.6.0—above 200% CPU use with no containers running. This is far more than I had with the 3.5.x series in the same state. I tried granting full disk access per earlier in the thread with no improvement.

My Diagnostics ID is C6E21428-588F-41DC-9CDF-33AED72834CC/20210826152010 if it helps.

[dokudoki]

It happened again with 3.5.2 with no containers running. Clean/Purge data seems to do the trick for me.

[leoncider]

not just docker related, but FWIW for whomever might be interested: minikube (v1.22.0-beta.0) w/ the parallels (17.0.1) driver seems to behave a lot better than with hyperkit/virtualbox. about 50% CPU consumption with the cluster & around 6 deployments (compared to 200%+)

seems like there's nonetheless substantial hypervisor overhead (CPU usage within the minikube VM is much lower than 50%), but it is easily bearable, and the laptop does not get hot.

[xavipanda]

1 year later, this stills not fixed... now not a free product anymore. Won't buy license until this bug is fixed.

[docker-robott]

Issues go stale after 90 days of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle stale

[sandstrom]

Not stale, this is still a problem

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked