com.docker.supervisor 100% cpu usage #2923

felixbuenemann · 2018-05-20T09:56:30Z

After recently updating to 18.05.0-ce-mac66 (24545) on macOS 10.13.4 (17E202) I noticed today that the com.docker.supervisor process had started to have 100% cpu usage over night.

I have uploaded a stack-trace of the com.docker.supervisor process when it was eating all cpu in the following gist: https://gist.github.com/09b8f0a0d7c960e06b671e4a9faab13a

The problem could be fixed by killing the supervisor process and restarting Docker for Mac.

felixbuenemann · 2018-05-20T10:34:27Z

Unfortunately I didn't export a CPU-Time analysis of the stack-trace, which would have been more useful in this case. I'll post one, if the issue re-appears.

Joshfindit · 2018-05-23T13:21:43Z

Having a similar issue, but mine hovers between ~95% and ~105%.

OSX 10.12.6 (16G1114)
Engine: 18.05.0-ce-mac66 (24545)

asoorm · 2018-05-23T19:30:04Z

My machine is about to take off I think!

$ docker version
Client:
 Version:      18.05.0-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   f150324
 Built:        Wed May  9 22:12:05 2018
 OS/Arch:      darwin/amd64
 Experimental: true
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.05.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.10.1
  Git commit:   f150324
  Built:        Wed May  9 22:20:16 2018
  OS/Arch:      linux/amd64
  Experimental: true

BretFisher · 2018-05-24T02:31:51Z

Same issue here, Diagnostic ID 63E96CD0-0C00-47D4-8722-55299AA803AC/20180523-222626. 100% CPU (single core) with no containers running since re-starting Docker and let it sit idle a few hours.

akimd · 2018-05-24T05:01:01Z

We believe that this issue was identified and fixed recently (https://github.com/docker/pinata/pull/9355).
Could you try with this version? https://download-stage.docker.com/mac/pr/24648/Docker.dmg

vegasbrianc · 2018-05-24T08:13:36Z

I also submitted a Diag yesterday for the same incident.

felixbuenemann · 2018-05-24T10:21:05Z

@akimd Is the docker/pinata repo private? I'm getting a 404 if I try to view the linked PR or repo.

akimd · 2018-05-24T10:52:02Z

Yes, it is, sorry about that.

sjwl · 2018-05-24T18:54:54Z

not just cpu, but also network is surging

BretFisher · 2018-05-24T19:35:22Z

@akimd looks like a newer version 24677 is out, I'm testing that today.

ear · 2018-05-25T07:44:07Z

I have been experiencing this issue (twice in two days) with Version 18.05.0-ce-mac66 (24545).

By the way @BretFisher how do you get 24677? The release notes link to 24545.

akimd · 2018-05-25T07:49:38Z

I gave a link to 24648 above. Using random builds by playing with the numbers over there is asking for trouble, be sure to use only versions one of us pointed to.

BretFisher · 2018-05-25T19:19:47Z

So far everything is fine on 24700

akimd · 2018-05-28T07:53:23Z

Good. Let's close this. Thanks all!

deitch · 2018-06-07T08:16:16Z

When does that build/those fixes get incorporated into the next Edge release?

bertolo1988 · 2018-08-03T00:40:55Z

Same issue here on Version 18.06.0-ce-mac70 (26399).

rkingon · 2018-10-14T18:03:57Z

yikez, this is killing me! I am on Version 18.06.1-ce-mac73 (26764)

I'm on battery life right now, was at 80%, walked away to get coffee and apparently it started as my machine went into sleep mode? I came back 20minutes later with 15% battery!

I absolutely love docker and have integrated it into all my workflows. Hope this resolves quickly :(

tatwater · 2018-11-02T01:44:19Z

Just caught this using 226% of my CPU, draining my battery like crazy, same release as rkingon above.

perhallstroem · 2018-11-02T12:17:54Z

I, too, am observing this on 18.06.1-ce-mac73. The described behaviour appears to commence without any overt action taken by any operator of the computing system. A typical scenario is that one retires in the eventide with a normal device—to the touch somewhat lukewarm—and returns on the morrow only to find a blistering gale issuing from the ventilation apertures. On further inspection, one then sees that the com.docker.supervisor process greedily performs its enigmatic operations and that a stream of packets, in tirelessness commensurate, is being transmitted across (as an observer above has noted) what appears to be a local interface: for such traffic is seemingly undisturbed by the disabling of all external network interfaces. Further, having observed the above-described process continuing uninterrupted for hours, one assumes that it is unceasing and therefore tires rather than awaits the conclusion of that which is ostensibly unceasing.

An interim measure that I have adopted is to terminate the offending process (that is, com.docker.supervisor). Such an action could, peradventure, give rise to a slight degradation in the feature richness manifested by the application components, however.

By which I mean +1.

benlangfeld · 2018-11-07T18:49:35Z

I'm experiencing the same issue. Diagnostic ID 7F2CE4A0-332F-4962-AB56-D21435E7EC0F/20181107184649.

benlangfeld · 2018-11-07T18:50:16Z

@akimd Could this please be re-opened? The issue is not fixed.

rkingon · 2018-11-07T18:51:36Z

100% not fixed -- happens to me almost every day! i'm pretty sure I've lost life out of my computer from letting it heat up and sit at 100+% cpu =/ (and not realizing it)

craigsansam · 2018-11-14T10:57:16Z

Also experiencing this issue here as well - Diagnostic ID 1C1DCE3A-775D-4ACD-9BBC-234420548FC0/20181114105459

jasonkarns · 2018-11-30T14:25:19Z

Same. still occurring for me: 2.0.0.0-mac81 (29211), Engine: 18.09.0

ltm · 2018-11-30T19:56:56Z

I'm seeing this as well on Docker version 18.09.0. The supervisor is sending 100,000s of (NTP?) packages per second to my local launchdns daemon:

$ sudo tcpdump -i lo0 -k NP -c 1000 -X
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 262144 bytes
14:36:00.865211 pid com.docker.supe.18169 svc BE IP localhost.51788 > localhost.55353: UDP, length 48
	0x0000:  4500 004c 7da9 0000 4011 0000 7f00 0001  E..L}...@.......
	0x0010:  7f00 0001 ca4c d839 0038 fe4b 2201 81ec  .....L.9.8.K"...
	0x0020:  0000 0000 0000 0000 4c4f 434c dfac 0fa0  ........LOCL....
	0x0030:  dd7e 0264 dfac 0fa0 dd7b 84db dfac 0fa0  .~.d.....{......
	0x0040:  dd7e 0264 dfac 0fa0 dd7e 0264            .~.d.....~.d
[...]
14:36:00.874709 pid launchdns.40263 svc BE IP localhost.55353 > localhost.51788: UDP, length 48
	0x0000:  4500 004c f5b3 0000 4011 0000 7f00 0001  E..L....@.......
	0x0010:  7f00 0001 d839 ca4c 0038 fe4b 2201 8103  .....9.L.8.K"...
	0x0020:  0000 0000 0000 0000 4c4f 434c dfac 0fa0  ........LOCL....
	0x0030:  dfea 5f84 dfac 0fa0 dfe5 a78f dfac 0fa0  .._.............
	0x0040:  dfea 5f84 dfac 0fa0 dfea 5f84            .._......._.
1000 packets captured
4226 packets received by filter
0 packets dropped by kernel

Instruments shows the following as the heaviest stack trace:

69279  com.docker.supervisor (18169) :0
 7472  <Unnamed Thread> 0x275557 :0
 6700  github.com/docker/pinata/mac/backend/ntp.NewServer.func1
 6684  github.com/docker/pinata/mac/backend/ntp.(*Server).Serve
 2910  net.(*UDPConn).WriteToUDP
 2882  net.(*UDPConn).writeTo
 2822  net.(*netFD).writeTo
 2787  internal/poll.(*FD).WriteTo
 2621  syscall.Sendto
 2589  syscall.Syscall6
   65  runtime.exitsyscall
   17  runtime.exitsyscallfast
    1  runtime.exitsyscallfast_reacquired

ltm · 2018-11-30T20:22:09Z

It should be noted that launchdns simply echos invalid DNS requests with bytes 3-4 set to 0x8103:

while (quit == 0) {
        int n = recvfrom(sd, msg, RECV_SIZE, flags, &caddr, &len);

        ...

        } else {
                // NXDomain
                msg[2] = 0x81;
                msg[3] = 0x03;
        }

        sendto(sd, msg, n, flags, &caddr, len);
}

The mystery to me is why the Docker supervisor is sending NTP packets to a DNS server.

kivikakk · 2019-01-21T02:47:55Z

@ltm:

The supervisor is sending 100,000s of (NTP?) packages per second to my local launchdns daemon

Nice find! I'm seeing this exact same behaviour:

13:42:45.356693 pid com.docker.supe.1013 svc BE IP localhost.57007 > localhost.55353: UDP, length 48
	0x0000:  4500 004c e22d 0000 4011 0000 7f00 0001  E..L.-..@.......
	0x0010:  7f00 0001 deaf d839 0038 fe4b 2201 81ec  .......9.8.K"...
	0x0020:  0000 0000 0000 0000 4c4f 434c dfef b025  ........LOCL...%
	0x0030:  5b4e 874c dfef b025 5b43 73f3 dfef b025  [N.L...%[Cs....%
	0x0040:  5b4e 874c dfef b025 5b4e 874c            [N.L...%[N.L
13:42:45.356700 pid launchdns.16631 svc BE IP localhost.55353 > localhost.57007: UDP, length 48
	0x0000:  4500 004c b234 0000 4011 0000 7f00 0001  E..L.4..@.......
	0x0010:  7f00 0001 d839 deaf 0038 fe4b 2201 8103  .....9...8.K"...
	0x0020:  0000 0000 0000 0000 4c4f 434c dfef b025  ........LOCL...%
	0x0030:  5b4c 1a8a dfef b025 5b40 5f6b dfef b025  [L.....%[@_k...%
	0x0040:  5b4c 1a8a dfef b025 5b4c 1a8a            [L.....%[L..

This keeps Docker, kernel_task and launchdns locked into consuming 2.5 cores:

Restarting launchdns seemed to stop Docker from flooding it with requests and brought things back to normal. Previously I was restarting the Docker daemon entirely, which was a huge fuss.

Definitely still an outstanding bug here.

chrisharrisonkiwi · 2019-03-17T22:18:10Z

I'm also unsure if this issue has actually been resolved.
Seeing hyperkit using 379% today over 18 threads...

rkingon · 2019-03-17T22:20:46Z

+1 i see this every day

ltm · 2019-03-17T22:23:26Z

@chrisharrisonkiwi A new issue was opened for this. Please see #3565.

docker-robott · 2020-06-28T03:09:00Z

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

rn added the status/acknowledged label May 24, 2018

akimd self-assigned this May 25, 2018

akimd closed this as completed May 28, 2018

flurdy mentioned this issue Jun 5, 2018

com.docker.supervisor uses 133% CPU when there are no containers running . #2967

Closed

EugenMayer mentioned this issue Jun 7, 2018

Synchronization stops. EugenMayer/docker-sync#517

Closed

bertolo1988 unassigned akimd Aug 3, 2018

docker-robott added the version/18.06.1-ce-mac73 label Nov 7, 2018

mistydemeo mentioned this issue Mar 5, 2019

com.docker.supervisor uses 100% CPU when sending NTP requests to launchdns #3565

Open

2 tasks

docker locked and limited conversation to collaborators Jun 28, 2020

docker-robott added the lifecycle/locked label Jun 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

com.docker.supervisor 100% cpu usage #2923

com.docker.supervisor 100% cpu usage #2923

felixbuenemann commented May 20, 2018 •

edited

Loading

felixbuenemann commented May 20, 2018

Joshfindit commented May 23, 2018 •

edited

Loading

asoorm commented May 23, 2018

BretFisher commented May 24, 2018

akimd commented May 24, 2018

vegasbrianc commented May 24, 2018

felixbuenemann commented May 24, 2018

akimd commented May 24, 2018

sjwl commented May 24, 2018

BretFisher commented May 24, 2018 •

edited

Loading

ear commented May 25, 2018

akimd commented May 25, 2018

BretFisher commented May 25, 2018

akimd commented May 28, 2018

deitch commented Jun 7, 2018

bertolo1988 commented Aug 3, 2018 •

edited

Loading

rkingon commented Oct 14, 2018

tatwater commented Nov 2, 2018

perhallstroem commented Nov 2, 2018

benlangfeld commented Nov 7, 2018

benlangfeld commented Nov 7, 2018

rkingon commented Nov 7, 2018

craigsansam commented Nov 14, 2018

jasonkarns commented Nov 30, 2018

ltm commented Nov 30, 2018

ltm commented Nov 30, 2018

kivikakk commented Jan 21, 2019

chrisharrisonkiwi commented Mar 17, 2019

rkingon commented Mar 17, 2019

ltm commented Mar 17, 2019

docker-robott commented Jun 28, 2020

com.docker.supervisor 100% cpu usage #2923

com.docker.supervisor 100% cpu usage #2923

Comments

felixbuenemann commented May 20, 2018 • edited Loading

felixbuenemann commented May 20, 2018

Joshfindit commented May 23, 2018 • edited Loading

asoorm commented May 23, 2018

BretFisher commented May 24, 2018

akimd commented May 24, 2018

vegasbrianc commented May 24, 2018

felixbuenemann commented May 24, 2018

akimd commented May 24, 2018

sjwl commented May 24, 2018

BretFisher commented May 24, 2018 • edited Loading

ear commented May 25, 2018

akimd commented May 25, 2018

BretFisher commented May 25, 2018

akimd commented May 28, 2018

deitch commented Jun 7, 2018

bertolo1988 commented Aug 3, 2018 • edited Loading

rkingon commented Oct 14, 2018

tatwater commented Nov 2, 2018

perhallstroem commented Nov 2, 2018

benlangfeld commented Nov 7, 2018

benlangfeld commented Nov 7, 2018

rkingon commented Nov 7, 2018

craigsansam commented Nov 14, 2018

jasonkarns commented Nov 30, 2018

ltm commented Nov 30, 2018

ltm commented Nov 30, 2018

kivikakk commented Jan 21, 2019

chrisharrisonkiwi commented Mar 17, 2019

rkingon commented Mar 17, 2019

ltm commented Mar 17, 2019

docker-robott commented Jun 28, 2020

felixbuenemann commented May 20, 2018 •

edited

Loading

Joshfindit commented May 23, 2018 •

edited

Loading

BretFisher commented May 24, 2018 •

edited

Loading

bertolo1988 commented Aug 3, 2018 •

edited

Loading