Hundreds Of Containers Stuck In Created Status

[blaines]

I have hundreds of containers that seem to be stuck in the CREATED status. It seems like too many of these could overload Docker and cause the system to be unresponsive.

CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS                PORTS                            NAMES
9a788a418deb        blaines/hello-worker             "node hello-worker.js"   4 minutes ago       Created                                                ecs-example-2-hello-worker-c2b0a2b8f1c6acee2400
3babf34ddead        blaines/hello-worker             "node hello-worker.js"   4 minutes ago       Created                                                ecs-example-2-hello-worker-d69ec8c6c1ece5f8d301
f6ec1789f5e8        blaines/hello-worker             "node hello-worker.js"   4 minutes ago       Created                                                ecs-example-2-hello-worker-c69791ded797aa907600
c14e0854858c        blaines/hello-worker             "node hello-worker.js"   4 minutes ago       Created                                                ecs-example-2-hello-worker-ac9dd4bebdb7f5c63700
80354ac2b536        blaines/hello-worker             "node hello-worker.js"   4 minutes ago       Created                                                ecs-example-2-hello-worker-c4feb0e0dce1c3d5ce01
1fd85a11a54b        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-b4aafdc8c8d3ace2b901
72e84e6b70db        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-a49189ad95f6cabe4300
4aa51387f5d5        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-eaebaaa689f1d2e20100
904fe4988d0d        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-aabe8df9a4d9faee1800
7e2545490fba        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-908686918abae88f2200
46a308101d65        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-beca91dcc6cd93d8b401
7934d0125ab8        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-d2b3b0e9a09e8da59c01
2c7b508c1d59        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-f6c4d9daa68397e38201
2dec64d2caa8        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Created                                                ecs-example-2-hello-worker-a8b0f989b7faabbb5c00
9e1af1f5bf81        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-decaace7c48380b47200
00c2d502abfe        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-86b4d4ba8e81dfbf0300
6e4fe929fc7b        blaines/hello-worker             "node hello-worker.js"   5 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-ba939ba2bcccac88e901
535aba7460b5        blaines/hello-worker             "node hello-worker.js"   6 minutes ago       Created                                                ecs-example-2-hello-worker-84eeebe0eeb4b5f51900
8e892a474c6b        blaines/hello-worker             "node hello-worker.js"   6 minutes ago       Created                                                ecs-example-2-hello-worker-88f6d78f819b979bb301
88ca3476e567        blaines/hello-worker             "node hello-worker.js"   6 minutes ago       Created                                                ecs-example-2-hello-worker-bec9b6b1f0fcce9e8501
33867168e194        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-9494bba2d48cfc878901
ecb291e8cc3b        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-82e7b2b796b9a7aeb801
de936e6d2351        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-aadb9b94c9a0cca40800
d3b0912ad246        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-b4aced95f295a3c6f601
1fef9b642833        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-de9288c893a6c3bd8b01
120b21fb8056        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-d8d78ac1d497aa9c2d00
95e2d9079d90        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-aee3ead6aaf4eedf0200
8ad617aafaec        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-b8b9cec298dcb4f43000
7b861e28b62e        blaines/hello-worker             "node hello-worker.js"   7 minutes ago       Created                                                ecs-example-2-hello-worker-dea597f4de9c8abdec01
b8edcb7a7899        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-b8dddac0858dc2db3300
c05a8fe40df7        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-caeed0a3beefaad7e301
0195e08b3252        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-98dd8ab7b3abc489bd01
f70b8357958c        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-aada978b849981d06300
daa78bdc5e6a        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-88e7e6d9f7aaa6a36e00
ad5c363a04a9        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-e6d99d9df5bbe1f02900
187e0eb17027        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-fe94b4a28fa9a7f6d901
38773e3f52fd        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-ae84bfd0fab289c6d701
3a8b23463eb6        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-dce088c382dca09bb001
1be689c42aaa        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-bc958a94c5b9ff917d00
c600ecb5dd56        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Removal In Progress                                    ecs-example-2-hello-worker-b699898ee0e2d6832400
8c170e5b4758        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-aaf2a2beb2c9ddb3a301
a6f2b9b2318e        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-c6fcf4ae9a84eb89b201
85582992a26f        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-80fba08bfbfdb2836b00
12317b78f17b        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-f8a1cee0ff96e8e68201
7ee5596851cf        blaines/hello-worker             "node hello-worker.js"   8 minutes ago       Created                                                ecs-example-2-hello-worker-8484bad2ea93e2c6d001
--- 300 more lines ---
23e93b86ec05        blaines/hello-worker             "node hello-worker.js"   3 hours ago         Created                                                ecs-example-2-hello-worker-c4acebe9b5fa96ac1f00
965dff417a15        skycatch/geoping-api:997dc72     "npm-start"              19 hours ago        Up 19 hours           0.0.0.0:4205->4205/tcp           ecs-geoping-api-30-geoping-api-eaa6f1809fa2e2951e00
7a2677f184f6        gliderlabs/logspout              "/bin/logspout syslog"   19 hours ago        Up 19 hours           80/tcp, 0.0.0.0:7273->7273/tcp   ecs-papertrail-9-papertrail-8ca9e683eba984e7ff01
f1d1ff5fb353        amazon/amazon-ecs-agent:latest   "/agent"                 19 hours ago        Up 19 hours           127.0.0.1:51678->51678/tcp       ecs-agent

Archive.zip

Another instance
Archive 2.zip

[blaines]

#306 may be related

[blaines]

[ec2-user@ip-172-31-42-194 ~]$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   5604cbe/1.11.1
 Built:
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.3
 Git commit:   5604cbe/1.11.1
 Built:
 OS/Arch:      linux/amd64
[ec2-user@ip-172-31-42-194 ~]$ docker info
Containers: 71
 Running: 4
 Paused: 0
 Stopped: 67
Images: 9
Server Version: 1.11.1
Storage Driver: devicemapper
 Pool Name: docker-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file:
 Metadata file:
 Data Space Used: 2.742 GB
 Data Space Total: 23.35 GB
 Data Space Available: 20.61 GB
 Metadata Space Used: 3.539 MB
 Metadata Space Total: 25.17 MB
 Metadata Space Available: 21.63 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.93-RHEL7 (2015-01-28)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 4.4.10-22.54.amzn1.x86_64
Operating System: Amazon Linux AMI 2016.03
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 14.94 GiB
Name: ip-172-31-42-194
ID: 6VJC:CHEI:D5XT:7UI6:H2A6:7Q66:TIVL:P6XO:M7GF:O2TE:TIAB:5G5D
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
[ec2-user@ip-172-31-42-194 ~]$ uname -a
Linux ip-172-31-42-194 4.4.10-22.54.amzn1.x86_64 #1 SMP Tue May 17 22:45:04 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

[blaines]

The instances seem to be overwhelmed now. I checked and stopped containers are simply building up and there are many jobs pending but not starting. I'm adding some screenshots so you can see what I'm seeing...

[blaines]

Ah I finally found an error

[aaithal]

@blaines thanks for debugging this issue. Seems like you may need to create a bigger volume to resolve this. I'm closing this for now. Feel free to reopen/create a new issue if you continue to run into this.

[blaines]

It's fine, it just took a while to figure out because the free disk space wasn't exposed anywhere easily, the logs had so many messages, and I had to learn how the logical volumes were being used to store containers. I think everything works fine on your end.

[Sciencegeek123]

Could you please expand on your resolution @blaines ? I'm seeing a similar issue I think, but I'm not sure how to check. We've got what I think is plenty of disk space.

[aaithal]

@Sciencegeek123 The documentation for determining the available disk space is available here. docker info | grep "Data Space" would be an easy way to check that.

[Sciencegeek123]

Huh that doesn't make much sense.

$ docker info | grep "Data Space"
 Data Space Used: 2.662 GB
 Data Space Total: 23.35 GB
 Data Space Available: 20.69 GB
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  838M  6.9G  11% /
devtmpfs        1.9G   80K  1.9G   1% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm

Would this mean that I do need more space?

[aaithal]

@Sciencegeek123 as per the documentation,

Amazon ECS-optimized AMIs from version 2015.09.d and later launch with an 8 GiB volume for the operating system that is attached at /dev/xvda and mounted as the root of the file system. There is an additional 22 GiB volume that is attached at /dev/xvdcz that Docker uses for image and metadata storage. The volume is configured as a Logical Volume Management (LVM) device and it is accessed directly by Docker via the devicemapper back end. Because the volume is not mounted, you cannot use standard storage information commands (such as df -h) to determine the available storage. However, you can use LVM commands and docker info to find the available storage by following the procedure below. For more information about LVM, see the LVM HOWTO in The Linux Documentation Project.

You should also check the output of docker info | grep Space, which would show the metadata space as well. You may need a bigger disk or you may need a faster disk if you have an IO intensive workload.

[Sciencegeek123]

That is very good to know, thanks! I found my issue after crawling through logs. The command I was issuing was not found. Not sure why it doesn't work yet, as I can run it when using docker run, but I'm sure some study of the documentation will make it more clear. Thanks for your help!

[jgonzalezd]

For anyone that has hit this same issue, the problem lies on the maximum Inodes exhausted.
Check for df -i
I will come back with a solution to this issue.