Tasks stuck in PENDING with containers stuck in Created state

[trharris78]

I'm using RunTask to schedule hundreds of tasks on a small ECS cluster. The tasks are short-lived, maybe a minute or two at most. Tasks that I can't schedule with a RunTask call are left on a queue and I try the RunTask call again later. This seems to work for awhile, but eventually the cluster gets bogged down and tasks get stuck in the PENDING state. Running docker ps -a shows tons of containers in the Created state and they seem to be stuck indefinitely.

I'm also using DescribeTasks to check on task states, and I'm getting DockerTimeoutError and CannotInspectContainerError fairly often. These start to happen at about the same time the containers start to get stuck in the Created state. Also, docker ps -a becomes unresponsive, sometimes sitting there for 10 minutes or more before I hit Ctrl+C.

I originally thought that the Docker daemon was getting overwhelmed with hundreds of exited containers, so I built the amazon-ecs-agent dev branch to try the new ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION variable. Containers now get cleaned up after a few minutes, but the PENDING problem persists.

I also tried updating to Docker 1.10.0 but that didn't help, either.

I'm using the ECS-optimized AMI in us-east-1, ami-cb2305a1, which includes Docker 1.9.1. I see similar issues #296, #300, and #305 that might be related.

Docker and ecs-agent logs are attached.

logs.zip

[samuelkarp]

Hi @trharris78, thanks for providing logs.

Your Docker logs show a panic that looks related to moby/moby#18481. The Agent logs show lines like this:

2016-02-08T17:00:00Z [INFO] Error transitioning container module="TaskEngine" task="xxxx:2 arn:aws:ecs:us-east-1:xxxx:task/xxxx, Status: (NONE->RUNNING) Containers: [xxxx (PULLED->RUNNING),]" container="xxxx(xxxx/xxxx) (PULLED->RUNNING)" state="CREATED"
2016-02-08T17:00:00Z [WARN] Error with docker; stopping container module="TaskEngine" task="xxxx:2 arn:aws:ecs:us-east-1:xxxx:task/xxxx, Status: (NONE->RUNNING) Containers: [xxxx (CREATED->RUNNING),]" container="xxxx(xxxx/xxxx) (CREATED->RUNNING)" err="Could not transition to created; timed out after waiting 3m0s"

The Agent logs are consistent with what I'd expect from that Docker issue: the Agent is attempting to talk to the Docker daemon but it's not being responsive. Can you provide the output of docker info, sudo vgdisplay, sudo lvdisplay, and anything that looks suspicious in dmesg? I think this is unrelated to #300 (since that's Docker 1.7.1 and related to behavior around instance termination); this could be related to the other two issues you linked but that's as yet unclear.

[trharris78]

I'm unable to provide the requested additional logs because we rearchitected our system to use services instead of tasks. Perhaps in the future we will revisit this.