Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: Windows Flakey integ tests #1835

Closed
sharanyad opened this issue Feb 6, 2019 · 5 comments
Closed

testing: Windows Flakey integ tests #1835

sharanyad opened this issue Feb 6, 2019 · 5 comments
Labels
kind/testing os/windows

Comments

@sharanyad
Copy link
Contributor

Summary

As mentioned in #1830 and #1833, TestVolumesFromRO, TestTaskLevelVolume are not passing consistently.

Description

Expected Behavior

test passes

Observed Behavior

test fails

Environment Details

Supporting Log Snippets
@adnxn
Copy link
Contributor

adnxn commented Feb 6, 2019

are both of these timeout failures?

@fenxiong fenxiong mentioned this issue Feb 13, 2019
Merged
8 tasks
@sharanyad sharanyad mentioned this issue Feb 14, 2019
Merged
@sharanyad
Copy link
Contributor Author 

goroutine 1654 [select, 4 minutes]:
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc0424a2bb0, 0xc042871b30, 0xc0424a2be8, 0x0, 0x1344b00, 0x133f5b8, 0x1344b00, 0x133f5b8, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:186 +0x229
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc0424a2bb0, 0xc042a9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:224 +0x19b
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc042002a20, 0xc042a9c000, 0x1000, 0x1000, 0xc04239ff00, 0xc042843808, 0x477657)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0xa5
net/http.(*persistConn).Read(0xc042958900, 0xc042a9c000, 0x1000, 0x1000, 0x0, 0xe1f, 0x1)
	C:/Go/src/net/http/transport.go:1391 +0x232
bufio.(*Reader).fill(0xc0428f27e0)
	C:/Go/src/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc0428f27e0, 0x1, 0x0, 0x0, 0x0, 0xc0424d33e0, 0x0)
	C:/Go/src/bufio/bufio.go:129 +0x63
net/http.(*persistConn).readLoop(0xc042958900)
	C:/Go/src/net/http/transport.go:1539 +0x2b8
created by net/http.(*Transport).dialConn
	C:/Go/src/net/http/transport.go:1186 +0xbac

goroutine 1548 [select, 4 minutes]:
net/http.(*persistConn).writeLoop(0xc042339680)
	C:/Go/src/net/http/transport.go:1759 +0x1c5
created by net/http.(*Transport).dialConn
	C:/Go/src/net/http/transport.go:1187 +0xbd1

goroutine 1642 [select, 4 minutes]:
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc042736c60, 0xc0429865d0, 0xc042736c98, 0x0, 0x1344b00, 0x133f5b8, 0x1344b00, 0x133f5b8, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:186 +0x229
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc042736c60, 0xc042a06000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:224 +0x19b
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc04234a6c0, 0xc042a06000, 0x1000, 0x1000, 0xc042477f00, 0xc0424f7808, 0x477657)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0xa5
net/http.(*persistConn).Read(0xc04298afc0, 0xc042a06000, 0x1000, 0x1000, 0x0, 0xf23, 0x1)
	C:/Go/src/net/http/transport.go:1391 +0x232

@sharanyad
Copy link
Contributor Author 

goroutine 2173 [select]:
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).asyncIo(0xc0426ad130, 0xc042c040f0, 0xc0426ad168, 0x0, 0x1344b00, 0x133f5b8, 0x1344b00, 0x133f5b8, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:186 +0x229
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32File).Read(0xc0426ad130, 0xc0429d2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/file.go:224 +0x19b
github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio.(*win32MessageBytePipe).Read(0xc0420f1e00, 0xc0429d2000, 0x1000, 0x1000, 0xc042cfe4c0, 0x13a6ce0, 0x477657)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/vendor/github.com/Microsoft/go-winio/pipe.go:116 +0xa5
net/http.(*persistConn).Read(0xc042be4360, 0xc0429d2000, 0x1000, 0x1000, 0x0, 0xec1, 0x1)
	C:/Go/src/net/http/transport.go:1391 +0x232
bufio.(*Reader).fill(0xc04270d140)
	C:/Go/src/bufio/bufio.go:97 +0x1b3
bufio.(*Reader).Peek(0xc04270d140, 0x1, 0x0, 0x0, 0x0, 0xc042b7cae0, 0x0)
	C:/Go/src/bufio/bufio.go:129 +0x63
net/http.(*persistConn).readLoop(0xc042be4360)
	C:/Go/src/net/http/transport.go:1539 +0x2b8
created by net/http.(*Transport).dialConn
	C:/Go/src/net/http/transport.go:1186 +0xbac

goroutine 2172 [select]:
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).waitEvent(0xc042503450, 0xc0429c6f60, 0xccfaf7)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:332 +0x31e
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).waitForTransition(0xc042503450, 0xc042dd0e40, 0xc0429c6f60, 0xc042c6c960)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:1043 +0x65
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressTask(0xc042503450)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:769 +0x769
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).overseeTask(0xc042503450)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:220 +0x7eb
created by github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).startTask
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:530 +0xda

goroutine 2210 [sleep]:
time.Sleep(0x93133014)
	C:/Go/src/runtime/time.go:65 +0x148
github.com/aws/amazon-ecs-agent/agent/utils/ttime.(*DefaultTime).Sleep(0x13a5e60, 0x93133014)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/utils/ttime/ttime.go:31 +0x40
github.com/aws/amazon-ecs-agent/agent/utils/retry.RetryWithBackoffCtx(0x1349740, 0xc042b4b320, 0x1345d00, 0xc042071ec0, 0xc042ab9de8, 0x0, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/utils/retry/retry.go:54 +0x8b
github.com/aws/amazon-ecs-agent/agent/utils/retry.RetryNWithBackoffCtx(0x1349740, 0xc042b4b320, 0x1345d00, 0xc042071ec0, 0x3, 0xc042d19e78, 0x0, 0x0)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/utils/retry/retry.go:74 +0x11c
github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi.(*dockerGoClient).PullImage.func1(0x1349740, 0xc042b4b320, 0xc0426ad080, 0xe99c06, 0x23, 0x0, 0xc042b4bb60)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi/docker_client.go:304 +0x181
created by github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi.(*dockerGoClient).PullImage
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi/docker_client.go:303 +0x1bc

goroutine 2174 [select]:
net/http.(*persistConn).writeLoop(0xc042be4360)
	C:/Go/src/net/http/transport.go:1759 +0x1c5
created by net/http.(*Transport).dialConn
	C:/Go/src/net/http/transport.go:1187 +0xbd1

goroutine 2183 [select]:
github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi.(*dockerGoClient).PullImage(0xc0426ad080, 0x1349740, 0xc042b4b320, 0xe99c06, 0x23, 0x0, 0x68c61714000, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/dockerclient/dockerapi/docker_client.go:315 +0x2bd
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).pullAndUpdateContainerReference(0xc042be4120, 0xc042364540, 0xc0429c1400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:798 +0x4ca
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).concurrentPull(0xc042be4120, 0xc042364540, 0xc0429c1400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:744 +0x6c4
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).pullContainer(0xc042be4120, 0xc042364540, 0xc0429c1400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:683 +0x36d
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).(github.com/aws/amazon-ecs-agent/agent/engine.pullContainer)-fm(0xc042364540, 0xc0429c1400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:175 +0xaf
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).applyContainerState(0xc042be4120, 0xc042364540, 0xc0429c1400, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:1175 +0x11d
github.com/aws/amazon-ecs-agent/agent/engine.(*DockerTaskEngine).transitionContainer(0xc042be4120, 0xc042364540, 0xc0429c1400, 0x1)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/docker_task_engine.go:1150 +0x80
github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).progressTask.func2(0xc0429c1400, 0x1)
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:745 +0xa5
created by github.com/aws/amazon-ecs-agent/agent/engine.(*managedTask).startContainerTransitions
	C:/Users/ADMINI~1/AppData/Local/Temp/amazon-ecs-agent/go/src/github.com/aws/amazon-ecs-agent/agent/engine/task_manager.go:844 +0x3e3
FAIL	github.com/aws/amazon-ecs-agent/agent/engine	1200.244s

@sharanyad sharanyad mentioned this issue Feb 15, 2019
Merged
@petderek
Copy link
Contributor

I'm seeing TestAWSLogsDriverMultilinePattern fail for unrelated changes too

@petderek petderek mentioned this issue Feb 22, 2019
Merged
@adnxn adnxn removed the kind/bug label Feb 26, 2019
@sharanyad
Copy link
Contributor Author

this was fixed with #1859
closing this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/testing os/windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@adnxn @petderek @sharanyad