Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(control): fix a flaky test in the controller shutdown #612

Merged
merged 1 commit into from
Jan 2, 2019

Conversation

jsternberg
Copy link
Contributor

There were a few problems in the controller shutdown test that caused it
to be flaky.

First, it didn't wait for all 10 queries to get into the controller
queue before checking the queue length. This caused it to call Fatal
and abort the test, but Shutdown had already been called and the
active queries needed to be finished.

After that, it attempted to run a query before the shutdown may have
been initiated so the query got enqueued correctly even though it was
not expecting that to happen. It now attempts to create the query and
tries multiple times to get the failure with a wait in between if it
can't.

It now succeeds when run with the following command line:

go test -c ./control
cpulimit -z -i -l 100 ./control.test -test.v -test.cpu 1

I was able to reproduce the issue by using the above.

Fixes #611.

There were a few problems in the controller shutdown test that caused it
to be flaky.

First, it didn't wait for all 10 queries to get into the controller
queue before checking the queue length. This caused it to call `Fatal`
and abort the test, but `Shutdown` had already been called and the
active queries needed to be finished.

After that, it attempted to run a query before the shutdown may have
been initiated so the query got enqueued correctly even though it was
not expecting that to happen. It now attempts to create the query and
tries multiple times to get the failure with a wait in between if it
can't.

It now succeeds when run with the following command line:

    go test -c ./control
    cpulimit -z -i -l 100 ./control.test -test.v -test.cpu 1

I was able to reproduce the issue by using the above.
@jsternberg jsternberg merged commit fbe1436 into master Jan 2, 2019
@jsternberg jsternberg deleted the fix/controller-shutdown-flaky-test branch January 2, 2019 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants