Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

500 response code in TestStatusLocalLogs #1940

Closed
tamird opened this issue Aug 4, 2015 · 12 comments
Closed

500 response code in TestStatusLocalLogs #1940

tamird opened this issue Aug 4, 2015 · 12 comments
Assignees
@BramGruneir
Labels
C-test-failure Broken test (automatically or manually discovered).

Comments

@tamird
Copy link
Contributor

tamird commented Aug 4, 2015

Discovered in #1839.

https://circleci.com/gh/cockroachdb/cockroach/5404
https://circle-artifacts.com/gh/cockroachdb/cockroach/5404/artifacts/0/tmp/circle-artifacts.oO88rMP/testrace.log
@tamird tamird added the C-test-failure Broken test (automatically or manually discovered). label Aug 4, 2015
@tamird
Copy link
Contributor Author

tamird commented Aug 5, 2015

Another instance.

https://circleci.com/gh/cockroachdb/cockroach/5526
https://circle-artifacts.com/gh/cockroachdb/cockroach/5526/artifacts/0/tmp/circle-artifacts.F4In60q/testrace.log

@BramGruneir
Copy link
Member

Yeah, I've seen it a few times now. I'll investigate.

@tamird tamird mentioned this issue Aug 21, 2015
Merged
@tamird
Copy link
Contributor Author

tamird commented Aug 23, 2015

Now the test times out (with the retry logic added):

relevant logs: https://gist.github.com/tamird/6df602f3931e4024e23d

build: https://circleci.com/gh/cockroachdb/cockroach/6313
test.log: https://circle-artifacts.com/gh/cockroachdb/cockroach/6313/artifacts/0/tmp/circle-artifacts.v625idu/test.log

@tamird
Copy link
Contributor Author

tamird commented Aug 24, 2015

OK that last report looks like just broken retry. Should be fixed in #2230, but this issue remains valid afaik.

@BramGruneir
Copy link
Member

Hopefully this should give us more insight into why we're getting a timeout. It seems to just get stuck occasionally.

@tamird
Copy link
Contributor Author

tamird commented Aug 24, 2015

I'm actually concerned that we won't see failures at all because the retry logic will paper over the issue. Remains to be seen, though.

@tamird
Copy link
Contributor Author

tamird commented Sep 3, 2015

Here we go, another one:
https://circleci.com/gh/cockroachdb/cockroach/6858
https://circle-artifacts.com/gh/cockroachdb/cockroach/6858/artifacts/0/tmp/circle-artifacts.c9cyXRx/testrace.log

@tamird
Copy link
Contributor Author

tamird commented Sep 6, 2015

Another one. https://circleci.com/gh/cockroachdb/cockroach/6949
https://circle-artifacts.com/gh/cockroachdb/cockroach/6949/artifacts/0/tmp/circle-artifacts.LtvEvpn/testrace.log

Looks like a bunch of timeouts:

    status_test.go:230: could not GET https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning - Get https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    status_test.go:230: could not GET https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning - Get https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    status_test.go:230: could not GET https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning - Get https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    status_test.go:230: could not GET https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning - Get https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    status_test.go:230: could not GET https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning - Get https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
    status_test.go:251: There was an error retrieving https://127.0.0.1:46511/_status/logs/local?level=INFO&pattern=Info|Error|Warning

Maybe the timeout is too short for a race build?

@tbg
Copy link
Member

tbg commented Sep 6, 2015

do we know what the timeout is? If it's relatively low maybe the test is just flaky on the slow machines (unlikely if it's something like 10s though).

@tamird
Copy link
Contributor Author

tamird commented Sep 6, 2015

3 seconds

@tbg
Copy link
Member

tbg commented Sep 6, 2015

hmm. Could picture that being too slow sometimes. Should increase that if we can.

@tamird
Copy link
Contributor Author

tamird commented Sep 10, 2015

Closing in favour of #2440.

@tamird tamird closed this as completed Sep 10, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@BramGruneir BramGruneir

Labels
C-test-failure Broken test (automatically or manually discovered).
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tamird @BramGruneir @tbg