sql/logictest: TestLogic/fakedist-disk/select_index_span_ranges timed out

[cockroach-teamcity]

(sql/logictest).TestLogic/fakedist-disk/select_index_span_ranges failed on master@4a25ab1bc85e404b1a8b3e28660e41e7e5063e0b:

Slow failing tests:
TestLogic/fakedist-disk/select_index_span_ranges - 0.00s

Slow passing tests:
<none>

More

Parameters:

• TAGS=
• GOFLAGS=-parallel=2

make stressrace TESTS=TestLogic/fakedist-disk/select_index_span_ranges PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

See this test on roachdash
_{powered by pkg/cmd/internal/issues}

[yuzefovich]

panic: test timed out after 40m0s

[asubiotto]

Looking at the goroutine stacks it doesn't look like any is stuck anywhere or runnable in an infinite loop. This looks like a testing issue where logictest was picked and is unable to complete in 40m under stress. I'm tempted to close this since this hasn't happened before in my memory and reinvestigate if it happens again. I've tried to find a Stress run where logictest is picked to see what is expected but it's not trivial.

[yuzefovich]

We have a bunch of logic tests timing out: https://github.com/cockroachdb/cockroach/issues?q=is%3Aissue+is%3Aopen+logictest+timed+out. Were all of those run under stress?

[asubiotto]

Yes, it looks like it. I've been trying to search for a successful nightly stress on the logictest package but no luck because the build logs are not indexed (and I think it's the only way to perform a search for successful logictest stress runs).

So far the oldest failure looks to be #53184. Our current thinking is that dynamic batch sizes have somehow affected the test runtime because the PR was merged a couple of days before #52453, but it's difficult to say without knowing what it was beforehand.

I'm going to step away from this for now because it feels like a timesink. @yuzefovich any ideas of what we could do?

[asubiotto]

Just ran the invocation on b289a79 and one set of logic tests completed in 30 minutes. By the end of the hour, there were two successful runs. I'm now running the same thing on 895125b to perform a crude bisect.

[asubiotto]

Interestingly, 895125b completes in 30 minutes as well. I'm going to start off an automatic bisect.

[asubiotto]

Hmm, bisect didn't find anything and running the current master completes even quicker (26 mins single run, 4 runs for 1 hour). The commit in question is eba9eed. The invocation is:

stdbuf -oL -eL make stress PKG=github.com/cockroachdb/cockroach/pkg/sql/logictest TESTTIMEOUT=40m GOFLAGS=-parallel=2 TAGS= 'STRESSFLAGS=-stderr -maxruns 100 -maxtime 1h0m0s -maxfails 1 -p 2'

[asubiotto]

@jordanlewis how do I get access to a teamcity agent?