sql/logictest: TestLogic failed

[cockroach-teamcity]

sql/logictest.TestLogic failed with artifacts on release-21.2 @ 1b1ae61781e15e5f93b9d0315ecb33a26e36938c:

=== RUN   TestLogic
    test_log_scope.go:79: test logs captured to: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic274596706
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestLogic
    logic.go:3437: -- test log scope end --
test logs left over in: /go/src/github.com/cockroachdb/cockroach/artifacts/logTestLogic274596706
--- FAIL: TestLogic (1260.39s)
=== RUN   TestLogic/experimental-span-configs/auto_span_config_reconciliation_job
=== PAUSE TestLogic/experimental-span-configs/auto_span_config_reconciliation_job
=== CONT  TestLogic/experimental-span-configs/auto_span_config_reconciliation_job
    logic.go:2324: 
         
        testdata/logic_test/auto_span_config_reconciliation_job:8: SELECT count(*), job_type, status FROM [SHOW AUTOMATIC JOBS] WHERE job_type = 'AUTO SPAN CONFIG RECONCILIATION' GROUP BY (job_type, status)
        expected:
            count  job_type                         status
            1      AUTO SPAN CONFIG RECONCILIATION  running
            
        but found (query options: "colnames") :
            count  job_type  status
--- done: testdata/logic_test/auto_span_config_reconciliation_job with config experimental-span-configs: 2 tests, 1 failures
    logic.go:3205: 
        testdata/logic_test/auto_span_config_reconciliation_job:14: error while processing
    logic.go:3205: testdata/logic_test/auto_span_config_reconciliation_job:14: too many errors encountered, skipping the rest of the input
--- total: 153771 tests, 0 failures
        --- FAIL: TestLogic/experimental-span-configs/auto_span_config_reconciliation_job (0.36s)
=== RUN   TestLogic/experimental-span-configs
    --- FAIL: TestLogic/experimental-span-configs (0.00s)

Reproduce

To reproduce, try:

make stressrace TESTS=TestLogic PKG=./pkg/sql/logictest TESTTIMEOUT=5m STRESSFLAGS='-timeout 5m' 2>&1

Parameters in this failure:

• GOFLAGS=-json

Same failure on other branches

• sql/logictest: TestLogic failed #63262 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-20.2]
• sql/logictest: TestLogic failed #61657 sql/logictest: TestLogic failed [C-test-failure O-robot branch-release-21.1]

/cc @cockroachdb/sql-queries jordanlewis

This test on roachdash | Improve this report!

Jira issue: CRDB-16769

[mgartner]

@irfansharif is this related to your recent work?

[github-actions]

We have marked this test failure issue as stale because it has been
inactive for 1 month. If this failure is still relevant, removing the
stale label or adding a comment will keep it active. Otherwise,
we'll close it in 5 days to keep the test failure queue tidy.

[irfansharif]

This code was disabled in 21.2, and further deflaked later in 22.1 (#71195). The disabling happened here:

cockroach/pkg/spanconfig/spanconfigmanager/manager.go

Lines 42 to 45 in 1b1ae61

	// jobEnabledSetting gates the activation of the span config reconciliation job.
	var jobEnabledSetting = settings.RegisterBoolSetting(
	"spanconfig.experimental_reconciliation_job.enabled",
	"enable the use of the kv accessor", false)

Hm I guess this test file specifically enables it:

cockroach/pkg/sql/logictest/testdata/logic_test/auto_span_config_reconciliation_job

Lines 3 to 4 in 1b1ae61

	statement ok
	SET CLUSTER SETTING spanconfig.experimental_reconciliation_job.enabled = true;

Looks like it's just missing a retry here:

cockroach/pkg/sql/logictest/testdata/logic_test/auto_span_config_reconciliation_job

Line 8 in 1b1ae61

query ITT colnames

Which we have on master:

cockroach/pkg/sql/logictest/testdata/logic_test/auto_span_config_reconciliation_job

Line 6 in 77adcaf

query ITT colnames,retry

Basically we need to backport #71195.

[irfansharif]

#85548.