jobs: mixed-version testing reports unable to find job_info

[knz]

Found here #99866 (comment)
Despite 858d874 there is a job_info issue:

=== RUN   TestCCLLogic_partitioning_enum
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/0817673d1f07fbf06f1393f987f3a3c7/logTestCCLLogic_partitioning_enum31111442
    test_log_scope.go:79: use -show-logs to present logs inline
    testcluster.go:391: migration-manager-find-jobs: system-jobs-scan: relation "system.job_info" does not exist

Jira issue: CRDB-26520

[rafiss]

Here is a repro that doesn't involve checking out my PR:

Apply this diff:

diff --git a/pkg/ccl/logictestccl/testdata/logic_test/partitioning_enum b/pkg/ccl/logictestccl/testdata/logic_test/partitioning_enum
index 26f4e6a29a2..6b6089b6a17 100644
--- a/pkg/ccl/logictestccl/testdata/logic_test/partitioning_enum
+++ b/pkg/ccl/logictestccl/testdata/logic_test/partitioning_enum
@@ -1,3 +1,4 @@
+# LogicTest: default-configs local-mixed-22.2-23.1

 # Enums are supported for partitioning both by list and range.
 # In 20.2 alphas, partitioning by enums was permitted but led to

Run ./dev testlogic ccl --config=local-mixed-22.2-23.1 --files=partitioning_enum.

It doesn't fail every time. But with ./dev testlogic ccl --config=local-mixed-22.2-23.1 --files=partitioning_enum --count 5 I usually get a failure.

[rafiss]

These are the settings used by the local-mixed-22.2-23.1:

cockroach/pkg/sql/logictest/logictestbase/logictestbase.go

Lines 447 to 454 in 44f3f31

	{
	Name: "local-mixed-22.2-23.1",
	NumNodes: 1,
	OverrideDistSQLMode: "off",
	BootstrapVersion: clusterversion.V22_2,
	DisableUpgrade: true,
	DeclarativeCorpusCollection: true,
	},

And this is how the versions are overridden:

cockroach/pkg/sql/logictest/logic.go

Lines 1382 to 1394 in 44f3f31

	if cfg.BootstrapVersion != clusterversion.Key(0) {
	if params.ServerArgs.Knobs.Server == nil {
	params.ServerArgs.Knobs.Server = &server.TestingKnobs{}
	}
	params.ServerArgs.Knobs.Server.(*server.TestingKnobs).BootstrapVersionKeyOverride = cfg.BootstrapVersion
	params.ServerArgs.Knobs.Server.(*server.TestingKnobs).BinaryVersionOverride = clusterversion.ByKey(cfg.BootstrapVersion)
	}
	if cfg.DisableUpgrade {
	if params.ServerArgs.Knobs.Server == nil {
	params.ServerArgs.Knobs.Server = &server.TestingKnobs{}
	}
	params.ServerArgs.Knobs.Server.(*server.TestingKnobs).DisableAutomaticVersionUpgrade = make(chan struct{})
	}

[knz]

@rafiss I don't see this test using MakeTestingClusterSettingsWithVersions. Is that needed?

[knz]

@celiala also reports:

TestServerStartupGuardrails is failing when I attempt to set 'developmentBranch to false' for release-23.1.
https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9431680?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

[rafiss]

I think @yuzefovich's change in 6ef18ed as he explained in #99987 (comment) fixes the issue for local-mixed-22.2-23.1 logic tests.

it turns out that for tenant configs we need to use MakeTestingClusterSetting but for non-tenant configs MakeClusterSettings

These interfaces are pretty hard to use correctly.