[CI] Failing test on master, CI didn't catch it.

[gui1117]

As reported by @clangenb https://github.com/paritytech/polkadot-sdk/pull/6294/files#r1822860138

A test on master is failing. The test is in a file conditioned by runtime-benchmarks, and is generated by construct_runtime:

polkadot-sdk/substrate/frame/session/benchmarking/src/mock.rs

Line 38 in d1fafa8

frame_support::construct_runtime!(

From what I see in the merge queue, the job running the benchmarking test is: https://github.com/paritytech/polkadot-sdk/actions/runs/12106844624/job/33753083856

The command is: forklift cargo nextest run --workspace --features runtime-benchmarks benchmark --locked --cargo-profile testnet --cargo-quiet

but this run only the test that contains the word benchmark. The test name is mock::test_genesis_config_builds so it was skipped.

[bkchr]

Yeah the benchmark is filtering out only the tests which are generated by the benchmarking macros. IMO the best would be to extend the call with the crates that contain benchmark. Still not the best and some tests could get skipped, but generally tests should not be hidden behind the runtime-benchmarks feature any way.

[gui1117]

I agree, it is good convention that test should not be hidden by runtime-benchmarks features. Even for pallet session benchmarking, the mock doesn't need to be hidden by runtime-benchmarks feature.