test-runtime does not re-run when the substrate binary on your PATH changes.

[jsdw]

test-runtime talks to a substrate node and compiles a subxt runtime based on that nodes metadata. As it stands, it will cache builds far too aggressively, and so if you update the substrate binary for instance, it will not know to re-build itself, and so the metadata now will be out of sync with the substrate binary.

The workaround at present is to make some change to build.rs to trigger a rebuild.

Cargo exposes two ways to trigger a rebuild, which we can (and do) use like so:

// Re-build if we point to a different substrate binary:
println!("cargo:rerun-if-env-changed={}", SUBSTRATE_BIN_ENV_VAR);
// Re-build if this file changes:
println!("cargo:rerun-if-changed=build.rs");

Ideally, we'd test that the substrate binary we point to is identical (eg mathcing hash), and re-run if it changes.

Failing that, we'd want to tell the thing to always re-run, which would be annoying but safer.

Is there a cunning way to achieve either of these given the current machinery available to us?