Config loading regression test #34
Description
We need to know if changes in the config loading causes regressions with OT config. One suggestion is to add a mechanism to dump the parsed config during a DVSim run, then compare the before and after to ensure no regressions have been introduced during a refactor.
This is not as simple as it sounds, due to the wildcard substitution mechanism within the HJSON. Which is resolved in a cascading manor throughout the DVSim Job preparation step. So for a more robust check we need an end to end test for the broadest coverage. Then perhaps refine this further with integration tests on specific behaviours.
Proposal
Add JSON dump mechanism to the DVSim job scheduler - so we can save the config of each job about to be run. Shouldn't be more than couple of lines of additional code (turned on with an environment variable flag). This is the simplest place I can think of for dumping the output of the config loading stage. It means we then have plain text diff-able artefact representing the actual jobs DVSim plans to run which can be compared for A/B testing.
This means we can do a manual A/B test and know if the stand-alone DVSim changes anything meaningful in terms of what and how the simulations are run.