TODO.md

• Minimum viable product
  • Construct sbatch files programmatically
  • Run constructed sbatch files
  • Ingest YAML of schema
  • Generate sbatch files from YAML
• Improvements
  • Results analysis
    • Result aggregation
    • Result analysis
    • Statistical re-runs
  • YAML schema
    • Multiple nested arguments
    • Pairwise arguments
    • Default groups for executables
    • Optional key to disable test benches in YAML
  • Optional extras
    • dry run flag
    • Argument parsing from the command line
    • Unit tests
    • Documentations
    • Publish to PyPI
    • Textual TUI
      • Original ideas
        • Visualise YAML schema structure
        • Spawn runs for benches/individual executables
        • View results/analysis of runs
        • Modify YAML via inbuilt editor?
      • Final plan
        • Tree-style bench hierarchy explorer
          • (is this useful? perhaps better as a list?)
        • Listing argument variations for test bench
        • Listing sbatch file contents for run configuration
        • metric table for parent test bench (even when run config selected), requires improving analysis code
        • metric plotting --"--
        • run configuration/bench button to send off jobs to slurm
        • modal dialog to wait on slurm queue (could be an async function?)
          • Add cancel button