[Bug]: spack test results with suite name not expected

[shahzebsiddiqui]

Bug Description

the problem is best described in test result https://my.cdash.org/test/67729705?graph=status

The test failure reports

Lmod has detected the following error: Swap failed: "PrgEnv-nvidia" is not
loaded.
==> Error: Test suite "spack_test_hdf5_e4s_21.11" already exists. Try another alias.

for this buildspec

buildspecs:
  spack_test_hdf5_e4s_21.11:
    type: spack
    executor: '(perlmutter|muller).local.sh'
    description: "Test hdf5@1.12.1 for e4s/21.11 stack for Perlmutter via spack test"
    tags: e4s
    pre_cmds: |
      module load e4s/21.11
      module swap PrgEnv-nvidia PrgEnv-gnu
    spack:
      root: /global/common/software/spackecp/perlmutter/e4s-21.11/spack/
      verify_spack: false
      test:
        run:
          specs: ['hdf5@1.12.1']
        results:
          option: '-l'
          specs: ['hdf5@1.12.1']

The issue is that we auto-generate the suite name and since the suite name is taken when rerunning the test this test never ran.

Steps to reproduce the error

Just run buildtest build against any spec in this directory https://github.com/buildtesters/buildtest-nersc/tree/devel/buildspecs/e4s/spack_test/perlmutter

Version and HEAD commit

devel branch

Relevant log output

No response

Post question in Slack

• I agree that I posted my question in slack before creating this issue

Is there an existing issue

• I confirm there is no existing issue for this issue

[shahzebsiddiqui]

this is related to issue NERSC/buildtest-nersc#157.

@wspear suggested we can auto-generate alias names for every test one idea could be to remove the suite property under results such as this test defined in https://buildtest.readthedocs.io/en/devel/buildspecs/spack.html#spack-test

buildspecs:
  spack_test_m4:
    type: spack
    executor: generic.local.bash
    description: "Run spack test for m4 package and report results"
    tags: [spack]
    spack:
      root: $HOME/spack
      test:
        run:
          specs: ['m4']
        results:
          suite: ['spack_test_m4']
          option: "-l"

The suite name is name of test spack_test_m4 so if the key spec is not defined under results we retrieve the suite name otherwise use the specs.

The question is whether we give user the ability to specify the suite name under spack test run --alias <alias_name> they must do this under the run section using options so it is doable and they can do the following

buildspecs:
  spack_test_m4:
    type: spack
    executor: generic.local.bash
    description: "Run spack test for m4 package and report results"
    tags: [spack]
    spack:
      root: $HOME/spack
      test:
        run:
          specs: ['m4']
          options: "--alias m4_test"
        results:
          suite: ['m4_test']
          option: "-l"

However i think that it might not be useful having suite property this only makes sense if user specifies the options property under run otherwise they wont know the alias names of hand since spack auto-generates them at runtime. Given that this becomes a problem when re-running the same test multiple times, the command spack test run --alias will fail if suite name is already present.