Update appliance for stackhpc.openhpc nodegroup/partition changes

[sjpb]

Update appliance to cope with stackhpc/ansible-role-openhpc#183 which changed openhpc_slurm_partitions to openhpc_nodegroups and openhpc_partitions.

As an example, a configuration like:

#environments/site/inventory/group_vars/all/openhpc.yml:
openhpc_slurm_partitions:
  - name: standard
    default: 'YES'
    groups:
       - name: largemem
       - name: gpu
         gres:
          - conf: gpu:A40:2
            file: /dev/nvidia[0-1]

would now be represented as:

#environments/site/inventory/group_vars/all/openhpc.yml:
openhpc_nodegroups:
  - name: gpu
      gres:
        - conf: gpu:A40:2
          file: /dev/nvidia[0-1]
openhpc_partitions:
  - name: standard
    default: 'YES'
    nodegroups:
      - largemem
      - gpu

Note that:

• Arbitrary slurm.conf Node-level parameters can now be included via the optional node_params entry on openhpc_nodegroup elements.
• The skeleton OpenTofu configuration no longer templates out partition configuration variables. Instead OpenTofu templates an Ansible variable cluster_compute_groups variable into the inventory/hosts.yml file. NB: This file should be removed once OpenTofu configurations have been updated.
• A default openhpc_nodegroups is calculated in environments/common/inventory/group_vars/all/openhpc.yml. This uses the above to creates a node group per key in the OpenTofu compute variable. This is usually what is required, but this variable will need overriding if custom Node-level Slurm parameters/configuration is required (e.g. for GRES).
• A default openhpc_partitions is calculated. This includes one partition per node group, plus a rebuild partition if the rebuild group is active. This should not generally be overriden. Instead normal (non-rebuild) partitions should be modified using openhpc_user_partitions, which is the same format.

This affects:

• common environment openhpc config
• skeleton templating
• caas environment infra creation and ansible variables
• stackhpc environment openhpc overides
• configuration required to support rebuild - now automatic
• can now call stackhpc.openhpc:validate.yml from ansible/validate.yml

#665 is replaced by this.

[sjpb]

TODO: test this with adding extra nodes in stackhpc env
[edit:] done, see below.

[sjpb]

Testing in .stackhpc env with adding "extra" nodes:

1. Default configuration, i.e. just adding nodes into 2nd compute group:

$ git diff environments/.stackhpc/tofu/main.tf
diff --git i/environments/.stackhpc/tofu/main.tf w/environments/.stackhpc/tofu/main.tf
index 8d78401b..ea58d4b0 100644
--- i/environments/.stackhpc/tofu/main.tf
+++ w/environments/.stackhpc/tofu/main.tf
@@ -84,7 +84,7 @@ module "cluster" {
         # Normally-empty partition for testing:
         extra: {
             nodes: []
-            #nodes: ["extra-0", "extra-1"]
+            nodes: ["extra-0", "extra-1"]
             flavor: var.other_node_flavor
         }
     }
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
extra        up 60-00:00:0      2   idle RL9-extra-[0-1]
standard*    up 60-00:00:0      2   idle RL9-compute-[0-1]

Ok.

2. With the above, changing to an explicitly-configured partition covering both compute groups:

$ git diff environments/.stackhpc/inventory/group_vars/all/openhpc.yml
diff --git i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
index 5aac5f8a..6dd18e9e 100644
--- i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
+++ w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
@@ -1,3 +1,8 @@
 openhpc_config_extra:
   SlurmctldDebug: debug
   SlurmdDebug: debug
+openhpc_user_partitions:
+  - name: hpc
+    nodegroups:
+      - standard
+      - extra
$ ansible-playbook ansible/slurm.yml --tags openhpc
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
hpc*         up 60-00:00:0      4   idle RL9-compute-[0-1],RL9-extra-[0-1]

OK.

3. With overlapping partitions:

$ git diff environments/.stackhpc/inventory/group_vars/all/openhpc.yml
diff --git i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
index 5aac5f8a..9235f8e0 100644
--- i/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
+++ w/environments/.stackhpc/inventory/group_vars/all/openhpc.yml
@@ -1,3 +1,11 @@
 openhpc_config_extra:
   SlurmctldDebug: debug
   SlurmdDebug: debug
+openhpc_user_partitions:
+  - name: normal
+    nodegroups:
+      - standard
+  - name: all
+    nodegroups:
+      - standard
+      - extra
$ ansible-playbook ansible/slurm.yml --tags openhpc
$ ansible login -a sinfo
RL9-login-0 | CHANGED | rc=0 >>
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
normal       up 60-00:00:0      2   idle RL9-compute-[0-1]
all*         up 60-00:00:0      4   idle RL9-compute-[0-1],RL9-extra-[0-1]

OK

[sjpb]

WIP: testing on azimuth as slurm-v21 - running, but need to redeploy slurm after last force-push

[jovial]

Looks sensible to me. Just need to wait for the openhpc role changes to merge so that we can update requirements.yml.

[sjpb]

Now fully tested on CaaS.

[sjpb]

Converted to draft because #668 must merge first.