Add sort_nodes.py

[mr0re1]

$ /slurm/scripts/sort_nodes.py --help
usage: sort_nodes.py [-h] [--nodelist NODELIST] [--ntasks-per-node NTASKS_PER_NODE] [--output OUTPUT]

This script sorts nodes based on their `physicalHost`.

See https://cloud.google.com/compute/docs/instances/use-compact-placement-policies

You can reduce latency in tightly coupled HPC workloads (including distributed ML training)
by deploying them to machines that are located close together.
For example, if you deploy your workload on a single physical rack, you can expect lower latency
than if your workload is spread across multiple racks.
Sending data across multiple rack requires sending data through additional network switches.

Example usage:
``` my_sbatch.sh
#SBATCH --ntasks-per-node=8
#SBATCH --nodes=64

export SLURM_HOSTFILE=$(/slurm/scripts/sort_nodes.py)

srun -l hostname | sort
```

options:
  -h, --help            show this help message and exit
  --nodelist NODELIST   Slurm 'hostlist expression' of nodes to sort, if not set the value of SLURM_NODELIST environment variable will be used
  --ntasks-per-node NTASKS_PER_NODE
                        Number of times to repeat each node in resulting sorted list.
                        If not set, the value of SLURM_NTASKS_PER_NODE environment variable will be used,
                        if neither is set, defaults to 1
  --output OUTPUT       Output file to write, defaults to 'hosts.<uuid>'

[samskillman]

Would suggest adding a test of numerical hostnames, and needs fix to get the lowest number first. Could be combined with a test of two groups of three nodes at depth 2, where the lowest number node is in the node with a higher first level string. something like

x/slurm-1
x/slurm-2
x/slurm-3
y/slurm-0
y/slurm-4
z/slurm-10

I'd expect 0,4,10,1,2,3

[mr0re1]

@samskillman , for

something like
x/slurm-1
x/slurm-2
x/slurm-3
y/slurm-0
y/slurm-4
z/slurm-10
I'd expect 0,4,10,1,2,3

The order 0, 4, 1, 2, 3, 10 seems to be more natural is the particular reason you wrote 0,4,10,1,2,3?

[mr0re1]

• Addressed comments about name-order:
  Don't use naive lexicographic order anywhere, always rely on order of names in nodelist.
  Significantly refactor def order.

• Rename topologicaly_sort_nodes -> sort_nodes to reduce confusion, since order is not "topological".

[samskillman]

@samskillman , for

something like
x/slurm-1
x/slurm-2
x/slurm-3
y/slurm-0
y/slurm-4
z/slurm-10
I'd expect 0,4,10,1,2,3

The order 0, 4, 1, 2, 3, 10 seems to be more natural is the particular reason you wrote 0,4,10,1,2,3?

I agree - I meant for that z to be a y. I think either would be acceptable solutions in principle, but I agree 0,4,1,2,3,10 is better. My mistake on the typo.

resolved