integrate non-node resources into flux resource tools

[garlick]

Problem: although rabbits (c.f. flux-coral2) can be drained, they have no visibility in flux native tools.

For example, flux resource status and flux resource drain do not list the state of rabbits.

We should think about how to integrate non-node resources into the core toolset and perhaps the resource module. This may also involve monitoring for availability.

Related rabbit integration issues:

• harden flux-core processing of jobspec for rabbit storage use case #5767
• Add rabbit entries to flux job info list of useful keys #6328
• stage-in: integrate rabbit support #4813

[jameshcorbett]

This flux-coral2 issue flux-framework/flux-coral2#229 is relevant.

For rabbits, I'm just spitballing but I'm imagining something like the following.

$ flux resource list --extras
     STATE QUEUE  NNODES NCORES NGPUS NODELIST
      free pci         3    192    24 tioga[14,16-17]
 allocated pci         1     64     8 tioga15
 allocated pdebug     18   1152   144 tioga[18-21,23-24,28-31,33-39,41]
 allocated mi300a      2    192     8 tioga[42-43]
      down pdebug      6    384    48 tioga[22,25-27,32,40]
 
RABBIT
    STATE NODELIST       CAPACITY
    up       rabbit[1-4,7]    1058
    down  rabbit[5-6]       0

where the --extras flag triggers some plugin, in this case a flux-coral2 plugin, to return some custom fields. I was imagining that the flux-coral2 plugin could send an RPC to a service that has this information.