Fix #87 - read GPU information directly from drivers

[lars-t-hansen]

TODO short term:

• UUID for AMD devices, can get this with the amdsmi library if available, but should not fail if not available
• A little better documentation in sonar-nvidia.h
• Linking issues on the build bot and indeed on the dev systems if the "right" gcc is not loaded, needs exploration
• amd_smi.rs: extract architecture somehow?
• amd_smi.rs: extract mem_pct somehow? do we need it?
• sonar-nvidia.c etc - perf state need not be formatted as string at this api, better if it isn't
• sonar-nvidia.c etc - compute mode can maybe be transmitted as some well-known integer instead
• (fixing those two allows us to derive the Default for the nvidia crad state)
• sonar-amd.c etc - clean up buffer sizes as much as possible, and fix the comment about this
• more testing, perhaps esp nvidia systems, the AMD tests look ok for now
• sysinfo now reports ... @ 7GB but it's 7.95GB or thereabouts, it should round and not truncate

TODO longer term:

• check all available hardware to make sure we don't need the fallback solutions
  • Saga a100
  • Saga accel
• remove the fallback code entirely
• File bug about dealing with .so versioning, see comment below
• Fix the documentation: it says that we run nvidia-smi and rocm-smi, but after this patch we don't

[lars-t-hansen]

This is WIP, but essentially complete for NVIDIA. AMD will be the same - though the AMD documentation is mostly for a C++ interface, there is a lower-level, documented C interface that will be fine for us.

As noted on the bug, there are issues with linking, and the fact that the static libraries (or at least object files) must be built on hosts that have the GPU SDKs installed. For now, the .a files are part of this patch.

[lars-t-hansen]

For reasons not understood, linking against the static libraries works on some hosts but fails on others (including on CI).

[lars-t-hansen]

The linking issue appears to be a GCC problem. If I ensure that GCC11 is loaded with the appropriate module load command then linking works on both ML1 (cuda+gcc11) and ML4 (hipSYCL+gcc11) ; with the default GCC (gcc8), it does not.

[lars-t-hansen]

Since betzy is literally on fire I guess I won't be checking those nodes but I'm pretty satisfied that this approach will work everywhere, for now. I'm a little worried about having to load rocm_smi64.so.7 (why ".7"??!) but that's how it is. If we had an error channel it would be possible to check for other versions if the .7 is suddenly not there, and report back if other versions are found.

Presumably nvidia has the same problem. There, we load a file that is a symlink to the .1 file which is a symlink to a file that has the driver version number encoded in it. It's possible we should load the .1 file directly, and then have a similar fallback / reporting mechanism for a version change.

I think dealing with the versioning will have to be a separate bug.

[bast]

Reviewing ...

[lars-t-hansen]

@bast, it'd be good to get this reviewed asap to see if this approach is sensible and acceptable, but probably best not to land quite yet even if review is positive, I'd like to fix the outstanding items on the to-do list above.

[bast]

I finally read up on #87 and also the diff and I think this is a good solution. It increases complexity a bit but also solves other problems and I would not be able to suggest a better solution at this point.

[lars-t-hansen]

I finally read up on #87 and also the diff and I think this is a good solution. It increases complexity a bit but also solves other problems and I would not be able to suggest a better solution at this point.

OK, thanks.

I'm not real fond of including compiled code - it's basically a vector for a hard-to-spot supply chain attack - but given that the data structures in the header files are not fully documented and available only on some systems there's really no good way for us to avoid the C compiler.

We might consider requiring 2fa when pushing to the official repo, and some routines for maintaining the integrity of the binary blobs. I'm already developing mostly on my own fork so this will hardly impact me.

[bast]

Good point about 2FA and careful review.

[lars-t-hansen]

@bast, I think this has no outstanding known issues and is ready for final review.

[bast]

Thank you for all the work!