slurm_scheduler: autodetect nomem setting

[d4l3k]

This automatically infers nomem by using sinfo to get the Memory setting for the specified partition. If we can't determine the memory setting or its <= 1000 MB we disable memory allocation checks.

#405

Test plan:

CI

Slurm integration tests w/ nomem removed

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[codecov]

Codecov Report

Merging #461 (7fefe00) into main (37c76eb) will decrease coverage by 0.01%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##             main     #461      +/-   ##
==========================================
- Coverage   94.59%   94.58%   -0.02%     
==========================================
  Files          65       65              
  Lines        3833     3858      +25     
==========================================
+ Hits         3626     3649      +23     
- Misses        207      209       +2     

Impacted Files	Coverage Δ
torchx/schedulers/slurm_scheduler.py	96.84% <92.85%> (-0.62%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37c76eb...7fefe00. Read the comment docs.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has updated the pull request. You must reimport the pull request before landing.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.