slurm_scheduler: add more runopts. Fixes #389

[d4l3k]

This adds 4 runopts:

• constraint
• comment
• mail-user
• mail-type

Test plan:

Added unit tests for all fields

Integ test for comment

Hard for us to test constraint since our cluster doesn't have any node constraints

torchx run --wait --scheduler slurm --scheduler_args partition=compute,time=10,comment=hello,mail-user=<addr>,mail-type=END utils.echo --num_replicas 3

This passes but I don't believe the mail from our cluster is deliverable.

[codecov]

Codecov Report

Merging #391 (e6285e0) into main (8b62ea8) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #391      +/-   ##
==========================================
+ Coverage   93.96%   93.98%   +0.01%     
==========================================
  Files          64       64              
  Lines        3565     3574       +9     
==========================================
+ Hits         3350     3359       +9     
  Misses        215      215              

Impacted Files	Coverage Δ
torchx/schedulers/slurm_scheduler.py	98.11% <100.00%> (+0.11%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8b62ea8...e6285e0. Read the comment docs.

[facebook-github-bot]

@d4l3k has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.