Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds support for PBS job scheduler based clusters #605

Merged
merged 20 commits into from
Jul 4, 2024
Merged

Adds support for PBS job scheduler based clusters #605

merged 20 commits into from
Jul 4, 2024

Conversation

shafeef901
Copy link
Contributor

@shafeef901 shafeef901 commented Jun 30, 2024 •
edited
Loading

Description

This pull request adds support for running IsaacLab on clusters that use PBS job schedulers (e.g. Franklin@IIT). Currently, it only supports SLURM. The job submission scripts have been modified to choose between SLURM or PBS job schedulers. The user can opt for the required job scheduler from docker/cluster/.env.base under cluster specific settings.

Tested successfully on Franklin@IIT HPC.

Fixes #599

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist

  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have run all the tests with ./isaaclab.sh --test and they pass
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@Mayankm96 Mayankm96 added documentation Improvements or additions to documentation enhancement New feature or request labels Jul 1, 2024
@Mayankm96 Mayankm96 changed the title Adds support for PBS job scheduler based clusters (Issue #599) Adds support for PBS job scheduler based clusters Jul 1, 2024
pascal-roth
pascal-roth requested changes Jul 3, 2024
View reviewed changes
Copy link
Collaborator

@pascal-roth pascal-roth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, thanks for making this PR! Major comment is that we should separate the submit_job.sh file for both workflows to improve the structure and make it easier for people to understand. After that change, we can merge the changes

docker/.env.base Outdated Show resolved Hide resolved
docker/cluster/submit_job.sh Outdated Show resolved Hide resolved
docker/cluster/submit_job.sh Outdated Show resolved Hide resolved
docker/container.sh Outdated Show resolved Hide resolved
@shafeef901
Copy link
Contributor Author

All requested changes sorted out as suggested :)

Mayankm96
Mayankm96 reviewed Jul 3, 2024
View reviewed changes
docker/container.sh Outdated Show resolved Hide resolved
Mayankm96
Mayankm96 reviewed Jul 3, 2024
View reviewed changes
docker/container.sh Outdated Show resolved Hide resolved
Mayankm96
Mayankm96 requested changes Jul 4, 2024
View reviewed changes
Copy link
Contributor

@Mayankm96 Mayankm96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changes. You had ### for sub-section but this isn't RST compliant.

I added suggestions on the sub-sections now to use RST heading style.

You can remove the title as it isn't needed anymore.

Job Submission and Execution
----------------------------

shafeef901 and others added 5 commits July 4, 2024 10:10
@shafeef901 @Mayankm96
Apply suggestions from code review
65e17a3 
Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com>
Signed-off-by: Shafeef Omar <shafeef901@gmail.com>
@shafeef901 @Mayankm96
Apply suggestions from code review
937d102 
Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com>
Signed-off-by: Shafeef Omar <shafeef901@gmail.com>
@shafeef901
Merge branch 'isaac-sim:main' into pbs-cluster
9a0e766
@shafeef901
Update cluster.rst
6833aca 
Signed-off-by: Shafeef Omar <shafeef901@gmail.com>
@shafeef901
Update cluster.rst
5cd9257 
Signed-off-by: Shafeef Omar <shafeef901@gmail.com>
@Mayankm96 Mayankm96 merged commit 08ebac7 into isaac-sim:main Jul 4, 2024
1 check passed
mohanksriram pushed a commit to mohanksriram/IsaacLab that referenced this pull request Jul 12, 2024
@shafeef901 @mohanksriram
Adds support for PBS job scheduler based clusters (isaac-sim#605)
e33eb7d 
# Description

This pull request adds support for running IsaacLab on clusters that use
PBS job schedulers (e.g. Franklin@IIT). Currently, it only supports
SLURM. The job submission scripts have been modified to choose between
SLURM or PBS job schedulers. The user can opt for the required job
scheduler from `docker/cluster/.env.base` under cluster specific
settings.

Tested successfully on Franklin@IIT HPC.

Fixes isaac-sim#599 

## Type of change

- New feature (non-breaking change which adds functionality)
- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have run all the tests with `./isaaclab.sh --test` and they pass
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
fatimaanes pushed a commit to fatimaanes/omniperf that referenced this pull request Aug 8, 2024
@shafeef901
Adds support for PBS job scheduler based clusters (isaac-sim#605)
13f71fa 
# Description

This pull request adds support for running IsaacLab on clusters that use
PBS job schedulers (e.g. Franklin@IIT). Currently, it only supports
SLURM. The job submission scripts have been modified to choose between
SLURM or PBS job schedulers. The user can opt for the required job
scheduler from `docker/cluster/.env.base` under cluster specific
settings.

Tested successfully on Franklin@IIT HPC.

Fixes isaac-sim#599 

## Type of change

- New feature (non-breaking change which adds functionality)
- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have run all the tests with `./isaaclab.sh --test` and they pass
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
iamdrfly pushed a commit to iamdrfly/IsaacLab that referenced this pull request Nov 21, 2024
@shafeef901
Adds support for PBS job scheduler based clusters (isaac-sim#605)
ba0bf99 
# Description

This pull request adds support for running IsaacLab on clusters that use
PBS job schedulers (e.g. Franklin@IIT). Currently, it only supports
SLURM. The job submission scripts have been modified to choose between
SLURM or PBS job schedulers. The user can opt for the required job
scheduler from `docker/cluster/.env.base` under cluster specific
settings.

Tested successfully on Franklin@IIT HPC.

Fixes isaac-sim#599 

## Type of change

- New feature (non-breaking change which adds functionality)
- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have run all the tests with `./isaaclab.sh --test` and they pass
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pascal-roth pascal-roth pascal-roth requested changes

@Mayankm96 Mayankm96 Mayankm96 approved these changes

@hhansen-bdai hhansen-bdai Awaiting requested review from hhansen-bdai hhansen-bdai is a code owner

Assignees

@shafeef901 shafeef901

Labels
documentation Improvements or additions to documentation enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Proposal] Submitting training jobs for HPCs using a PBS job scheduler
3 participants
@shafeef901 @Mayankm96 @pascal-roth