Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds throughput benchmarking scripts for different learning workflows #759

Merged
merged 9 commits into from
Aug 2, 2024

Conversation

kellyguo11
Copy link
Contributor

Description

  • Removes unneeded extensions from headless app file
  • Limits CPU threads when running multi-GPU training to minimize thread context switching
  • Apply multi GPU renderer setting to avoid unnecessary GPU context initialization
  • Decreases 8k SH startup time on 8 GPUs from > 5 min to 1 min
  • Adds benchmarking scripts

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

kellyguo11 and others added 5 commits July 18, 2024 10:44
# Description

- Adds a new script to run benchmark without RL in the loop
- Adds a new script to run benchmark with RL Games
- Adds a new script to run benchmark with RSL RL

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Co-authored-by: Alexander <143108850+nv-apoddubny@users.noreply.github.com>
Copy link
Contributor

@Mayankm96 Mayankm96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly looks fine. Would be nice if we can separate the two things into different MRs (startup time and benchmarking) since they are independent of each other (and fairly important changes/additions).

@Mayankm96 Mayankm96 added the enhancement New feature or request label Aug 1, 2024
@Mayankm96 Mayankm96 changed the title Improves startup time for headless and multi GPU; Adds benchmarking scripts Adds throughput benchmarking scripts for different learning workflows Aug 1, 2024
Copy link
Collaborator

@jsmith-bdai jsmith-bdai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for adding!

CC: @hhansen-bdai as this might relate to device selection efforts (specifically the .kit files)

@kellyguo11 kellyguo11 merged commit 809d090 into isaac-sim:main Aug 2, 2024
1 of 2 checks passed
iamdrfly pushed a commit to iamdrfly/IsaacLab that referenced this pull request Nov 21, 2024
…isaac-sim#759)

# Description
- Removes unneeded extensions from headless app file
- Limits CPU threads when running multi-GPU training to minimize thread
context switching
- Apply multi GPU renderer setting to avoid unnecessary GPU context
initialization
- Decreases 8k SH startup time on 8 GPUs from > 5 min to 1 min
- Adds benchmarking scripts

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------

Co-authored-by: Alexander <143108850+nv-apoddubny@users.noreply.github.com>
Co-authored-by: David Hoeller <dhoeller@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants