Skip to content

island option #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

island option #16

wants to merge 1 commit into from

Conversation

thowell
Copy link
Collaborator

@thowell thowell commented Mar 21, 2025

an argument island is added to put_model. island determines if tile operations use multiple tiles (island=True) or one tile (island=False). additionally, the flag island is added to the viewer.

@thowell thowell requested a review from erikfrey March 21, 2025 18:46
@erikfrey
Copy link
Collaborator

What's the use case for allowing users to change the tile behavior?

@thowell
Copy link
Collaborator Author

thowell commented Mar 21, 2025

cuda graph capture failure can occur for scenes where individual islands have different numbers of dofs.

for example, this xml

<mujoco>
  <worldbody>
    <body pos="-1 0 0">
      <joint type="slide" axis="1 0 0"/>
      <geom type="sphere" size=".1"/>
    </body>
    <body pos="1 0 0">
      <joint type="slide" axis="1 0 0"/>
      <joint type="slide" axis="0 1 0"/>
      <geom type="sphere" size=".1"/>
    </body>
  </worldbody>
</mujoco>

produces

raise RuntimeError(f"CUDA graph capture failed. {runtime.get_error_string()}")
RuntimeError: CUDA graph capture failed. Warp CUDA error 900: operation not permitted when stream is capturing (in function cuda_load_module, /builds/omniverse/warp/warp/native/warp.cu:3417)

when loaded in the viewer.

the scene loads in the viewer when island=False.

Warp 1.7.0.dev20250321
CUDA Toolkit 12.8, Driver 12.2

@adenzler-nvidia
Copy link
Collaborator

I do think we need a way to switch between one island and multiple, there is probably a corner in that space where it doesn't make sense to split things. That being said - I'm surprised that xml above doesn't work. I thought the entire point of that tiling was to exploit block-sparsity and batch by size to just run a kernel for each size?

@erikfrey
Copy link
Collaborator

Hmm, this model works OK for me, this command succeeds:

mjwarp-testspeed --function=step --mjcf=taylor_example.xml --batch_size=8192 --iterations=2 --ls_iterations=10 --solver=newton

@thowell
Copy link
Collaborator Author

thowell commented Mar 25, 2025

@erikfrey after upgrading cuda and upgrading the driver the test xml above works

@thowell
Copy link
Collaborator Author

thowell commented Mar 25, 2025

@adenzler-nvidia @erikfrey should i close this pr or do we want to incorporate this feature?

@erikfrey
Copy link
Collaborator

Let’s close it for now but can always come back if we think we need this flag.

@erikfrey erikfrey closed this Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikfrey erikfrey Awaiting requested review from erikfrey

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@thowell @erikfrey @adenzler-nvidia