GPUDirect Storage prototype tutorial #3317
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3317
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 7d9de66 with merge base 7cb6915 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| @@ -31,6 +31,7 @@ | |||
| "prototype_source/vmap_recipe", | |||
| "prototype_source/torchscript_freezing", | |||
| "prototype_source/nestedtensor", | |||
| "prototype_source/gpu_direct_storage", # requires specific filesystem + GPUDirect Storage to be set up | |||
There was a problem hiding this comment.
Doesn't it run in compat mode with a random machine?
There was a problem hiding this comment.
You need a specific filesystem
| # The loading flow is the inverse, we can ``torch.load`` under the ``torch.serialization.skip_data`` context | ||
| # manager to load everything except the storage bytes. This means that any tensors in the checkpoint will be | ||
| # created but their storages will be empty (i.e. the tensors will be created via ``torch.empty``). If the | ||
| # tensors to be loaded to are persistent, one can use the ``torch.cuda.gds.gds_register_buffer`` API to register |
There was a problem hiding this comment.
The register API is not used here?
| f.load_storage(v.untyped_storage(), offset) | ||
| assert torch.equal(v, sd[k]) | ||
|
|
||
| del f |
There was a problem hiding this comment.
Similar synchronization question as above
There was a problem hiding this comment.
I don't think synchronization is needed after the call as cuFileRead/Write are blocking operations that block until IO is complete https://docs.nvidia.com/gpudirect-storage/api-reference-guide/index.html#cufileread. You might need to synchronize before these ops (rather than after) though
| # If you are continuously saving the same state dictionary during training, you | ||
| # would only need to obtain the offsets once and the same offsets can be re-used. Similarly if tensor is going to | ||
| # be loaded to repeatedly one can use the ``torch.cuda.gds.gds_register_buffer`` which wraps | ||
| # ``cuFileBufRegister`` to register the storages as gds buffers. |
There was a problem hiding this comment.
@albanD is this better?
|
|
||
| del f | ||
|
|
||
| # Conclusion |
There was a problem hiding this comment.
This part is rendered as a code comment: https://docs-preview.pytorch.org/pytorch/tutorials/3317/prototype/gpu_direct_storage.html#using-gpudirect-storage-with-torch-save-and-torch-load
The above fix should fix it.
No description provided.