Skip to content

Commit

Permalink
Add hint in docs for how to use shared memory (#6036)
Browse files Browse the repository at this point in the history
  • Loading branch information
awaelchli authored Feb 17, 2021
1 parent e0bb33c commit ad36c7b
Showing 1 changed file with 27 additions and 1 deletion.
28 changes: 27 additions & 1 deletion docs/source/benchmarking/performance.rst
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,34 @@ Refer to the :doc:`distributed computing guide for more details <../advanced/mul


Sequential Model Parallelism with Checkpointing
---------------------------------------------------------------------
-----------------------------------------------
PyTorch Lightning integration for Sequential Model Parallelism using `FairScale <https://github.com/facebookresearch/fairscale>`_.
Sequential Model Parallelism splits a sequential module onto multiple GPUs, reducing peak GPU memory requirements substantially.

For more information, refer to :ref:`sequential-parallelism`.


Preload Data Into RAM
---------------------

When your training or preprocessing requires many operations to be performed on entire dataset(s) it can
sometimes be beneficial to store all data in RAM given there is enough space.
However, loading all data at the beginning of the training script has the disadvantage that it can take a long
time and hence it slows down the development process. Another downside is that in multiprocessing (e.g. DDP)
the data would get copied in each process.
One can overcome these problems by copying the data into RAM in advance.
Most UNIX-based operating systems provide direct access to tmpfs through a mount point typically named ``/dev/shm``.

0. Increase shared memory if necessary. Refer to the documentation of your OS how to do this.

1. Copy training data to shared memory:

.. code-block:: bash
cp -r /path/to/data/on/disk /dev/shm/
2. Refer to the new data root in your script or command line arguments:

.. code-block:: python
datamodule = MyDataModule(data_root="/dev/shm/my_data")

0 comments on commit ad36c7b

Please sign in to comment.