New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove requirement to initialize MPI for tensor+data parallelism #20

Open

siddharth9820 wants to merge 2 commits into main from non-mpi

Collaborator

siddharth9820 commented Feb 5, 2024 •

edited

Loading

MPI.Init() can cause unnecessary delays or even hangs at scale. This PR makes it such that MPI.Init() is optional.
Adding an option to only cache the weights of the first MLP layer, which has least overlap with the preceeding layer,

siddharth9820 added 2 commits

January 24, 2024 07:23


          remove mandatory mpi requirement

0a0f7df


          remove mpi, add 2 levels of caching

658be49

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet