Test Ruisi's commit #138
Test Ruisi's commit #138
Conversation
[ghstack-poisoned]
[ghstack-poisoned]
meta-cla
bot
added
the
CLA Signed
ruisizhang123
left a comment
ruisizhang123
left a comment
There was a problem hiding this comment.
LGTM! thank you for adding tests.
examples/example_llama3.py
Outdated
| if True: | ||
| backend = "nccl" | ||
| fake_store = None | ||
| kwargs = {} |
There was a problem hiding this comment.
qq: is there a reason we want to use a real pg instead of fake pg? iiuc, the test job in github only has 4 gpus. 🤔 we have to update world_size thing to 4 if we decide to use real pg.
There was a problem hiding this comment.
The reason was because I thought the benchmarking might need a real PG, although I haven't really tested to see if it works with a FakeStore
There was a problem hiding this comment.
ohhhh, I see. I tried using fake PG and the code went through. The current llama3 tests are use fake PG, but there are new issues with memory estimation caused by a recent change in pytorch. I plan to land my autobucketing PR today, and will land this one after the pytorch issue is fixed.
ruisizhang123
force-pushed
the
fmassa/test_ruisi_branch
branch
from
f2dd1ef to
a368fbc
Compare
ruisizhang123
force-pushed
the
fmassa/test_ruisi_branch
branch
from
a368fbc to
f1cc5fc
Compare
No description provided.