You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Nov 3, 2023. It is now read-only.
Create a new "distributed" folder inside parlai/nn. Inside parlai/nn/distributed/init.py, create a helper class which finds and instantiates the "right" version of data parallel. Something like
classDistributedFactory:
@classmethoddefadd_cmdline_args(cls, parser, partial_opt):
# --distributed-method here, as well as moving options from https://github.com/facebookresearch/ParlAI/blob/67433e376fc361dee5aa045cb6bb2b68d3faa478/parlai/core/params.py#L760-L768@classmethoddeffactory(cls, model: torch.nn.Module, opt: Opt)
# based on opt['distributed_method'], instantiate a DDP or ShardedDDP object
Upgrade TGA/TCA/TRA to use this helper.
It would also be nice to use a @register_distributed pattern (see how we do it for Agents, Teachers, and scripts presently), so that we can add internal-only approaches.
The text was updated successfully, but these errors were encountered:
Let's add support for Fairscale's Sharded DDP.
Right now we hardcode that we must use PyTorch's DDP, but let's generalize this:
ParlAI/parlai/core/torch_generator_agent.py
Lines 550 to 552 in 67433e3
Create a new "distributed" folder inside parlai/nn. Inside parlai/nn/distributed/init.py, create a helper class which finds and instantiates the "right" version of data parallel. Something like
Upgrade TGA/TCA/TRA to use this helper.
It would also be nice to use a
@register_distributed
pattern (see how we do it for Agents, Teachers, and scripts presently), so that we can add internal-only approaches.The text was updated successfully, but these errors were encountered: