remove SharedDDP as it is deprecated

[statelesshz]

What does this PR do?

As mentioned previously(see), fairscale's ShardedDDP is deprecated, and PyTorch FSDP is the recommended method for scaling to large NN models. Now it's time to say goodbye to this library👋.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerz Good day. Could you please review this PR? Thanks😄

[statelesshz]

@sgugger sorry for bothering you, but would you mind taking a look at this PR?

[sgugger]

cc @muellerzr and @pacman100 who will take over the Trainer.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[statelesshz]

Rebase my commits to master HEAD to fix the merge conflict.
BTW, Is this PR still under reviewing? Any review suggestions? Please let me know if there is anything else that needs to be done.

[pacman100]

Thank you @statelesshz for removing the FairScale support as it is deprecated. The changes LGTM except for the TPU-related bits which Zach can review. Left a comment.

[muellerzr]

Thanks! This is looking great! Just a small note in terms of gradient scaling. We need to either keep this for now in this PR and merge, or coordinate a PR in accelerate with the right logic before this merges. I don't particularly have a leaning one way or another :)

[statelesshz]

@muellerzr Could you please take a second look at this PR?
Some modifications were made based on the code review comments.

[muellerzr]

Thanks! This all looks great to me, do you have access to a multi-gpu system by chance to run the sharded DDP yourself/test? Probably also good to have @pacman100 give it one more look as well :)

[statelesshz]

I've tested this PR using 4xA100-80G on FastChat with the following scripts

CUDA_VISIBLE_DEVICES=4,5,6,7  torchrun --nproc_per_node=4 --master_port=20001 fastchat/train/train_mem.py \
    --model_name_or_path lmsys/vicuna-7b-v1.5  \
    --data_path data/dummy_conversation.json \
    --bf16 True \
    --output_dir output_vicuna \
    --num_train_epochs 1 \
    --max_steps 10 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 10 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard offload auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True \
    --model_max_length 2048 \
    --gradient_checkpointing False \
    --lazy_preprocess True

[statelesshz]

@pacman100 Could you please take a look once again :-) I think it's ready to be merged.

[statelesshz]

Resolve merge conflicts by rebasing to main branch

[statelesshz]

Rebasing my commits to master HEAD and resolving merge conflicts

[statelesshz]

Hi there. This PR is approved and the tests are green :D
cc @muellerzr and @pacman100

[statelesshz]

This PR is approved and the tests are green. @muellerzr Could you help to merge it?

[LysandreJik]

Thank you @statelesshz!

[LysandreJik]

Let me rebase quickly and merge if tests are green

[LysandreJik]

Great, LGTM! Thanks @statelesshz