Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Model] Add HuggingFace GPT-2 #45

Merged
merged 7 commits into from
Feb 7, 2023
Merged

[Model] Add HuggingFace GPT-2 #45

merged 7 commits into from
Feb 7, 2023

Conversation

comaniac
Copy link
Contributor

@comaniac comaniac commented Feb 7, 2023

Description

Add an example of HuggingFace GPT-2. Later we should rename examples/gpt to examples/gpt-neo to reduce the confusion.

Example benchmark commands:

deepspeed ./examples/gpt2/deepspeed_hf.py --pmp 2 --tmp 2 --batch_size 32 --micro_batch_size 4 --model_name gpt2-xl --iter_nums 170 --hidden-size 2048 --nlayers 24 --num-attn-heads 16 --dropout 0.1 --activation_function gelu --seq_len 2048

Checklist

  • PR's title starts with a category (e.g. [Bugfix], [Model], [Tutorial], etc)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage
  • Code is well-documented

cc @szhengac

@szhengac szhengac merged commit bc9094b into awslabs:main Feb 7, 2023
@comaniac comaniac deleted the gpt2 branch February 8, 2023 00:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants