Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimizer CPU offload for single GPU training #584

Merged
merged 31 commits into from
Aug 6, 2024
Merged

Optimizer CPU offload for single GPU training #584

merged 31 commits into from
Aug 6, 2024

Commits on Aug 1, 2024

  1. initial commit

    @gau-nernst
    gau-nernst committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3cd42d2 View commit details
    Browse the repository at this point in the history

  2. use fused=True by default for PyTorch adam

    @gau-nernst
    gau-nernst committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    c044d88 View commit details
    Browse the repository at this point in the history

  3. detach param

    @gau-nernst
    gau-nernst committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    d85b172 View commit details
    Browse the repository at this point in the history

  4. try overlap D2H grad copy with backward

    @gau-nernst
    gau-nernst committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    d468e6f View commit details
    Browse the repository at this point in the history

  5. add customizable profile num steps

    @gau-nernst
    gau-nernst committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    d7a07eb View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. add v2

    @gau-nernst
    gau-nernst committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    fe653e9 View commit details
    Browse the repository at this point in the history

  2. fix various bugs

    @gau-nernst
    gau-nernst committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    8ae42c3 View commit details
    Browse the repository at this point in the history

  3. fix v1 impl

    @gau-nernst
    gau-nernst committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    b2c00e5 View commit details
    Browse the repository at this point in the history

  4. add full BF16 option

    @gau-nernst
    gau-nernst committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    68835e3 View commit details
    Browse the repository at this point in the history

  5. change n_profile_steps to 5

    @gau-nernst
    gau-nernst committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    b5393cb View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2024

  1. add v3

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    3069b23 View commit details
    Browse the repository at this point in the history

  2. fix gradient accumulation

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    7af8518 View commit details
    Browse the repository at this point in the history

  3. add note

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    5ff2e5a View commit details
    Browse the repository at this point in the history

  4. add deepspeed offload

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    a8a7b5a View commit details
    Browse the repository at this point in the history

  5. update deepspeed config

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    40aea0c View commit details
    Browse the repository at this point in the history

  6. add some notes

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    dff2f9c View commit details
    Browse the repository at this point in the history

  7. update instructions. make some packages optional. change to AdamW

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    bd8db68 View commit details
    Browse the repository at this point in the history

  8. add last updated ordered dict

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    c3883ce View commit details
    Browse the repository at this point in the history

  9. update deepspeed version

    @gau-nernst
    gau-nernst committed Aug 3, 2024
    Configuration menu
    Copy the full SHA
    0e9235c View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2024

  1. remove old versions

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    b6e4c6a View commit details
    Browse the repository at this point in the history

  2. update docs

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    c514dba View commit details
    Browse the repository at this point in the history

  3. say deepspeed is untuned

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    cfdfe5d View commit details
    Browse the repository at this point in the history

  4. add test

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    c4ea68b View commit details
    Browse the repository at this point in the history

  5. add test for offload_gradients. update benchmark script

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    6478be9 View commit details
    Browse the repository at this point in the history

  6. update benchmark results. fix test. fix benchmark script

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    03cf0ad View commit details
    Browse the repository at this point in the history

  7. fix language

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    a144b22 View commit details
    Browse the repository at this point in the history

  8. add save and load

    @gau-nernst
    gau-nernst committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    d344817 View commit details
    Browse the repository at this point in the history

  9. Merge branch 'pytorch:main' into optim_cpu_offload

    @gau-nernst
    gau-nernst authored Aug 4, 2024
    Configuration menu
    Copy the full SHA
    fc358b1 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'main' into optim_cpu_offload

    @gau-nernst
    gau-nernst committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    5a5253e View commit details
    Browse the repository at this point in the history

  2. pre-allocate CPU params. add note about gradient clipping

    @gau-nernst
    gau-nernst committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    7aa31eb View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. update README and remove unused imports

    @gau-nernst
    gau-nernst committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    231a6ef View commit details
    Browse the repository at this point in the history