Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add FSDP2 support for low-bit optimizers #484

Merged
merged 43 commits into from
Jul 16, 2024
Merged

Add FSDP2 support for low-bit optimizers #484

merged 43 commits into from
Jul 16, 2024

Commits on Jul 8, 2024

  1. add test for FSDP2

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    9331495 View commit details
    Browse the repository at this point in the history

  2. fix optim

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    e159e93 View commit details
    Browse the repository at this point in the history

  3. add some fsdp2 ops

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    2fc8830 View commit details
    Browse the repository at this point in the history

  4. add DTensor

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    e30f142 View commit details
    Browse the repository at this point in the history

  5. try DTensor

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    a88750c View commit details
    Browse the repository at this point in the history

  6. undo changes

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    9593c07 View commit details
    Browse the repository at this point in the history

  7. add DTensor support for adamw

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    e3408d8 View commit details
    Browse the repository at this point in the history

  8. update imports

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    a71c9bc View commit details
    Browse the repository at this point in the history

  9. remove whitespace

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    1aa4600 View commit details
    Browse the repository at this point in the history

  10. fix view issue with compiler

    @gau-nernst
    gau-nernst committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    17e4cb8 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Merge branch 'pytorch:main' into low_bit_optim_fsdp2

    @gau-nernst
    gau-nernst authored Jul 11, 2024
    Configuration menu
    Copy the full SHA
    a0aef48 View commit details
    Browse the repository at this point in the history

  2. small refactoring

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    8bfff6d View commit details
    Browse the repository at this point in the history

  3. static horizontal fusion

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    7d28556 View commit details
    Browse the repository at this point in the history

  4. add test for CPU offload (may timeout)

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    96c4286 View commit details
    Browse the repository at this point in the history

  5. update note

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    f9817fb View commit details
    Browse the repository at this point in the history

  6. update benchmarks

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    4549aa1 View commit details
    Browse the repository at this point in the history

  7. remove CPUOffloadPolicy test

    @gau-nernst
    gau-nernst committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    bb34b00 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Merge branch 'pytorch:main' into low_bit_optim_fsdp2

    @gau-nernst
    gau-nernst authored Jul 12, 2024
    Configuration menu
    Copy the full SHA
    ddbdd9c View commit details
    Browse the repository at this point in the history

  2. fix version test

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    5ffab6d View commit details
    Browse the repository at this point in the history

  3. fix typo

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    ccc5904 View commit details
    Browse the repository at this point in the history

  4. add custom 4-bit

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    99ac35d View commit details
    Browse the repository at this point in the history

  5. revert version check

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    d39d750 View commit details
    Browse the repository at this point in the history

  6. replace 4-bit adam

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    8b19e23 View commit details
    Browse the repository at this point in the history

  7. switch 4-bit impl

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e859fbd View commit details
    Browse the repository at this point in the history

  8. refactor

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    0b05caa View commit details
    Browse the repository at this point in the history

  9. update test

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    a0390af View commit details
    Browse the repository at this point in the history

  10. update test. some fixes

    @gau-nernst
    gau-nernst committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    4b0d22c View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. bring back 4bit subclass

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    f9e40e5 View commit details
    Browse the repository at this point in the history

  2. fixes

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    b7ccb60 View commit details
    Browse the repository at this point in the history

  3. add DTensor to custom optim_4bit

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    9a5a722 View commit details
    Browse the repository at this point in the history

  4. fix default block_size

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    108df09 View commit details
    Browse the repository at this point in the history

  5. separate scale and quantize

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    7bbf46f View commit details
    Browse the repository at this point in the history

  6. add 4-bit optim fix

    @gau-nernst
    gau-nernst committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    f75c2ee View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2024

  1. fix 4-bit subclass. replace default 4-bit optim impl

    @gau-nernst
    gau-nernst committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    c448e86 View commit details
    Browse the repository at this point in the history

  2. update table

    @gau-nernst
    gau-nernst committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    db207e4 View commit details
    Browse the repository at this point in the history

  3. add BF16 smoke test

    @gau-nernst
    gau-nernst committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    0c39755 View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. remove unused 4-bit impl

    @gau-nernst
    gau-nernst committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    320b0e5 View commit details
    Browse the repository at this point in the history

Commits on Jul 16, 2024

  1. Merge branch 'pytorch:main' into low_bit_optim_fsdp2

    @gau-nernst
    gau-nernst authored Jul 16, 2024
    Configuration menu
    Copy the full SHA
    ac72ea3 View commit details
    Browse the repository at this point in the history

  2. debug FSDP test

    @gau-nernst
    gau-nernst committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    6b1d7d2 View commit details
    Browse the repository at this point in the history

  3. print cache size limit

    @gau-nernst
    gau-nernst committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    d936a28 View commit details
    Browse the repository at this point in the history

  4. increase cache size limit

    @gau-nernst
    gau-nernst committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    fe5864c View commit details
    Browse the repository at this point in the history

  5. add compute capability check for FP8 in FSDP test

    @gau-nernst
    gau-nernst committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    fb12060 View commit details
    Browse the repository at this point in the history

  6. revert debug tests

    @gau-nernst
    gau-nernst committed Jul 16, 2024
    Configuration menu
    Copy the full SHA
    87ae147 View commit details
    Browse the repository at this point in the history