Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add KD distributed recipe #1631

Merged
merged 50 commits into from
Oct 29, 2024
Merged

Add KD distributed recipe #1631

merged 50 commits into from
Oct 29, 2024

Commits on Sep 5, 2024

  1. sft recipes to eval kd

    @lindawangg
    lindawangg committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    2933cd6 View commit details
    Browse the repository at this point in the history

  2. setup kd files

    @lindawangg
    lindawangg committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    bff065a View commit details
    Browse the repository at this point in the history

  3. delete test config

    @lindawangg
    lindawangg committed Sep 5, 2024
    Configuration menu
    Copy the full SHA
    9dd7b47 View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2024

  1. added student config

    @lindawangg
    lindawangg committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    a39e99c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-initial-kd-recipe

    @lindawangg
    lindawangg committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    6dbcd38 View commit details
    Browse the repository at this point in the history

  3. added teacher model loading

    @lindawangg
    lindawangg committed Sep 6, 2024
    Configuration menu
    Copy the full SHA
    0c4e4f9 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2024

  1. added loss

    @lindawangg
    lindawangg committed Sep 7, 2024
    Configuration menu
    Copy the full SHA
    380f267 View commit details
    Browse the repository at this point in the history

Commits on Sep 10, 2024

  1. kd initial experiment config

    @lindawangg
    lindawangg committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    da2b4bb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-initial-kd-recipe

    @lindawangg
    lindawangg committed Sep 10, 2024
    Configuration menu
    Copy the full SHA
    8beaca0 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2024

  1. separated out loss func and added test

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b54929a View commit details
    Browse the repository at this point in the history

  2. added documentation

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    b31c56d View commit details
    Browse the repository at this point in the history

  3. added prereq command to config

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    fe5ed97 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into add-initial-kd-recipe

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    8b9ea41 View commit details
    Browse the repository at this point in the history

  5. re-add 8B config

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    3f7fe70 View commit details
    Browse the repository at this point in the history

  6. added kd ratio

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    a87aa0c View commit details
    Browse the repository at this point in the history

  7. revert 8b config

    @lindawangg
    lindawangg committed Sep 11, 2024
    Configuration menu
    Copy the full SHA
    f5feac4 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. add kd recipe test

    @lindawangg
    lindawangg committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    8c3c42a View commit details
    Browse the repository at this point in the history

  2. mark as integration test

    @lindawangg
    lindawangg committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    6ba0514 View commit details
    Browse the repository at this point in the history

  3. add save and load weights test

    @lindawangg
    lindawangg committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    04ea649 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. fix comments 1

    @lindawangg
    lindawangg committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    62faa1d View commit details
    Browse the repository at this point in the history

  2. address kd loss test comments

    @lindawangg
    lindawangg committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    bf15406 View commit details
    Browse the repository at this point in the history

  3. change to qwen2

    @lindawangg
    lindawangg committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    ac9eb0e View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2024

  1. addressing recipe comments

    @lindawangg
    lindawangg committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    87a80b6 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-initial-kd-recipe

    @lindawangg
    lindawangg committed Sep 15, 2024
    Configuration menu
    Copy the full SHA
    1fc3f64 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. distributed recipe

    @lindawangg
    lindawangg committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    106aa3e View commit details
    Browse the repository at this point in the history

  2. remove todo comment and test activation checkpointing

    @lindawangg
    lindawangg committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    0f4e922 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into add-initial-kd-recipe

    @lindawangg
    lindawangg committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    22fddca View commit details
    Browse the repository at this point in the history

  4. Merge branch 'add-initial-kd-recipe' into add-kd-distributed

    @lindawangg
    lindawangg committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    526a4dc View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. qwen2 distributed recipe

    @lindawangg
    lindawangg committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    0bb49dc View commit details
    Browse the repository at this point in the history

  2. added to recipe registry

    @lindawangg
    lindawangg committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    c73857d View commit details
    Browse the repository at this point in the history

  3. fdsp teacher model

    @lindawangg
    lindawangg committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    59eff44 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. added kd distributed test

    @lindawangg
    lindawangg committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    85d76bb View commit details
    Browse the repository at this point in the history

  2. fixed command

    @lindawangg
    lindawangg committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    dba57c4 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Sep 19, 2024
    Configuration menu
    Copy the full SHA
    04e2282 View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. changed to knowledge_distillation

    @lindawangg
    lindawangg committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    44123b9 View commit details
    Browse the repository at this point in the history

  2. cleaned up tests

    @lindawangg
    lindawangg committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    a04244d View commit details
    Browse the repository at this point in the history

  3. added gpu test

    @lindawangg
    lindawangg committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    1ff9934 View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    703e7dc View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    0031bfb View commit details
    Browse the repository at this point in the history

  2. added llama3 config and addressed comments

    @lindawangg
    lindawangg committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    307791d View commit details
    Browse the repository at this point in the history

  3. added custom sharding layers

    @lindawangg
    lindawangg committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    fefc24d View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    15c5be2 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. add test_loss back in

    @lindawangg
    lindawangg committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    46473ee View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    2e212ec View commit details
    Browse the repository at this point in the history

  2. rebase

    @lindawangg
    lindawangg committed Oct 24, 2024
    Configuration menu
    Copy the full SHA
    557396e View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2024

  1. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    4d376e3 View commit details
    Browse the repository at this point in the history

  2. grad accumulation changes

    @lindawangg
    lindawangg committed Oct 25, 2024
    Configuration menu
    Copy the full SHA
    53c47ba View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2024

  1. remove extra num_tokens

    @lindawangg
    lindawangg committed Oct 26, 2024
    Configuration menu
    Copy the full SHA
    f193d02 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2024

  1. addressed comments

    @lindawangg
    lindawangg committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    227e69d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into add-kd-distributed

    @lindawangg
    lindawangg committed Oct 28, 2024
    Configuration menu
    Copy the full SHA
    cf5f01a View commit details
    Browse the repository at this point in the history