Code

RLHF with PPO#1005

Code

Merged

SalmanMohammadi merged 44 commits intopytorch:mainpytorch/torchtune:mainfrom SalmanMohammadi:ppoSalmanMohammadi/torchtune:ppoCopy head branch name to clipboardAug 5, 2024

+3,639-245

Commits

Commits on May 9, 2024

• Refactoring TransformerDecoder and adding value-head transformers

  SalmanMohammadi

  committed
  11d88a2

  Copy full SHA for 11d88a2

Commits on May 10, 2024

• adding ppo config and recipe to registry

  SalmanMohammadi

  committed
  2849ec5

  Copy full SHA for 2849ec5

Commits on May 12, 2024

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  f0c1410

  Copy full SHA for f0c1410

Commits on May 15, 2024

• implemented ppo recipe structure, advantage and return estimation, trajectory generation, tests for advantage and return estimation

  SalmanMohammadi

  committed
  57c67bf

  Copy full SHA for 57c67bf

• finished first pass implementation of ppo. added tests for ppo loss

  SalmanMohammadi

  committed
  03cba4b

  Copy full SHA for 03cba4b

• reverting changes

  SalmanMohammadi

  committed
  f50f047

  Copy full SHA for f50f047

Commits on May 16, 2024

• adding lora to ppo recipe, adding lora value head component and model builders

  SalmanMohammadi

  committed
  b034af7

  Copy full SHA for b034af7

Commits on May 19, 2024

• added lora training, added value head checkpointing and recipe resuming from checkpoints

  SalmanMohammadi

  committed
  466b683

  Copy full SHA for 466b683

Commits on May 21, 2024

• removing test model builders, adding batched generation to ppo recipe, adding support for saving value head checkpoints

  SalmanMohammadi

  committed
  928037d

  Copy full SHA for 928037d

• fixing bug in _checkpointer.py

  SalmanMohammadi

  committed
  68b6162

  Copy full SHA for 68b6162

Commits on May 30, 2024

• Adding support for user-provided masks in attention

  SalmanMohammadi

  committed
  65ca12a

  Copy full SHA for 65ca12a

Commits on May 31, 2024

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  9d8c5a8

  Copy full SHA for 9d8c5a8

Commits on Jun 4, 2024

• merging transformer custom masking, adding support for generation with padded inputs, added tests for new generation

  SalmanMohammadi

  committed
  b99102c

  Copy full SHA for b99102c

• adding functionality for truncation in generation, and further tests for generation

  SalmanMohammadi

  committed
  a1cde1c

  Copy full SHA for a1cde1c

Commits on Jun 6, 2024

• updated lora recipe to use custom generation

  SalmanMohammadi

  committed
  b032778

  Copy full SHA for b032778

• Merge branch 'pytorch:main' into ppo

  SalmanMohammadi

  authored
  f126e9a

  Copy full SHA for f126e9a

Commits on Jun 7, 2024

• added support for correct truncation and padding of responses, added rejection sampling and reward model masking, moved utils

  SalmanMohammadi

  committed
  04d514a

  Copy full SHA for 04d514a

Commits on Jun 8, 2024

• added correct mask and position id trajectory generation, score rejection sampling masking, tests for get_causal_mask

  SalmanMohammadi

  committed
  4854908

  Copy full SHA for 4854908

• bugfixing in ppo recipe. refactoring ppo_utils and tests to individual files, added tests for ppo collation

  SalmanMohammadi

  committed
  c885833

  Copy full SHA for c885833

• updating ppo_utils namespace

  SalmanMohammadi

  committed
  57d57fa

  Copy full SHA for 57d57fa

Commits on Jun 10, 2024

• fixing bug in collation, updating loss tests

  SalmanMohammadi

  committed
  cce5548

  Copy full SHA for cce5548

Commits on Jun 12, 2024

• bugfixes in masking and indexing logprobs and values, added fixed kl loss coefficient and refactored kl controllers

  SalmanMohammadi

  committed
  c289566

  Copy full SHA for c289566

Commits on Jun 14, 2024

• added loss and value masking

  SalmanMohammadi

  committed
  a3fa1ea

  Copy full SHA for a3fa1ea

Commits on Jun 16, 2024

• some refactoring, lots of testing and docs

  SalmanMohammadi

  committed
  c3db142

  Copy full SHA for c3db142

• improved early training stability by adding value head init. from reward model

  SalmanMohammadi

  committed
  589bf7d

  Copy full SHA for 589bf7d

Commits on Jun 18, 2024

• updating metrics

  SalmanMohammadi

  committed
  346c30b

  Copy full SHA for 346c30b

• reworking causal masking

  SalmanMohammadi

  committed
  2e9d779

  Copy full SHA for 2e9d779

• freeing up memory after steps to avoid mem leaks

  SalmanMohammadi

  committed
  46b75be

  Copy full SHA for 46b75be

Commits on Jul 16, 2024

• Merge branch 'main' into ppo

  SalmanMohammadi

  committed
  0fd885e

  Copy full SHA for 0fd885e

• cleaning up; verifying results; switching to full finetune

  SalmanMohammadi

  committed
  1942b0f

  Copy full SHA for 1942b0f

• tidying up

  SalmanMohammadi

  committed
  58d92ab

  Copy full SHA for 58d92ab

Commits on Jul 18, 2024

• detaching losses for metric logging

  SalmanMohammadi

  committed
  1fbb6dc

  Copy full SHA for 1fbb6dc

Commits on Jul 25, 2024

• removing 1b, merging main

  SalmanMohammadi

  committed
  65ef9dc

  Copy full SHA for 65ef9dc

• merging

  SalmanMohammadi

  committed
  c7bbff1

  Copy full SHA for c7bbff1

Commits on Jul 29, 2024

• deleting logits in loss

  SalmanMohammadi

  committed
  1129f9e

  Copy full SHA for 1129f9e

Commits on Aug 2, 2024

• Merge branch 'main' into ppo

  SalmanMohammadi

  authored
  fe87dfb

  Copy full SHA for fe87dfb

• cleaning conf

  SalmanMohammadi

  committed
  662ab2c

  Copy full SHA for 662ab2c

• pYdOcLiNt

  SalmanMohammadi

  committed
  76b124f

  Copy full SHA for 76b124f

Commits on Aug 3, 2024

• downloading weights

  SalmanMohammadi

  committed
  dc4887c

  Copy full SHA for dc4887c

Commits on Aug 5, 2024

• addressing comments

  SalmanMohammadi

  committed
  ef85dba

  Copy full SHA for ef85dba

• updating test

  SalmanMohammadi

  committed
  fd87fe6

  Copy full SHA for fd87fe6

• let's finish this the way we started... together

  SalmanMohammadi

  committed
  ba365a8

  Copy full SHA for ba365a8

• Merge branch 'main' into ppo

  SalmanMohammadi

  authored
  e76304c

  Copy full SHA for e76304c

• lInTiNG

  SalmanMohammadi

  committed
  4e6be43

  Copy full SHA for 4e6be43