Code

Add support for GPTNeoX models#32

Code

Merged

yukang2017 merged 17 commits intodvlab-research:maindvlab-research/LongLoRA:mainfrom naubull2:mainnaubull2/LongLoRA:mainCopy head branch name to clipboardOct 3, 2023

+202-9

Commits

Commits on Sep 27, 2023

• [add] gpt-neox support

  naubull2

  committed
  7baa4f7

• [update] readme

  naubull2

  committed
  41977df

Commits on Sep 28, 2023

• [fix] some of the bugs preventing fine-tune run

  Show description for 9c9d0a2
  naubull2

  committed
  9c9d0a2

• [fix] dimesion discrepancy between attention mask and the query length

  Show description for a5111ef
  naubull2

  committed
  a5111ef

• [fix] SFT to match the same mods in finetune.py

  naubull2

  committed
  5862050

• Merge branch 'forked-only'

  naubull2

  committed
  0cf0dfd

Commits on Sep 30, 2023

• [add] parallel group attention then reshape back to original form

  naubull2

  committed
  1532c4b

• [fix] non-contiguous dimensions changing view issue

  naubull2

  committed
  6fdffbb

• [add] attention mask to align with the grouped batching

  naubull2

  committed
  fe97f86

Commits on Oct 2, 2023

• [add] torch autocast for flash attention safety

  Show description for 9e30a15
  naubull2

  committed
  9e30a15

• [fix] HF built-in rotary embedding is not compatible with flash-attention

  Show description for 3f9c47c
  naubull2

  committed
  3f9c47c

• [add] missing local reference for rotate_half

  naubull2

  committed
  b21e949

• [rollback] torch.cuda autocast causes half precision error

  Show description for b224273
  naubull2

  committed
  b224273

Commits on Oct 3, 2023

• [fix] flash attention causing in-place operation runtime errors

  naubull2

  committed
  9123e42

• [fix] mixed use of tabs and spaces

  naubull2

  committed
  7203de2

• [change] readme back to where it came from the original repo

  naubull2

  committed
  8a11ef8

• [remove] unused comments

  naubull2

  committed
  02e4c1c