Code

add alpha scaling to lora#8483

Code

Merged

michal2409 merged 29 commits intoadithyare/lora_alphaNVIDIA/NeMo:adithyare/lora_alphafrom mfutrega/lora_alphaNVIDIA/NeMo:mfutrega/lora_alphaCopy head branch name to clipboardFeb 23, 2024

+3,355-725

Commits

Commits on Feb 22, 2024

• coldfix (#8412)

  Show description for 1489ac7
  

  Jorjeous

  authored and

  michal2409

  committed
  1489ac7

• Fixed errors in the CTM gen functions (#8416) (#8420)

  Show description for 5c70924
  

  2 peopleauthored and

  michal2409

  committed
  5c70924

• Add change_vocabulary and save_tokenizers() support to Multitask ASR models (#8357) (#8367)

  Show description for 859b8f2
  

  4 peopleauthored and

  michal2409

  committed
  859b8f2

• fix path location and branch (#8314)

  Show description for b951d64
  

  3 peopleauthored and

  michal2409

  committed
  b951d64

• Add TP comm overlap knobs to AutocastTransformerLayer (#8290)

  Show description for d43d29f
  

  2 peopleauthored and

  michal2409

  committed
  d43d29f

• add deallocate pipeline output optimization (#8279) (#8318)

  Show description for 7ae94a0
  

  4 peopleauthored and

  michal2409

  committed
  7ae94a0

• remove assertion (#8302) (#8321)

  Show description for 344f655
  

  2 peopleauthored and

  michal2409

  committed
  344f655

• Keep max_seqlen and cu_seqlens_argmin for later micro-batches when PP>1 (#8334) (#8346)

  Show description for 9a0c79d
  

  3 peopleauthored and

  michal2409

  committed
  9a0c79d

• Enable megatron core loggers for GPT pretraining (#8354) (#8384)

  Show description for a28e153
  

  5 peopleauthored and

  michal2409

  committed
  a28e153

• Fix dreambooth data sampler issue (#8400) (#8413)

  Show description for e89f8be
  

  3 peopleauthored and

  michal2409

  committed
  e89f8be

• add ensemble decoding fix (#8427) (#8433)

  Show description for 8575c31
  

  2 peopleauthored and

  michal2409

  committed
  8575c31

• NeVA Tutorial Notebook (#8217)

  Show description for d440d7d
  

  2 peopleauthored and

  michal2409

  committed
  d440d7d

• mcore customization doc minor fix (#8421) (#8437)

  Show description for cca9141
  

  2 peopleauthored and

  michal2409

  committed
  cca9141

• Add loop_labels algorithm for TDT greedy decoding (#8215)

  Show description for 7110f68
  

  artbataev

  authored and

  michal2409

  committed
  7110f68

• Add dist ckpt support for regular optimizers (#7749) (#8293)

  Show description for 84c6c5a
  

  6 peopleauthored and

  michal2409

  committed
  84c6c5a

• Multimodal r1.23.0 bug fix (#8315) (#8339)

  Show description for 8d3ddb9
  

  4 peopleauthored and

  michal2409

  committed
  8d3ddb9

• mcore ds fix (#8283) (#8385)

  Show description for 9231b83
  

  8 peopleauthored and

  michal2409

  committed
  9231b83

• MCore dataset compatibility for tokenizers (#8390) (#8397)

  Show description for 1e119a9
  

  4 peopleauthored and

  michal2409

  committed
  1e119a9

• Canary: inference tokenization improvements; preserving custom keys when creating tarred manifests (#8432)

  Show description for ef17d91
  

  pzelasko

  authored and

  michal2409

  committed
  ef17d91

• add sbert to IR (#8445)

  Show description for 1e1cf63
  

  3 peopleauthored and

  michal2409

  committed
  1e1cf63

• Update readme (#8440)

  Show description for 150784c
  

  2 peopleauthored and

  michal2409

  committed
  150784c

• NeMo-Mistral to HF converter bugfix. (#8353) (#8442)

  Show description for 5406d91
  

  2 peopleauthored and

  michal2409

  committed
  5406d91

• Fixing mcore bert for TP, PP and SP (#8336) (#8443)

  Show description for b810c8c
  

  4 peopleauthored and

  michal2409

  committed
  b810c8c

• Add LoRA support to all linear layers (#7988)

  Show description for 76b9eea
  

  6 peopleauthored and

  michal2409

  committed
  76b9eea

• Add Neva Template for NV-DPO Models (#8358)

  Show description for 223456a
  

  HuiyingLi

  authored and

  michal2409

  committed
  223456a

• Rebase scaling alpha

  Show description for 25e02f4
  michal2409

  committed
  25e02f4

• default for alpha

  Show description for 6d3ad7b
  

  arendu

  authored and

  michal2409

  committed
  6d3ad7b

• Rebase scaling alpha

  Show description for 1ec1892
  

  pre-commit-ci[bot]

  authored and

  michal2409

  committed
  1ec1892

Commits on Feb 23, 2024

• Merge branch 'adithyare/lora_alpha' into mfutrega/lora_alpha

  Show description for 1912933
  michal2409

  authored
  1912933