Code

Validation#866

Code

Merged

XiaohanZhangCMU merged 72 commits intomosaicml:byod/data_validationmosaicml/llm-foundry:byod/data_validationfrom XiaohanZhangCMU:validationXiaohanZhangCMU/llm-foundryX:validationCopy head branch name to clipboardJan 12, 2024

+94-1,271

Commits

Commits on Dec 23, 2023

• add validation script

  xiaohanzhan-db

  committed
  8cb6522

  Copy full SHA for 8cb6522

Commits on Jan 3, 2024

• update

  xiaohanzhan-db

  committed
  c59c11f

  Copy full SHA for c59c11f

• change token count function

  

  xiaohanzhan-db

  committed
  66f34eb

  Copy full SHA for 66f34eb

Commits on Jan 5, 2024

• reorganize cells

  

  xiaohanzhan-db

  committed
  2cd387b

  Copy full SHA for 2cd387b

• Add unit tests

  xiaohanzhan-db

  committed
  3eac3bf

  Copy full SHA for 3eac3bf

Commits on Jan 6, 2024

• Add a printout for CPT

  xiaohanzhan-db

  committed
  d2d9767

  Copy full SHA for d2d9767

• update question

  xiaohanzhan-db

  committed
  be25591

  Copy full SHA for be25591

Commits on Jan 8, 2024

• Add questions

  

  xiaohanzhan-db

  committed
  4651be7

  Copy full SHA for 4651be7

• Fix lints

  xiaohanzhan-db

  committed
  5cd6a94

  Copy full SHA for 5cd6a94

• Merge branch 'main' into validation

  XiaohanZhangCMU

  authored
  8e2c1f4

  Copy full SHA for 8e2c1f4

• update format

  xiaohanzhan-db

  committed
  e6e4a81

  Copy full SHA for e6e4a81

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  34c5690

  Copy full SHA for 34c5690

• update

  xiaohanzhan-db

  committed
  1668b9a

  Copy full SHA for 1668b9a

• nb source

  xiaohanzhan-db

  committed
  2219135

  Copy full SHA for 2219135

• add validation script

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  86c6e87

  Copy full SHA for 86c6e87

• update

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  678b376

  Copy full SHA for 678b376

• change token count function

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  297e057

  Copy full SHA for 297e057

• reorganize cells

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  09d0ebb

  Copy full SHA for 09d0ebb

• Add unit tests

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  460df65

  Copy full SHA for 460df65

• Add a printout for CPT

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  3ffd200

  Copy full SHA for 3ffd200

• update question

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  9362886

  Copy full SHA for 9362886

• Add questions

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  898e5ac

  Copy full SHA for 898e5ac

• Fix lints

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  a4bef71

  Copy full SHA for a4bef71

• update format

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  4ca9cc6

  Copy full SHA for 4ca9cc6

• update

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  d636a0f

  Copy full SHA for d636a0f

• nb source

  

  xiaohanzhan-db

  authored and

  XiaohanZhangCMU

  committed
  827d155

  Copy full SHA for 827d155

• Remove license insert for validation notebook

  xiaohanzhan-db

  committed
  6bbf3fc

  Copy full SHA for 6bbf3fc

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  4f6a4fb

  Copy full SHA for 4f6a4fb

Commits on Jan 11, 2024

• Add validation utils

  xiaohanzhan-db

  committed
  5966b68

  Copy full SHA for 5966b68

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  da17813

  Copy full SHA for da17813

• Minor cleanups (mosaicml#858)

  Show description for a7c36bc
  mvpatel2000

  authored
  a7c36bc

  Copy full SHA for a7c36bc

• update utils/__init__.py to include extra validation functions

  xiaohanzhan-db

  committed
  55e4626

  Copy full SHA for 55e4626

• update notebook

  

  xiaohanzhan-db

  committed
  45544a1

  Copy full SHA for 45544a1

• update

  xiaohanzhan-db

  committed
  d2797b3

  Copy full SHA for d2797b3

• Merge branch 'validation' of github.com:XiaohanZhangCMU/llm-foundryX into validation

  xiaohanzhan-db

  committed
  019da77

  Copy full SHA for 019da77

• update

  xiaohanzhan-db

  committed
  756fdae

  Copy full SHA for 756fdae

• Read UC delta table (mosaicml#773)

  Show description for 6de8c37
  

  XiaohanZhangCMU

  and

  xiaohanzhan-db

  authored
  6de8c37

  Copy full SHA for 6de8c37

• Add download remote function to util

  xiaohanzhan-db

  committed
  93b5a9f

  Copy full SHA for 93b5a9f

• update

  xiaohanzhan-db

  committed
  b47c878

  Copy full SHA for b47c878

• remove fused layernorm (mosaicml#859)

  mvpatel2000

  authored
  fa8f3d9

  Copy full SHA for fa8f3d9

• update

  xiaohanzhan-db

  committed
  13fd34c

  Copy full SHA for 13fd34c

• update

  xiaohanzhan-db

  committed
  610f669

  Copy full SHA for 610f669

• update

  xiaohanzhan-db

  committed
  9f2e51b

  Copy full SHA for 9f2e51b

• update

  xiaohanzhan-db

  committed
  ec68f10

  Copy full SHA for ec68f10

• update

  xiaohanzhan-db

  committed
  1e76068

  Copy full SHA for 1e76068

• update

  xiaohanzhan-db

  committed
  7a5c164

  Copy full SHA for 7a5c164

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  e76038f

  Copy full SHA for e76038f

• update

  xiaohanzhan-db

  committed
  5b413f5

  Copy full SHA for 5b413f5

• update

  xiaohanzhan-db

  committed
  a1aa31f

  Copy full SHA for a1aa31f

• update

  xiaohanzhan-db

  committed
  d24fd5c

  Copy full SHA for d24fd5c

Commits on Jan 12, 2024

• Remove hardcoded combined.jsonl with a flag (mosaicml#861)

  Show description for da3bea1
  

  XiaohanZhangCMU

  and

  xiaohanzhan-db

  authored
  da3bea1

  Copy full SHA for da3bea1

• bump (mosaicml#828)

  mvpatel2000

  authored
  936e3a1

  Copy full SHA for 936e3a1

• Add dask and dataframe_to_mds

  xiaohanzhan-db

  committed
  55fce37

  Copy full SHA for 55fce37

• update

  xiaohanzhan-db

  committed
  86e2412

  Copy full SHA for 86e2412

• update

  xiaohanzhan-db

  committed
  bbfec65

  Copy full SHA for bbfec65

• update

  xiaohanzhan-db

  committed
  b2e880d

  Copy full SHA for b2e880d

• update

  xiaohanzhan-db

  committed
  596443a

  Copy full SHA for 596443a

• Add notebook

  xiaohanzhan-db

  committed
  ea65187

  Copy full SHA for ea65187

• update

  xiaohanzhan-db

  committed
  378a4e0

  Copy full SHA for 378a4e0

• update

  

  xiaohanzhan-db

  committed
  af6e9aa

  Copy full SHA for af6e9aa

• remove script and tests, keep notebook

  xiaohanzhan-db

  committed
  4e286ec

  Copy full SHA for 4e286ec

• update

  xiaohanzhan-db

  committed
  09c4892

  Copy full SHA for 09c4892

• update

  xiaohanzhan-db

  committed
  c82da6c

  Copy full SHA for c82da6c

• update

  xiaohanzhan-db

  committed
  e5f83cc

  Copy full SHA for e5f83cc

• update

  xiaohanzhan-db

  committed
  17d2b9f

  Copy full SHA for 17d2b9f

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  6579d55

  Copy full SHA for 6579d55

• Merge branch 'byod/data_validation' into validation

  XiaohanZhangCMU

  authored
  56308ff

  Copy full SHA for 56308ff

• Always initialize dist (mosaicml#864)

  Show description for 6517a30
  mvpatel2000

  authored
  6517a30

  Copy full SHA for 6517a30

• updated notebook

  

  xiaohanzhan-db

  committed
  4daa324

  Copy full SHA for 4daa324

• Merge branch 'main' into validation

  xiaohanzhan-db

  committed
  b809691

  Copy full SHA for b809691

• remove scripts keep notebook

  xiaohanzhan-db

  committed
  8b75f94

  Copy full SHA for 8b75f94

• merge with byod/data_validation

  xiaohanzhan-db

  committed
  99bf2cd

  Copy full SHA for 99bf2cd