Code

Switch to new data passing API#109

Code

Merged

nicklan merged 113 commits intodelta-io:maindelta-io/delta-kernel-rs:mainfrom nicklan:new-data-passing-apinicklan/delta-kernel-rs-old:new-data-passing-apiCopy head branch name to clipboardMar 4, 2024

+2,262-1,345

Commits

Commits on Jan 24, 2024

• rename table client -> engine client

  Show description for 6c9efe1
  nicklan

  committed
  6c9efe1

• add engine_data and trait method

  Show description for ec08901
  nicklan

  committed
  ec08901

Commits on Jan 25, 2024

• checkpoint, working parsing metadata/protocol

  nicklan

  committed
  f6c1072

Commits on Jan 26, 2024

• checkpoint, can parse metadata in snapshot

  nicklan

  committed
  0643945

Commits on Jan 27, 2024

• make scan tests pass (with some todos)

  nicklan

  committed
  53c8850

• All internal tests pass

  nicklan

  committed
  32a28a9

• clean up parsing a bit

  nicklan

  committed
  d7930e6

• remove old comment

  nicklan

  committed
  0e54c24

• add todo comment

  nicklan

  committed
  438a1e2

Commits on Jan 29, 2024

• New better way to do list/map

  nicklan

  committed
  5ab0021

• put back old test

  nicklan

  committed
  ab89728

Commits on Jan 30, 2024

• add dv support

  nicklan

  committed
  e182e3d

• start to think about DVs

  nicklan

  committed
  6b3efe3

Commits on Jan 31, 2024

• make dv.rs tests work

  nicklan

  committed
  fd80a97

• fmt and small fix

  nicklan

  committed
  4bf5b47

• add some tests back

  nicklan

  committed
  304da21

• fix "list_from"

  nicklan

  committed
  a7393fe

• remove todo

  nicklan

  committed
  54a0c1a

Commits on Feb 1, 2024

• allow default client to use EngineData

  nicklan

  committed
  54a0d7f

• use some unsafe magic to make more tests pass

  nicklan

  committed
  a8e8920

• fmt

  nicklan

  committed
  33923bb

• some clippy guided cleanup

  nicklan

  committed
  82cff1b

Commits on Feb 3, 2024

• switch to selection vec

  nicklan

  committed
  dc6e963

Commits on Feb 5, 2024

• make parse_json not refrence record batch anymore

  nicklan

  committed
  1cff896

• almost have arrow out of lib.rs

  nicklan

  committed
  e6d5eb5

Commits on Feb 6, 2024

• add back md test

  nicklan

  committed
  28031fa

• put back dataskippingfilter (but it still uses arrow)

  nicklan

  committed
  4294020

• make comment more clear

  nicklan

  committed
  ef0e79c

• doc updates

  nicklan

  committed
  b6b560c

• add back parquet reader test

  nicklan

  committed
  ae12b63

• remove commented code

  nicklan

  committed
  f6cdd00

• put back default client json tests

  nicklan

  committed
  f4c8507

• remove unsafe (woo!)

  nicklan

  committed
  4d3631a

• minor cleanup

  nicklan

  committed
  95c75ce

• rename to try_from_engine_data

  nicklan

  committed
  3902fcb

Commits on Feb 7, 2024

• all arrow out of lib :)

  nicklan

  committed
  200a8ac

• move simple_client to a feature, make it default

  nicklan

  committed
  3ac637f

• add materialize for map_item

  nicklan

  committed
  49253b7

• mostly remove old action parsing code and make maps work right

  nicklan

  committed
  9a286e6

• fmt + clippy

  nicklan

  committed
  8b0a60f

Commits on Feb 8, 2024

• test_read_files

  nicklan

  committed
  15f0122

Commits on Feb 12, 2024

• Apply suggestions from code review

  Show description for bdf540b
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  bdf540b

Commits on Feb 14, 2024

• address some minor comments

  nicklan

  committed
  5905ed2

• switch to add_string

  nicklan

  committed
  1f83540

• switch to using as_list

  nicklan

  committed
  5f3e6d4

• address more comments:

  Show description for 9aa7d36
  nicklan

  committed
  9aa7d36

• cleaner min_file_name

  Show description for 2fb11f5
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  2fb11f5

• clean up list_from a bit + fmt

  nicklan

  committed
  9834460

• Update kernel/src/simple_client/json.rs

  Show description for cff7b98
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  cff7b98

• Apply suggestions from code review

  Show description for eb0f3c6
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  eb0f3c6

• fixups for review merge

  nicklan

  committed
  7f8ab66

• address comment re itereator chain

  nicklan

  committed
  01bcd9a

• fix bug

  nicklan

  committed
  e5f0116

• Apply suggestions from code review

  Show description for 93e346b
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  93e346b

• validate magic in DV

  nicklan

  committed
  d1f5531

• get_json_filename

  nicklan

  committed
  7198c90

• only default-client needs tokio

  nicklan

  committed
  ab64aec

• rename res_arry -> res_array

  nicklan

  committed
  1194be2

• Update kernel/src/simple_client/parquet.rs

  Show description for c288f2e
  

  nicklan

  and

  ryan-johnson-databricks

  authored
  c288f2e

• fix lints

  nicklan

  committed
  ef33863

• address comments

  nicklan

  committed
  5b47c93

Commits on Feb 15, 2024

• Initial bit of extract_into

  nicklan

  committed
  23b7196

• doc comments and fmt

  nicklan

  committed
  65a21cf

• fully switch to extract_into

  nicklan

  committed
  ff85d93

• fmt

  nicklan

  committed
  5fea8d3

Commits on Feb 16, 2024

• initial work on row-getter data passing

  nicklan

  committed
  423853e

• add get_data_item.rs

  nicklan

  committed
  e2913ff

• cleanup extract, support all previous types, better errors

  nicklan

  committed
  b30bb2c

• extract references to Maps/Lists

  nicklan

  committed
  baf1997

• cleanup extract

  nicklan

  committed
  f96314c

Commits on Feb 17, 2024

• switch to new data passing style

  nicklan

  committed
  89f920d

• reformat a bit

  nicklan

  committed
  46b7e1f

• only getters need to implement ExtractInto

  nicklan

  committed
  3536132

• make GetDataItem macro for impls

  nicklan

  committed
  190c05c

Commits on Feb 20, 2024

• remove trivial casts warning, address minor comments

  nicklan

  committed
  1a6b444

Commits on Feb 21, 2024

• get rid of DataItem

  nicklan

  committed
  0dbb5ed

• break once we've found p&m

  nicklan

  committed
  582e8be

• extract error handling

  nicklan

  committed
  ab88f14

• add doc comment

  nicklan

  committed
  4458c9a

• fold Extractor into EngineData trait

  nicklan

  committed
  ca37359

• add materialize for list

  nicklan

  committed
  a6ed21e

• fmt

  nicklan

  committed
  cee7d16

• make magic constant a const

  nicklan

  committed
  7ff8589

• use try_collect

  nicklan

  committed
  72dcb54

• remove commented code

  nicklan

  committed
  b9601a7

• comment updates

  nicklan

  committed
  8d78cf7

• error improvements

  nicklan

  committed
  67fe622

• fix comment

  nicklan

  committed
  3fab759

• better macro format

  nicklan

  committed
  ace56fa

• impl TypedGetData for Vec and Map

  nicklan

  committed
  d7afb3b

• return Options for try_new_from_data

  nicklan

  committed
  db8d28d

• created_time is optional

  nicklan

  committed
  2d57456

• stats are String

  nicklan

  committed
  22d85ca

• doc comment and fmt

  nicklan

  committed
  3f13937

• refactor action defs and parsing. remove ActionType enum

  nicklan

  committed
  619bffd

• add todo

  nicklan

  committed
  a72eac6

• add missing files

  nicklan

  committed
  977e251

• add comment

  nicklan

  committed
  6750a2a

Commits on Feb 22, 2024

• GIANT Merge branch 'main' into new-data-passing-api

  Show description for bd9e17b
  nicklan

  committed
  bd9e17b

Commits on Feb 23, 2024

• checkpoint, actually compiles. needs expressions to be put back

  nicklan

  committed
  a2dedf5

• woo, all tests working.

  Show description for f5d80c3
  nicklan

  committed
  f5d80c3

• add is_empty to make clippy happy

  nicklan

  committed
  0a9cd11

• final clippy fixes

  nicklan

  committed
  b268b4e

• import style

  nicklan

  committed
  aab7fb8

• put back test that merge inexplicably removed

  nicklan

  committed
  8d397e6

• use error constructors where strings are static

  nicklan

  committed
  a2cb373

• missed one

  nicklan

  committed
  1a19d19

• impl From for SimpleData -> RecordBatch

  nicklan

  committed
  c130692

• bunch of doc fixes

  nicklan

  committed
  fdd927a

Commits on Feb 26, 2024

• small comment changes

  nicklan

  committed
  b4699c0

• fmt

  nicklan

  committed
  1c922c9

Commits on Mar 4, 2024

• Update kernel/src/snapshot.rs

  Show description for 4a14694
  

  nicklan

  and

  zachschuermann

  authored
  4a14694

• fix typo

  nicklan

  committed
  14d0aff