Code

Avro Table Provider#910

Code

Merged

alamb merged 56 commits intoapache:masterapache/datafusion:masterfrom Igosuki:avro2_mIgosuki/arrow-datafusion:avro2_mCopy head branch name to clipboardSep 15, 2021

+3,258-38

Commits

Commits on Sep 14, 2021

• Add avro as a datasource, file and table provider

  Igosuki

  committed
  7e14f91

  Copy full SHA for 7e14f91

• wip

  Igosuki

  committed
  0f940ab

  Copy full SHA for 0f940ab

• Added support composite identifiers for struct type.

  

  jorgecarleitao

  authored and

  Igosuki

  committed
  2ca7ccb

  Copy full SHA for 2ca7ccb

• Fixed build.

  

  jorgecarleitao

  authored and

  Igosuki

  committed
  fcbf43b

  Copy full SHA for fcbf43b

• cheat and add unions to valid composite column types

  Igosuki

  committed
  bae2f00

  Copy full SHA for bae2f00

• Implement the AvroArrayReader

  Igosuki

  committed
  a352dad

  Copy full SHA for a352dad

• Add binary types

  Igosuki

  committed
  d42cdd1

  Copy full SHA for d42cdd1

• Enable Avro as a FileType

  Igosuki

  committed
  cbdc9c7

  Copy full SHA for cbdc9c7

• Enable registering an avro table in the sql parsing

  Igosuki

  committed
  7d6078e

  Copy full SHA for 7d6078e

• Change package name for datafusion/avro

  Igosuki

  committed
  44cfa87

  Copy full SHA for 44cfa87

• Implement Avro datasource tests and fix avro_rs::Value resolution to Arrow types

  Igosuki

  committed
  6ef92d8

  Copy full SHA for 6ef92d8

• Test for AvroExec::try_from_path

  Igosuki

  committed
  01cbb18

  Copy full SHA for 01cbb18

• external table avro test

  Igosuki

  committed
  537577a

  Copy full SHA for 537577a

• Basic schema conversion tests

  Igosuki

  committed
  75566d1

  Copy full SHA for 75566d1

• Complete test for avro_to_arrow_reader on alltypes_dictionnary

  Igosuki

  committed
  7fdf233

  Copy full SHA for 7fdf233

• fix_stable: .rewind is 'unstable'

  Igosuki

  committed
  697e3a9

  Copy full SHA for 697e3a9

• Fix license files and remove the unused avro-converter crate

  Igosuki

  committed
  e1d6df8

  Copy full SHA for e1d6df8

• fix example test in avro_to_arrow

  Igosuki

  committed
  66a5901

  Copy full SHA for 66a5901

• add avro_sql test to default workflow

  Igosuki

  committed
  9ea942c

  Copy full SHA for 9ea942c

• Adress clippies

  Igosuki

  committed
  84ee28a

  Copy full SHA for 84ee28a

• Enable avro as a valid datasource for client execution

  Igosuki

  committed
  9edac57

  Copy full SHA for 9edac57

• Add avro to available logical plan nodes

  Igosuki

  committed
  e8a6206

  Copy full SHA for e8a6206

• Add ToTimestampMillis as a scalar function in protos

  Igosuki

  committed
  45eac7c

  Copy full SHA for 45eac7c

• Allow Avro in PhysicalPlan nodes

  Igosuki

  committed
  b4340ac

  Copy full SHA for b4340ac

• Remove remaining confusing references to 'json' in avro mod

  Igosuki

  committed
  408f759

  Copy full SHA for 408f759

• rename 'parquet' words in avro test and examples

  Igosuki

  committed
  f34b995

  Copy full SHA for f34b995

• Handle Union of nested lists in arrow reader

  Igosuki

  committed
  6ce0904

  Copy full SHA for 6ce0904

• test timestamp arrays

  Igosuki

  committed
  e1e40ef

  Copy full SHA for e1e40ef

• remove debug statement

  Igosuki

  committed
  1c79ffb

  Copy full SHA for 1c79ffb

• Make avro optional

  Igosuki

  committed
  c96b781

  Copy full SHA for c96b781

• Remove debug statement

  Igosuki

  committed
  060c644

  Copy full SHA for 060c644

• Remove GetField usage (see apache#628)

  Igosuki

  committed
  72ad35c

  Copy full SHA for 72ad35c

• Fix docstring in parser tests

  Igosuki

  committed
  3c8f6ce

  Copy full SHA for 3c8f6ce

• Test batch output rather than just rows individually

  Igosuki

  committed
  0db3013

  Copy full SHA for 0db3013

• Remove 'csv' from error strings in physical_plan::avro

  Igosuki

  committed
  faa2152

  Copy full SHA for faa2152

• Avro sample sql and explain queries tests in sql.rs

  Igosuki

  committed
  a6cd7f1

  Copy full SHA for a6cd7f1

• Activate avro feature for cargo tests in github workflow

  Igosuki

  committed
  8b4fcae

  Copy full SHA for 8b4fcae

• Add a test for avro registering multiple files in a single table

  Igosuki

  committed
  b2b7915

  Copy full SHA for b2b7915

• Switch to Result instead of Option for resolve_string

  Igosuki

  committed
  8acf062

  Copy full SHA for 8acf062

• Address missing clippy warning should_implement_trait in arrow_to_avro/reader

  Igosuki

  committed
  65c9b9e

  Copy full SHA for 65c9b9e

• Add fmt display implementation for AvroExec

  Igosuki

  committed
  1668b96

  Copy full SHA for 1668b96

• ci: fix cargo sql run example, use datafusion/avro feature instead of 'avro'

  Igosuki

  committed
  c9218e2

  Copy full SHA for c9218e2

• license: missing license file for avro_to_arrow/schema.rs

  Igosuki

  committed
  be09a64

  Copy full SHA for be09a64

• only run avro datasource tests if features have 'avro'

  Igosuki

  committed
  68df268

  Copy full SHA for 68df268

• refactor: rename infer_avro_schema_from_reader to read_avro_schema_from_reader

  Igosuki

  committed
  09c4ecb

  Copy full SHA for 09c4ecb

• Pass None as props to avro schema schema_to_field_with_props until further notice

  Igosuki

  committed
  94b9dcb

  Copy full SHA for 94b9dcb

• Change schema inferance to FixedSizeBinary(16) for Uuid

  Igosuki

  committed
  a03a3c2

  Copy full SHA for a03a3c2

• schema: prefix metadata coming from avro with 'avro'

  Igosuki

  committed
  405c63a

  Copy full SHA for 405c63a

• make num traits optional and part of the avro feature flag

  Igosuki

  committed
  291637a

  Copy full SHA for 291637a

• Fix avro schema tests regarding external props

  Igosuki

  committed
  00f109c

  Copy full SHA for 00f109c

• split avro physical plan test feature wise and add a non-implemented test

  Igosuki

  committed
  ea31e02

  Copy full SHA for ea31e02

• submodule: switch back to apache/arrow-testing

  Igosuki

  committed
  a264858

  Copy full SHA for a264858

• fix_test: columns are now prefixed in the plan

  Igosuki

  committed
  c62d931

  Copy full SHA for c62d931

• avro_test: fix clippy warning cmp-owned

  Igosuki

  committed
  aa45189

  Copy full SHA for aa45189

• avro: move statistics to the physical plan

  Igosuki

  committed
  0b746db

  Copy full SHA for 0b746db

Commits on Sep 15, 2021

• Increase min stack size for cargo tests

  Igosuki

  committed
  767c469

  Copy full SHA for 767c469