This repository has been archived by the owner on Sep 6, 2023. It is now read-only.

Store in parquet format#20

Merged

DuttaSoumya merged 8 commits intomainfrom StoreInParquetFormat

May 31, 2022

+1,017-621

Commits on May 24, 2022

First draft
Soumya Dutta
committed
Confirmed dataflow works
Soumya Dutta
committed
Corrected the pull request ID
Soumya Dutta
committed

Commits on May 27, 2022

Synapse can now handle both CSV as well as Parquet content, depending on the configuration specified at data manifest. By default, the flow creates the parquet staging. If the entity is required to…
Soumya Dutta
committed
Change flow to conditionally split data into both the CSV as well as parquet folders together (better performance), copy the staged files using the right filter. Remove the staging folder when done.
Soumya Dutta
committed
Change flow to conditionally split data into both the CSV as well as parquet folders together (better performance), copy the staged files using the right filter. Remove the staging folder when done.
Soumya Dutta
committed

Commits on May 30, 2022

Code review comments from Henri incorporated.
Soumya Dutta
committed

Commits on May 31, 2022

No defaulting of the container name is actually done.
Soumya Dutta
committed