Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Richer export types (parquet, avro) #412

Open
djoanes opened this issue Feb 10, 2023 · 1 comment
Open

Richer export types (parquet, avro) #412

djoanes opened this issue Feb 10, 2023 · 1 comment
Labels
core A change which affects the synth core enhancement New feature or request help wanted Extra attention is needed integration A change which affects or introduces an integration

Comments

@djoanes
Copy link

djoanes commented Feb 10, 2023

Required Functionality
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
With the current functionality of synth, the output artifacts are all typeless. This causes there to be a necessity for auto type derivation downstream or an explicit redeclaration of the types of all the columns.

Proposed Solution
Add support for new uris, parquet:, avro:. This will allow for richer importing and exporting where the explicitly defined types are preserved.

Use case
I'd like to use synth to generate data and bulk load it into a big data ecosystem (ie. Hadoop, BigQuery)

@iamwacko
Copy link
Collaborator

Avro integration is possible, avrow should work well. Parquet integration would take a lot more work, as it doesn't seem to have a decent serde implantation.

@iamwacko iamwacko added enhancement New feature or request help wanted Extra attention is needed core A change which affects the synth core integration A change which affects or introduces an integration labels Feb 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core A change which affects the synth core enhancement New feature or request help wanted Extra attention is needed integration A change which affects or introduces an integration
Projects
None yet
Development

No branches or pull requests

2 participants