Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

override seed types #708

Merged
merged 8 commits into from
Apr 23, 2018
Merged

override seed types #708

merged 8 commits into from
Apr 23, 2018

Conversation

drewbanin
Copy link
Contributor

@drewbanin drewbanin commented Mar 28, 2018

The following config will force the create table statement for the country_codes.csv file to use a text type for the id field.

seeds:
    my_project:
        country_codes:
            column_types:
                id: text
                other_field: numeric(12,4)

I don't love the idea of adding another arbitrary config to the seeds block, but I'm unsure where else we can implement this.

I know some folks also want this functionality for models too, but we don't have a good way of overriding column types for create table as (...) statements. @cmcarthur really keen to hear your thoughts about all of this.

Regarding the code in this PR: I think a lot of it should be ripped out of the adapters. If possible, I'd like to make a seed materialization which encodes this logic, then execute the seed nodes as we would any other node. I don't love the amount of business logic in the SeedRunner class

TODO:

  • Extend tests to also run for BQ, Snowflake

Copy link
Member

@cmcarthur cmcarthur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is fine. I agree that this code should be ripped out of adapters. This PR is sort of confusing to read because of adapter inheritance and because of this being done in slightly different ways on BQ vs PG/Snowflake/Redshift (_agate_to_schema in one place, inline in the other). But I don't want to commit to that type of rewrite right now, so this is approved

@drewbanin
Copy link
Contributor Author

Just added a couple more tests for bq, snowflake. Will merge when green again

@drewbanin drewbanin merged commit 3567e20 into development Apr 23, 2018
@drewbanin drewbanin deleted the feature/override-seed-types branch April 23, 2018 00:03
@drewbanin
Copy link
Contributor Author

Closes: #703, #690

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants