Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

do something else with CSV loading--it sucks #74

Closed
drewbanin opened this issue Jul 26, 2016 · 2 comments
Closed

do something else with CSV loading--it sucks #74

drewbanin opened this issue Jul 26, 2016 · 2 comments
Milestone

Comments

@drewbanin
Copy link
Contributor

drewbanin commented Jul 26, 2016

option 1: maybe an ENV function like load_csv_file(...)?

option 2: force the user to write the model as follows (this is terrible pseudocode):

{ for item in dict(whatever)
  return (build a concatenated select statement with a UNION ALL at the end)
}

option 3: other stuff?

option 4: kill the feature?

right now we have a fairly strong assumption that dbt does not load data into your warehouse. this feature in its current form defies that. I'd really prefer to re-implement it in a more "dbt-y" way. option 2 feels the most appropriate to me, although forcing the user to write this code is terrible. if we built a wrapper function that essentially did exactly this, then turned CSVs into an appropriate way to load configuration data, then we're getting to a place that feels more like a good solution to this, however. it would allow the user to materialize this model however they chose to.

@jthandy jthandy changed the title more comprehensive templating do something else with CSV loading--it sucks Sep 16, 2016
@jthandy
Copy link
Member

jthandy commented Sep 16, 2016

@drewbanin did a bit more thinking here. would appreciate your thoughts on the "option 2" I presented above and my commentary on why I liked it.

@drewbanin drewbanin modified the milestone: CSVs Oct 18, 2016
@drewbanin drewbanin modified the milestone: CSVs Jun 20, 2017
@drewbanin drewbanin modified the milestones: CSVs, 0.9.2 Jan 12, 2018
@drewbanin
Copy link
Contributor Author

dbt seed is in a much better place now than it once one. In the future, we should consider other configuration options for csv files like custom schemas, sort/dist keys on Redshift, etc.

We can create new features for those issues when they come up! For now though, closing: #618

yu-iskw pushed a commit to yu-iskw/dbt that referenced this issue Aug 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants