Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to CSVImports for Redshift upload #218

Open
patr1ckm opened this issue Oct 23, 2019 · 0 comments
Open

Switch to CSVImports for Redshift upload #218

patr1ckm opened this issue Oct 23, 2019 · 0 comments
Milestone

Comments

@patr1ckm
Copy link
Contributor

patr1ckm commented Oct 23, 2019

In general:

  1. Add a utility get_headers.
  2. Switch from imports_post to imports_post_files_csv in write_civis.character and write_civis.numeric.
  3. Before uploading, files need to be cleaned withfiles_post_process_csv. Note in the reference PR the steps that are needed to process the cleaning results.
  4. The client will need to be regenerated to access the new endpoints.
  5. Support multiple file ids in write_civis.numeric. If multiple files are uploaded, check that the schema, headers, delimiter, and compression are consistent across files. Upload the files in parallel.
  6. Document the upsert option to if_exists.

See civisanalytics/civis-python#328 for all details.

@patr1ckm patr1ckm changed the title Switch to CSVImports Switch to CSVImports for Redshift upload Oct 23, 2019
@patr1ckm patr1ckm added this to the 2.2 milestone Oct 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant