Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Template for Data Pipelines #253

Closed
caldeirav opened this issue Jan 22, 2023 · 3 comments
Closed

Template for Data Pipelines #253

caldeirav opened this issue Jan 22, 2023 · 3 comments
Assignees
@caldeirav

Comments

@caldeirav
Copy link
Contributor

No description provided.

@caldeirav caldeirav self-assigned this Jan 22, 2023
@caldeirav
Copy link
Contributor Author

Creation of a clean base template for developers creating a new data pipeline, under https://github.com/os-climate/data-pipeline-template. The template should have the base structure for data, metadata and quality tests ingestion (via DBT and great_expectations integration with OpenMetadata).

@MichaelTiemannOSC
Copy link
Contributor

MichaelTiemannOSC commented Jan 22, 2023 •
edited
Loading

Quick note that the python version should be at least 3.9, not 3.8 (as stated in Pipfile). There are several other minimum versions we should stipulate. See #234. Of course this requires wiring up JupyterHub environments that support Python 3.9 as a base, not 3.8.

@caldeirav
Copy link
Contributor Author

Template has been completed for extraction / load / transform. Remaining to be tested is integration with OpenMetadata which is now failing due to integration with Airflow Managed APIs. Will proceed to close this issue and open new issues for additional work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caldeirav caldeirav

Labels
None yet
Projects
Data Commons Platform
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@caldeirav @MichaelTiemannOSC