-
Notifications
You must be signed in to change notification settings - Fork 84
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kedro-datasets: ExternalTableDataset for Databricks #817
Comments
Hey @astrojuanlu and the Kedro team, |
Hi @MinuraPunchihewa , if you want to open a pull request please go ahead! |
Thanks, @astrojuanlu. Will do. |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I use Kedro on Databricks heavily, but in most of my projects the datasets in use are External tables (for a myriad of reasons) and I am currently unable to perform my IO operations easily.
Context
As I have said above, my datasets are maintained as External tables and I am certain there are many others users (and organizations) out there that follow a similar pattern, at least for some of their datasets.
Possible Implementation
It should be possible to implement at
ExternalTableDataset
for Kedro to allow the said IO operations.Possible Alternatives
I have considered converting all of my datasets on Databricks to Managed tables, but at the moment, this does not make sense for me from an architectural point of view.
The text was updated successfully, but these errors were encountered: