support for duckdb

[geoHeil]

https://openmetadata.slack.com/archives/C02B6955S4S/p1697099341468359

OpenMetadata/ingestion/src/metadata/ingestion/lineage/models.py

Line 75 in f63881b

DUCKDB = "duckdb"

is listing duckdb

DBT supports duckdb

but for OM to ingest dbts nodes somehow the duckdb tables would need to be loaded beforehand

[pmbrull]

duckdb is a supported dialect for the lineage engine, but we do not have a connector yet.

It should be good contribution from our community by watching the tutorial or reviewing similar PRs

Thanks!

[saurabhyadav1985]

I see this issue is still open. I am interested in this project and contributing to it. Please assign a good first issue to me to work on. Thank you!

[pmbrull]

hi @saurabhyadav1985, assigned, thanks

[Supan90-Shah3006]

Hey @saurabhyadav1985 , You Forgot To Add DuckDB.md File In openmetadata-ui/src/main/resources/ui/public/locales/en-US/Database . Please Check It.

[pmbrull]

hi @saurabhyadav1985 I see that the duckdb connection is a copy of the greenplum connection https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/entity/services/connections/database/duckdbConnection.json

What properties do we actually need to connect there?

Actually, looks like almost all the greenplum code has been copy-pasted into DuckDB. Not sure that's the best approach here, since DuckDB might have its own types, connection specifications etc. This might need another iteration to get the right information.

I think it's worth it to revert the change and put some time to review the requirements of the connector, rather than shipping it as-is and having troubles ingesting and migrating data after this gets updated

[geoHeil]

@pmbrull what is the state of this? What exactly is missing? do you have some clear instructions? Maybe I find time to contribute

[pmbrull]

@geoHeil the past contribution did not solve the actual problem at hand, so it was reverted to avoid any confusion.

If you'd like to contribute, you can follow this guide https://docs.open-metadata.org/v1.3.x/developers/contribute/developing-a-new-connector

Thanks

[geoHeil]

This is really generic - if we want to reuse a DB connector let`s say postgres as a template for duckdb - can we speed up the process? I.e. is it enough to perhaps create the ingest connector but for the data model keep whatever postgres is offering (as that should be the same on the OM server side)

[pmbrull]

This is really generic - if we want to reuse a DB connector let`s say postgres as a template for duckdb - can we speed up the process? I.e. is it enough to perhaps create the ingest connector but for the data model keep whatever postgres is offering (as that should be the same on the OM server side)

You can take other PRs as examples. I shared one above. But in the end, type mapping, sqlalchemy etc. needs to be dependant on each connector. The overall framework is already designed to force you to touch as few things as possible.

[geoHeil]

I have created some preliminary DDB support - however outside of OMs standard ingestion framework - simply manually calling the API - would anyone be interested in re-using this?