Druid dataset URN is not generated correctly

[MrdjanPoletanovic]

Describe the bug
When I sync Druid metadata, the URNs are generated using only the dataset name. For example, if the Druid dataset is named 'test-dataset', the URN is created as urn:li:dataset:(urn:li:dataPlatform:druid,test-dataset,PROD) without including a schema name before the dataset name. However, if I add the platform_instance property (e.g., marketing) in the Druid metadata spec configuration, the URN is created as urn:li:dataset:(urn:li:dataPlatform:druid,marketing.marketing.adevents,PROD), repeating the platform_instance twice.

Why is this an issue? When I try to sync Superset dashboards, Datahub cannot establish lineages between them due to the inconsistency in how the URNs are generated.

To Reproduce
Steps to reproduce the behavior:

1. Go to 'Ingestion' tab and specify Druid configuration
2. Go to imported datasets and select one of them
3. Look at the generated URN in browser url

Expected behavior
I believe the expected behavior is for the dataset name to be concatenated with the schema name. For example: urn:li:dataset:(urn:li:dataPlatform:druid,druid.test-dataset,PROD) if the schema name is druid, or urn:li:dataset:(urn:li:dataPlatform:druid,segments.test-dataset,PROD) if the schema name is segments.

Desktop (please complete the following information):

• Browser : Firefox
• Chart version: 0.14.0.2

** Additional Context **

• Check the file and method get_identifier in Druid config: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/druid.py#L52
• Check the generating datasets URN in SqlAlchemySource class method _process_table: https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/src/datahub/ingestion/source/sql/sql_common.py#L767

[jjoyce0510]

Hi @MrdjanPoletanovic - thanks for raising this.

This definitely looks like a serious issue in how we construct URNs for Druid. While the core team will not be focusing on this in the short term, we are open to contributions to adjust the constructions of URNs for Datasets and Containers. We'll certainly need to publish a major update in our release notes warning folks about this change since it will likely invalidate previous URNs ingested from Druid.

Cheers
John

[Rasnar]

This simple PR should be able to fix the main issue, being the fact that the platform_instance is added twice to the URN:
#12716

We have been using this fix internally for some time, and it works as expected.

There is more work to be done on the Druid ingestion, because some things are not exactly correct, for example, we have the database name being set to druid/v2/sql which is because of the underlying implementation of the SQL alchemy.

[sgomezvillamor]

Solved with #12716