openlineage metadata not supported with dbt-glue adaptor

[bhawesh490]

Hi
Am using dbt glue adaptor for one of my client.
however there is the need for openlineage solution and we are trying to implement marquez with dbt glue adaptor
Whenever i am using dbt glue adaptor along with cosmos using dbtdag operator i am not able to see the lineage in marquez ui
however same is visible in any other adaptor like dbt-redshift dbt-snowflake.
can you suggest any workaround to solve this

[dosubot]

To address the issue of OpenLineage metadata not appearing in Marquez when using the dbt-glue adapter with Cosmos, ensure the following configurations are in place:

1. OpenLineage Provider: Make sure you have the appropriate OpenLineage provider installed. If you are using Airflow 2.7 or higher, install apache-airflow-providers-openlineage. For earlier versions, install Cosmos with astronomer-cosmos[openlineage] ^[1].

2. Namespace Configuration: Configure the namespace using the Airflow [openlineage] namespace property, the OPENLINEAGE_NAMESPACE environment variable, or default to "cosmos" ^[1].

3. Logging for Debugging: Utilize the logging features in Cosmos to diagnose issues. The codebase includes debug logging that can help identify problems with OpenLineage event parsing. Look for debug messages like "Unable to parse OpenLineage events" to gain insights into potential issues ^[2].

These steps should help in troubleshooting and potentially resolving the issue with OpenLineage metadata not appearing in Marquez when using the dbt-glue adapter.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[tatiana]

Hi @bhawesh490 , thanks for reaching out.

This is a known limitation of the open-source library Cosmos uses to extract open-lineage data after running dbt commands, openlineage-integration-common.

These are the dbt adaptors currently supported :
https://github.com/OpenLineage/OpenLineage/blob/41afe4dd50773a5c0902679c91622031d64a06b9/integration/common/openlineage/common/provider/dbt/processor.py#L36C1-L47C22

Would you be interested in contributing to the OpenLineage project to extend it to support dbt-glue? Not only the OL community would benefit from this, but also the Cosmos community.

@mobuchowski do you know if there are any plans of supporting dbt-glue in this component?