dbt-spark-cde

The dbt-spark-cde adapter allows you to use dbt Cloudera Data Platform with CDE API server support. This code bases use the dbt-spark project (https://github.com/dbt-labs/dbt-spark), and provides CDE API based connectivity support over it.

Getting started

• Install dbt
• Read the introduction and viewpoint

Requirements

Python >= 3.9

dbt-core ~= 1.3.0

requests >= 2.28.1

requests-toolbelt >= 0.9.1

pyspark

requests_kerberos

requests-toolbelt

python-decouple

Installing dbt-spark-cde

pip install dbt-spark-cde

Profile Setup

demo_project:
  target: dev
  outputs:
    dev:
     type: spark_cde
     method: cde
     schema: my_db
     auth_endpoint: https://service.spark-cde-gateway.my.org.com/
     host: https://spark-cde-gateway.my.org.com/dex/api/v1/
     user: my_user
     password: my_pass

• To obtain auth_endpoint follow the steps here: (https://docs.cloudera.com/data-engineering/cloud/api-access/topics/cde-api-get-access-token.html)

Caveats

• While using cde , in the Livy UI if you notice sessions change state to dead from starting instead of idle, make sure there is a proper mapping for the user in the IDBroker mapping section
• Actions > Manage Access > IDBroker Mappings . Reference
• Also make sure the workload password is set either through UI or CLI. Reference

Supported features

Please see the original adapter documentation: https://github.com/dbt-labs/dbt-spark and https://docs.getdbt.com/reference/warehouse-profiles/spark-profile