Genesis: Provide dbt adapter for CrateDB #2

amotl · 2024-11-20T01:36:02Z

About

Provide dbt adapter for CrateDB, carved out of the canonical PostgreSQL adapter dbt-postgres, making a few test cases succeed.

Details

Unit Tests: 34 passed, 2 skipped
Integration Tests: 426 passed, 103 skipped

References

dbt: Start using dbt-cratedb2 cratedb-examples#754

/cc @surister, @kneth

CrateDB does not understand: - SQL's `CREATE SCHEMA` and `DROP SCHEMA` operations - SQL's `EXCEPT` operation - SQL's `CASCADE` modifier - SQL's `CREATE TEMPORARY TABLE - PostgreSQL's synthetic `pg_rewrite` table - PostgreSQL's `pg_is_other_temp_schema()` function dbt features disabled: - Snapshots CrateDB needs: - Write synchronization using `REFRESH TABLE ...`

- Use dbt-core>=1.9.0b4 - Add configuration snippets for black, flake8, mypy, pytest

CrateDB does not support MATERIALIZED VIEW.

amotl · 2024-11-27T00:32:21Z

README.md

 ## Getting started

- [Install dbt](https://docs.getdbt.com/docs/installation)
- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)
+- [Install dbt](https://docs.getdbt.com/docs/core/installation-overview)
+- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and
+  [viewpoint](https://docs.getdbt.com/community/resources/viewpoint)


Also link to the new canonical integration page at the CrateDB Guide here.

https://cratedb-guide--145.org.readthedocs.build/integrate/dbt/

It has been added per 0147433.

amotl · 2024-11-28T08:57:57Z

tests/functional/adapter/test_cratedb.py

+rename_table_sql = """
+{% set exists, table_source = get_or_create_relation(database=None, schema=target.schema, identifier="table_source", type="table") %}
+{% set exists, table_target = get_or_create_relation(database=None, schema=target.schema, identifier="table_target", type="table") %}
+
+{# Create table `source`. #}
+{% set sql = get_create_table_as_sql(False, table_source, "SELECT 42 as value") %}
+{% do run_query(sql) %}
+
+{# Rename table to `target`. #}
+{% do adapter.rename_relation(table_source, table_target) %}
+
+-- Dummy statement.
+SELECT TRUE AS id
+"""
+
+
+rename_view_sql = """
+{% set exists, view_source = get_or_create_relation(database=None, schema=target.schema, identifier="view_source", type="view") %}
+{% set exists, view_target = get_or_create_relation(database=None, schema=target.schema, identifier="view_target", type="view") %}
+
+{# Create view `source`. #}
+{% set sql = get_create_view_as_sql(view_source, "SELECT 42 as value") %}
+{% do run_query(sql) %}
+
+{# Rename view to `target`. #}
+{% do adapter.rename_relation(view_source, view_target) %}
+
+-- Dummy statement.
+SELECT TRUE AS id
+"""


Dear @hlcianfagna,

you asked about the rename_relation macro over at crate/cratedb-examples#754 (review). Thank you very much.

Hereby, I am adding an integration test with CrateDB that is using vanilla dbt Core's / dbt-postgres' rename_relation macro, and it seems like it works well without much ado.

So, do you think the other rename_relation you provided per ¹, originally taken from ², will still be neeeded?

With kind regards,
Andreas.

Footnotes

https://github.com/crate/cratedb-examples/blob/91c0bb9f74cf62b67a4aabe5a79f7c80b475d16b/framework/dbt/basic/macros/overrides.sql#L50-L86 ↩

https://community.cratedb.com/t/using-dbt-with-cratedb/1566 ↩

I think it could be removed now, thank you.

amotl · 2024-11-28T08:58:41Z

tests/functional/adapter/test_cratedb.py

+    def test_rename_relation_table(self, project):
+        """
+        Validate the vanilla `rename_relation` macro from dbt-postgres on tables.
+        """
+
+        result = run_dbt(["run", "--select", "rename_table"])
+        assert len(result) == 1
+
+        records = common.get_records(project, "table_target")
+        assert records == [(42,)]
+
+    def test_rename_relation_view(self, project):
+        """
+        Validate the vanilla `rename_relation` macro from dbt-postgres on views.
+        """
+
+        result = run_dbt(["run", "--select", "rename_view"])
+        assert len(result) == 1
+
+        records = common.get_records(project, "view_target")
+        assert records == [(42,)]


This is related to my last comment, just to outline the actual test cases which use both of the macro scripts above, to validate renaming both tables and views works well.

Perfect, yes, starting from CrateDB 5.5 as long as it does an ALTER TABLE and not an ALTER VIEW it should work fine without a need for the override.

dbt/include/cratedb/macros/adapters.sql

amotl

Hi. I just added a few comments about the proper slotting-in of the CrateDB macro override cratedb__reset_csv_table, with a humble request for feedback about your concerns, any improvements, or acknowledgements. Thanks!

amotl · 2024-11-28T21:43:30Z

dbt/include/cratedb/macros/materializations/seed.sql

+{# CrateDB: Use `DELETE FROM` instead of `TRUNCATE` #}
+{% macro cratedb__reset_csv_table(model, full_refresh, old_relation, agate_table) %}
+    {% set sql = "" %}
+    {% if full_refresh %}
+        {{ adapter.drop_relation(old_relation) }}
+        {% set sql = create_csv_table(model, agate_table) %}
+    {% else %}
+        {{ adapter.truncate_relation(old_relation) }}
+        {% set sql = "delete from " ~ old_relation.render() %}
+    {% endif %}
+
+    {{ return(sql) }}
+{% endmacro %}


I've derived this from @hlcianfagna's contributions at Using dbt with CrateDB. ~~However, it has not been activated yet, and I don't know yet what it is about / which code paths, subsystems, and test cases it could unlock.~~

dbt: Start using dbt-cratedb2 cratedb-examples#754 (comment)

This has been activated now, accompanied by a corresponding software test. See next comment.

amotl · 2024-11-28T21:50:36Z

tests/functional/adapter/test_cratedb.py

+reset_csv_table = """
+{# Create a random table. #}
+{% set exists, random_table = get_or_create_relation(database=None, schema=target.schema, identifier="random_table", type="table") %}
+{% set sql = get_create_table_as_sql(False, random_table, "SELECT 42 as value") %}
+{% do run_query(sql) %}
+
+{#
+This is a little excerpt from dbt/include/global_project/macros/materializations/seeds/seed.sql
+#}
+{% set full_refresh = (should_full_refresh()) %}
+{% set agate_table = None %}
+{% set sql_ddl = reset_csv_table(model=model, full_refresh=full_refresh, old_relation=random_table, agate_table=agate_table) %}
+{% set sql_dml = load_csv_rows(model, agate_table) %}
+{% set sql = get_csv_sql(sql_ddl, sql_dml) %}
+{% do run_query(sql) %}
+
+SELECT TRUE
+"""


That's the test case body for validating CrateDB's macro override cratedb__reset_csv_table. It is being used by the upstream materialization macro materialization seed, default in macros/materializations/seeds/seed.sql.

The test case is a bit too verbose for my taste, but I haven't been able to come up with a better one, because of my lack of knowledge and fluency in dbt.

@hlcianfagna: Maybe you have a better and more concise dbt recipe at hand, which (maybe indirectly) tests whether CrateDB's custom override cratedb__reset_csv_table is properly in place?

I was testing this manually, I had a tiny csv in the seeds folder and I was running dbt seed twice.

Thanks. I added a corresponding software test that invokes dbt seed twice per ad768c5, but this one apparently does not trigger reset_csv_table. So I've kept the previous software test in place for now. 🤷

amotl · 2024-11-28T21:51:23Z

tests/functional/adapter/test_cratedb.py

+    def test_reset_csv(self, project):
+        """
+        CrateDB needs an override for the `reset_csv_table` macro. Make sure it is in place.
+        """
+
+        result = run_dbt(["run", "--select", "reset_csv_table"])
+        assert len(result) == 1
+
+        records = common.get_records(project, "reset_csv_table")
+        assert records == [(True,)]


FYI: It's not much that is being validated here within the test case definition. Effectively, it is just about invoking the dbt run.

hlcianfagna

LGTM

amotl added 8 commits November 12, 2024 06:37

Establish cratedb namespace

fa80cfd

CrateDB adapter: Genesis

3136c01

CrateDB adapter: Adjust macros (naming things)

b3da468

CrateDB adapter: Satisfy a few software tests

029d9d4

Update pyproject.toml

f322544

- Use dbt-core>=1.9.0b4 - Add configuration snippets for black, flake8, mypy, pytest

Fix CI: Don't invoke scripts/setup_test_database.sql

040a25e

CrateDB: Derive from dbt-postgres. Remove PostgreSQL macros.

926a551

amotl force-pushed the genesis branch 2 times, most recently from 54f1401 to 6d37e3f Compare November 20, 2024 03:47

amotl added 5 commits November 20, 2024 05:31

CrateDB: Derive from dbt-postgres. Remove Python adapter code.

5b829a2

CrateDB does not support "UNLOGGED" tables, and "OOMMENT ON"

98f1a61

CrateDB: Does not support pg_sleep

a5391a7

Tests: Fix `tests/functional/test_store_test_failures.py

dc43dac

Tests: Fix tests/functional/test_timezones.py

9908d47

amotl force-pushed the genesis branch 5 times, most recently from 580a70e to b7146f7 Compare November 24, 2024 02:13

Tests: Unlock more test cases by removing ignores, skip only selectively

1048624

amotl force-pushed the genesis branch from b7146f7 to 1048624 Compare November 24, 2024 02:25

Macros: Refactor drop_relation and truncate_relation macros

85daf99

amotl mentioned this pull request Nov 25, 2024

dbt: Start using dbt-cratedb2 crate/cratedb-examples#754

Merged

Macros: Override list_relations_without_caching

3df86f5

CrateDB does not support MATERIALIZED VIEW.

amotl force-pushed the genesis branch from 3417f86 to 3df86f5 Compare November 25, 2024 22:12

amotl added 4 commits November 26, 2024 01:17

Dependencies: Trim list, effectively use dbt-postgres only.

1aba3e5

Release v0.0.1

668bbb9

Chore: Update installation instructions in README

7edc10b

Documentation: Update README. Add CrateDB logo. Refactor psycopg2 notes.

e25d2b4

amotl requested a review from hammerhead November 26, 2024 23:07

amotl commented Nov 27, 2024

View reviewed changes

amotl marked this pull request as ready for review November 27, 2024 01:05

This was referenced Nov 27, 2024

Genesis Backlog #3

Open

[Meta] dbt - PG connector compatibility tracker crate/crate#13234

Open

amotl force-pushed the genesis branch 3 times, most recently from 2c00602 to 915f3e0 Compare November 27, 2024 21:09

Documentation: Update README, rework "About", add project badges

0147433

amotl force-pushed the genesis branch from 915f3e0 to 0147433 Compare November 27, 2024 21:12

amotl commented Nov 28, 2024

View reviewed changes

dbt/include/cratedb/macros/adapters.sql Outdated Show resolved Hide resolved

amotl added 2 commits November 28, 2024 22:41

Add integration test case with CrateDB using rename_relation

c51c715

Macro: Add cratedb__reset_csv_table for DELETE FROM not TRUNCATE

4a5d3b2

amotl force-pushed the genesis branch from 3385045 to 4a5d3b2 Compare November 28, 2024 21:42

amotl commented Nov 28, 2024

View reviewed changes

hlcianfagna approved these changes Dec 2, 2024

View reviewed changes

amotl mentioned this pull request Dec 11, 2024

Platform connections: Add new data platform adapter for CrateDB dbt-labs/docs.getdbt.com#6564

Merged

4 tasks

Tests: Add test case using a CSV data seed

c6f3a95

amotl force-pushed the genesis branch from ad768c5 to e5d0802 Compare December 21, 2024 21:30

amotl added 2 commits December 21, 2024 22:45

CI: Use hatch --verbose run

36136eb

Dependencies: Update to "dbt-postgres>=1.9.0"

d49182e

amotl force-pushed the genesis branch from 19eb754 to d17fcb8 Compare December 21, 2024 21:47

amotl added 4 commits December 21, 2024 22:54

Dependencies: Use dbt-tests-adapter 1.10. Remove other dbt- packages.

c22811e

Dependencies: Pin versions for testing packages

0a220a3

CI: Verify support for Python 3.13

05e901e

Project: Advance status to "beta"

26a8399

amotl force-pushed the genesis branch from d17fcb8 to 26a8399 Compare December 21, 2024 21:57

amotl merged commit fa164d4 into main Dec 21, 2024
5 checks passed

amotl deleted the genesis branch December 21, 2024 22:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Genesis: Provide dbt adapter for CrateDB #2

Genesis: Provide dbt adapter for CrateDB #2

amotl commented Nov 20, 2024 •

edited

Loading

amotl Nov 27, 2024 •

edited

Loading

amotl Nov 27, 2024 •

edited

Loading

amotl Nov 28, 2024

hlcianfagna Dec 2, 2024

amotl Nov 28, 2024 •

edited

Loading

hlcianfagna Dec 2, 2024

amotl left a comment

amotl Nov 28, 2024

amotl Nov 28, 2024 •

edited

Loading

hlcianfagna Dec 2, 2024

amotl Dec 21, 2024 •

edited

Loading

amotl Nov 28, 2024 •

edited

Loading

hlcianfagna left a comment

Genesis: Provide dbt adapter for CrateDB #2

Genesis: Provide dbt adapter for CrateDB #2

Conversation

amotl commented Nov 20, 2024 • edited Loading

About

Details

References

amotl Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Nov 28, 2024

Choose a reason for hiding this comment

Footnotes

hlcianfagna Dec 2, 2024

Choose a reason for hiding this comment

amotl Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

hlcianfagna Dec 2, 2024

Choose a reason for hiding this comment

amotl left a comment

Choose a reason for hiding this comment

amotl Nov 28, 2024

Choose a reason for hiding this comment

amotl Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

hlcianfagna Dec 2, 2024

Choose a reason for hiding this comment

amotl Dec 21, 2024 • edited Loading

Choose a reason for hiding this comment

amotl Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

hlcianfagna left a comment

Choose a reason for hiding this comment

amotl commented Nov 20, 2024 •

edited

Loading

amotl Nov 27, 2024 •

edited

Loading

amotl Nov 27, 2024 •

edited

Loading

amotl Nov 28, 2024 •

edited

Loading

amotl Nov 28, 2024 •

edited

Loading

amotl Dec 21, 2024 •

edited

Loading

amotl Nov 28, 2024 •

edited

Loading