dbt Core

[amotl]

About

Make adapter shippable, and reflect its capabilities.

Details

I would like to introduce @hlcianfagna and @andnig to each other, because they seem to have a shared interest to make dbt-cratedb fly again. I think Hernan already has valuable improvements in some of his feature branches at dbt-cratedb1.5. Hereby, we are aiming for a designated and canonical dbt-cratedb2.

• https://pypi.org/project/dbt-cratedb/
• https://github.com/crate-workbench/dbt-cratedb

References

• [Meta] dbt - PG connector compatibility tracker  crate#13234

[amotl]

Status quo

The current canonical documentation is Using dbt with CrateDB by @hlcianfagna and @hammerhead.

Outlook

In an offline discussion, @hlcianfagna and me agreed to spend some time on making the dbt adapter shippable. After that, we may want to adjust the documentation accordingly.

[amotl]

Q: Can we do the same with CrateDB and its dbt adapter like what Snowflake is doing here? If so, my proposal is to also use »CrateDB Toolkit« to absorb what is presented here on behalf of »Snowpark ML Toolkit + dbt«.

About what's Snowflake is doing, that is a nice summarizing article, also referencing a few previous ones of the same series. Coming from our initial articles on this topic, currently being published, do you think it would be possible to level up to be en par with those, @andnig and @ckurze?

-- https://medium.com/snowflake/discover-the-new-snowpark-ml-toolkit-dbt-python-models-4f85b85a8c88
-- https://github.com/Snowflake-Labs/sfguide-intro-to-machine-learning-with-snowpark-ml-for-python

I did not investigate about their secret sauce yet, but I guess it is something along the lines of Ibis. It also looks like they are also leveraging dbt, so working on something like that could be a team effort bundling and building upon a few of our previous individual endeavours around those topics.

-- via: https://crate.slack.com/archives/C05R7FU17EV/p1697221704129079

[hlcianfagna]

Hi,
crate/dbt-cratedb2@main...hlcianfagna:dbt-cratedb2:hlcianfagna/2023MayUpdate
and
crate-workbench/dbt-cratedb-examples@main...hlcianfagna:dbt-cratedb-examples:hlcianfagna/2023MayUpdate
may be useful for this exercise.

Please note that since I last worked on that code CrateDB has added crate/crate#14064 and crate/crate#14953

[amotl]

Thanks. Do you suggest to ship your variant, no matter what the authors of https://pypi.org/project/dbt-cratedb/ are doing? Or do you think we should investigate once again on which details both variants show differences, maybe together?

I will be happy to follow both directions, depending on your suggestion.

[amotl]

Hi. After a conversation with @vvulf, I think we can count him in on the special interest group of people aiming to improve presentation of dbt technologies together with CrateDB.

[amotl]

Apparently, I already took the chance to translate the example repository ¹ by the author of dbt-cratedb, Julio Sánchez Jiménez, into English ² in preparation for a real "project". I just haven't used it yet, otherwise I would probably know more about dbt ;]. Maybe it could be used as a reasonable starting point, or even as the foundation for corresponding integration tests, when bringing all together?

Edit: I am only seeing now that @hlcianfagna already referenced it ² at https://github.com/crate/crate-clients-tools/issues/188, and that he also added a few commits over there.

Footnotes

1. https://github.com/jsnchzjmnz/ejercicios-dbt-cratedb ↩

2. https://github.com/crate-workbench/dbt-cratedb-examples/ ↩ ↩²

[amotl]

@geragray knows the CTO of dbt, and would like to get a corresponding well-maintained CrateDB adapter into the list of their trusted adapters, in order to improve visibility, co-marketing etc.

I am talking to dbt labs CTO about getting into the trusted adapter pages. Going through criteria for trusted adapters, it seems like it is fairly easy to get in ¹. It just needs to work plus maintained actively (by a company, not a community).

Footnotes

1. https://docs.getdbt.com/guides/adapter-creation?step=7 ↩

[amotl]

Status Quo

dbt-cratedb2 exists now.

• PyPI: https://pypi.org/project/dbt-cratedb2/
• GitHub: https://github.com/crate/dbt-cratedb2
• Documentation (onsite): https://cratedb.com/docs/guide/integrate/dbt/
• Documentation (remote): https://docs.getdbt.com/docs/core/connect-data-platform/cratedb-setup

[amotl]

Backlog 2024-12

• Platform connections: Add new data platform adapter for CrateDB dbt-labs/docs.getdbt.com#6564
• Genesis Backlog dbt-cratedb2#3
• Enhancement Backlog dbt-cratedb2#4