Determine a system for cataloging dbt plugins

[drewbanin]

I love that there are so many new dbt plugins in the world! At the time of this writing, I know about:

• https://github.com/mikaelene/dbt-sqlserver
• https://github.com/jacobm001/dbt-mssql
• an in-progress Vertica plugin
• an in-progress Azure Data Warehouse plugin
• an in-progress MySQL plugin

We should figure out a good way to catalog these plugins such that users can determine the list of community-contributed plugins out there in the world. This is useful for two groups of people:

1. dbt users who want to use dbt with their database of choice
2. dbt users who want to develop (or contribute to) an existing plugin

An obvious place to start is the dbt documentation. I've already added a link to the dbt-mssql plugin in the list of supported databases in the documentation. This sort of breaks down if a plurality of plugins exist for a given database - I think it would be unwise for us to try to keep track of the latest documentation for every community-contributed plugin out there! Maybe instead, we can add a list of community-contributed plugins, separated out by database type, then leave the documentation to the owner of the plugin.

We should also consider what the path looks like to graduate a plugin from "community supported" to "officially supported". I think this will mostly entail:

1. more complete documentation at docs.getdbt.com
2. contributing patches when dbt-core is updated
3. the level of support we (core dbt maintainers) are able to provide if users have questions / issues

I'm very curious to hear what the maintainers of dbt plugins think about all of this! Ultimately, this discussion is motivated by a desire to help you in your plugin-building endeavors, so please don't hesitate to let me know what you think!

cc @ichavez @jacobm001 @mikaelene @norton120

[mikaelene]

Hi,

Sorry for late replay. I've been on a long vacation :-).

I think it's great to be able to create our own adapters and that others can make use of them and contribute.

From my experience when developing my sqlserver plugin, there are a lot of missing test cases in dbt-integration-tests. An update to that to include newer features, like snapshots etc, would be great and make it easier.

There is of course problems with multiple adapters for the same database. From my perspective I will need to continue develop my sqlserver plugin. It supports features that we need in our version of sqlserver (2017) and @jacobm001s mssql support features he needs in sqlserver 2008. With that in mind maybe a list is better until we have an adapter that support a wider set of versions?

I think it is quite a large job to make a new adapter a first class citizen in dbt. For sqlserver you would need pretty much the same level of customization as you have done for Bigquery. I think you guys at Fishtown has to decide if it worth it on your end as it will need some of your time to coordinate and make sure we just don't ad a crappy adapter to the great dbt :-).

/Mikael

[drewbanin]

I really appreciate that perspective @mikaelene! Totally agree that improving dbt-integration-tests is a super high-leverage thing we can do here.

I think I agree with you, we should provide a list of "known" plugins, then contributors combine their efforts or develop code in parallel. I think your example of sqlserver 2008 vs 2017 is a really good one!

Edit: just created #1685 to track improvements to dbt-integration-tests

[jordan8037310]

@drewbanin - is there a link somewhere to the in progress dbt-vertica plugin?

[drewbanin]

hey @jordan8037310 - I'm not aware of any public vertica plugins for dbt, but maybe you could do a quick google search and see if there are any out there on the net! If so, I'd love to document it!

[mpcarter]

@jordan8037310 I have been working on a dbt plugin for Vertica the past week or so. It is not quite ready yet, but you can check it out here https://github.com/mpcarter/dbt-vertica

I have successfully passed 2 of the 3 features of dbt-integration-tests.

[jordan8037310]

@mpcarter that's great! we're not yet near an implementation phase but collecting some info at this point. happy to contribute once we get moving, too

[drewbanin]

@mpcarter just an FYI if you haven't seen it elsewhere: we're going to put some work in to build a better integration test suite that makes it easier to test community-supported plugins. I can follow up with more info on that here when we have some code in place!

[mpcarter]

@drewbanin Sounds good 👍 FYI the Vertica plugin seems to work. Have not really put it through the paces yet though.

[noel]

Came across this thread today and thought I would add that we created a registry of dbt related plugins
https://datacoves.com/dbt-libs

There's a specific page for dbt database connectors
https://datacoves.com/dbt-ibraries-categories/database-adapters

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest; add a comment to notify the maintainers.