Run tests on BigQuery, Snowflake, Databricks, duckdb

[katy-sadowski]

Get access to the BigQuery, Snowflake, and Databricks DBs and add them to our remote DB tests. Also add duckdb via Eunomia.

As part of this work, refactor so that the remote DB tests all run in parallel, to save time: https://testthat.r-lib.org/articles/parallel.html

Snowflake form: https://docs.google.com/forms/d/e/1FAIpQLSe_1ZaEqvfHjWrOUiIV9vGwCpFjlPQCLvwm84_eASFsCSv8Ww/viewform

[natb1]

If helpful, I am testing DQD against a Databricks deployment currently. I can't offer "access" to a Databricks deployment, but I could assist with executing some tests for development purposes. I could potentially offer to adapt any existing tests for Databricks, but I haven't looked at that area of the code to know if that makes sense.

[katy-sadowski]

Thanks so much for your comment @natb1 ! I believe there might be some possible arrangement to set up our automated tests on a Databricks instance (for which John Gresh is the contact - for some reason I can't tag him here). I just haven't taken the time to follow up on that yet 🙃 hence me filing this issue to make it public that I need to do this.

It's actually quite easy to adapt our tests to Databricks - just a matter of adding it to the list here if and when I can get open access to a test instance:

DataQualityDashboard/tests/testthat/test-executeDqChecks.R

Lines 119 to 124 in 6ef7ee2

	dbTypes <- c(
	"oracle",
	"postgresql",
	"sql server",
	"redshift"
	)

In the meantime, please don't hesitate to report any issues or bugs you run into with DQD on Databricks! No issue is too small; I am happy to help support 😄

[natb1]

Cool, ty. I'll let you know if anything pops up - good to know everything should probably run smooth.