Test for Koalas DataFrames #5928
Conversation
| @@ -57,6 +57,25 @@ def model(dbt, session): | |||
| return df | |||
| """ | |||
|
|
|||
| KOALAS_MODEL = """ | |||
| import databricks.koalas as ks | |||
There was a problem hiding this comment.
@dbeatty10 Seems like this is only for databricks, I would put this one as a separate test in dbt-spark, maybe somewhere here? Just as a new test class should be good enough
There was a problem hiding this comment.
I think that Koalas DataFrames can also be used on BigQuery even though the package is published by Databricks.
I did manual testing of Koalas on BigQuery here:
dbt-labs/dbt-bigquery#321
There was a problem hiding this comment.
Koalas is a separate package from Spark and should be installed in the target cluster to pass this test.
There was a problem hiding this comment.
There was a problem hiding this comment.
When the target cluster is DBR >=7.1,<11.0, Koalas should be available as it's pre-installed; otherwise, you need to install it in the cluster UI.
resolves #5927
Continuing work merged in #5906
Description
Test case for Koalas DataFrames (the predecessor to pandas-on-Spark DataFrames AKA pandas API DataFrames).
Group of related pull requests
Checklist
I have opened an issue to add/update docs, ordocs changes are not required/relevant for this PRI have runchangie newto create a changelog entry