Pydeequ on Shared Databricks Cluster (OR) Serverless

[nehaphuloria]

I have been trying to use pydeequ on Databricks Shared cluster and Serverless but I keep getting the below error:

cannot import name 'deprecated' from 'typing_extensions'

Could someone confirm if pydeequ works on Shared cluster or serverless

[SemyonSinchenko]

Before you go father, I would like to draw your attention to the fact that python-deequ is based on the py4j calls while Databricks shared clusters / serverless are based on Spark-Connect and these clusters do not support py4j. There is an issue about adding support for Spark Connect. I still have a fully working prototype of Deequ port to Spark Connect with a PySpark client and I'm ready to return to the discussion about merging it to the Deequ-core and python-deequ at any time! On top of benefits of having Connect support (Databricks Serverless / Shared Clusters, EMR Serverless, etc.) it provides some additional benefits like a fix for the row-level results or missing implementations (like AnomalyCheckConfig that are tricky to implement with py4j).

@MrPowers CC

[nehaphuloria]

@SemyonSinchenko Thank you for getting back ! Quite interesting to see the prototype and its flexibility for SparkConnect. I am infact more interested to see the row level results which has been a challenge with great-expectations. Would this solution be integrated in the pydeequ library anytime soon?