Commits on Apr 11, 2024

1. Updated Spark 3.3 dependency …

   This commit updates the Spark 3.3 dependency of Deequ. There are some breaking changes to the Scala APIs, from a Py4J perspective. In order to work around that, we use the Spark version to switch between the updated API and the old API. This is not sustainable and will be revisited in a future PR, or via a different release mechanism. The issue is that we have multiple branches for multiple Spark versions in Deequ, but only one branch in PyDeequ.
   
   The changes were verified by running the tests in Docker against Spark version 3.3. The docker file was also updated so that it copies over the pyproject.toml file and installs dependencies in a separate layer, before the code is copied. This allows for fast iteration of the code, without the need to install dependencies every time the docker image is built.

   

   rdsharma26 committed Apr 11, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for 0cdb2db
   • Browse repository at this point

   Copy the full SHA

   0cdb2db View commit details

   Browse the repository at this point in the history