Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(mlop-2269): bump versions #355

Merged
merged 30 commits into from
May 29, 2024
Merged

feat(mlop-2269): bump versions #355

merged 30 commits into from
May 29, 2024

Conversation

ralphrass
Copy link
Contributor

@ralphrass ralphrass commented May 23, 2024

Why? 📖

Modernize Butterfree by adjusting support to Spark 3.5. We also need No Time Zone definitions and maybe Spark Connect that are available in this version.

What? 🔧

  • Support to Spark 3.5. Ready for Databricks runtime >= 13.x. Ready for No timezone definitions.
  • Bump Python from 3.7 to 3.9.
  • Bump pyArrow to 16.1.0
  • Bump setuptools to 70.0.0
  • Bump wheel to 0.43.0
  • Bump h3 to 3.7.7
  • Bump mypy to 1.10.0
  • Bump flake to 4.0.1
  • Adjust tests due Spark new behaviors (default dir to store parquet files, schema validations and floating-point precision calculations)

Type of change

  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

How everything was tested? 📏

Local and CI tests

Attention Points ⚠️

Require MINOR version upgrade rather than PATCH.

Several libs could't be easily updated such as typer, cassandra-driver, pytest, flake and even mypy. This libs need detailed investigation due incompatibilities to the current state of the project and dependencies crashes. Mypy was updated in a forced way because the old version is not compatible with Python 3.9. I had to turn almost all of mypy's validations off because it raises over 123 errors.

@ralphrass ralphrass self-assigned this May 23, 2024
@ralphrass ralphrass changed the title Feat: bump versions for Spark 3.4 feat(mlop-2269): bump versions for Spark 3.4 May 28, 2024
@ralphrass ralphrass marked this pull request as ready for review May 28, 2024 20:53
@ralphrass ralphrass requested a review from a team as a code owner May 28, 2024 20:53
@ralphrass ralphrass changed the title feat(mlop-2269): bump versions for Spark 3.4 feat(mlop-2269): bump versions May 28, 2024
.checklist.yaml Outdated Show resolved Hide resolved
butterfree/_cli/migrate.py Show resolved Hide resolved
butterfree/clients/cassandra_client.py Show resolved Hide resolved
Copy link

sonarcloud bot commented May 29, 2024

Quality Gate Passed Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
1.8% Duplication on New Code

See analysis details on SonarCloud

Copy link

@michellyrds michellyrds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ralphrass ralphrass merged commit 887fbb2 into staging May 29, 2024
6 checks passed
ralphrass added a commit that referenced this pull request Jun 5, 2024
* feat(mlop-2269): bump versions (#355)

* fix: bump versions adjust tests

* add checklist

* chore: bump python

* bump pyspark

* chore: java version all steps modified

* fix: sphinx version (#356)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants