Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dropping Spark 3.2 support #565

Closed
huaxingao opened this issue Jun 13, 2024 · 8 comments · Fixed by #581
Closed

Dropping Spark 3.2 support #565

huaxingao opened this issue Jun 13, 2024 · 8 comments · Fixed by #581

Comments

@huaxingao
Copy link
Contributor

As we actively work on enhancing Comet by adding support for the latest Apache Spark 3.5, and with the upcoming release of Spark 4.0, I propose that we discontinue support for the older Spark 3.2 version. This change will reduce the burden of maintaining compatibility across multiple versions of Spark, allowing us to focus our efforts on integrating and optimizing for the most current releases. Please share any concerns or questions you may have regarding this proposal.

@andygrove
Copy link
Member

+1

@viirya
Copy link
Member

viirya commented Jun 13, 2024

+1 if no others have strong demand on Spark 3.2 support.

@huaxingao
Copy link
Contributor Author

also cc @sunchao @kazuyukitanimura @parthchandra

@parthchandra
Copy link
Contributor

I'm +1 on this too. Maintaining more than two versions definitely slows down development.

@kazuyukitanimura
Copy link
Contributor

+1
It might be a good idea to post the link of this issue on ASF slack #datafusion-comet channel to notify more folks

@eejbyfeldt
Copy link
Contributor

+1

@advancedxy
Copy link
Contributor

+1.

Maintaining more than two versions definitely slows down development.

Maybe we should define a supporting policy about how and which Spark versions should be supported.

@sunchao
Copy link
Member

sunchao commented Jun 18, 2024

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants