Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AUDIT] test the RAPIDS plugin with the spark-connect plugin #9375

Open
gerashegalov opened this issue Oct 4, 2023 · 6 comments
Open

[AUDIT] test the RAPIDS plugin with the spark-connect plugin #9375

gerashegalov opened this issue Oct 4, 2023 · 6 comments
Labels
audit General label for audit related tasks audit_3.5.0 Spark 3.5+ Spark 3.5+ issues test Only impacts tests

Comments

@gerashegalov
Copy link
Collaborator

Spark connect modifies how jars are associated with the spark context / session apache/spark@caa3df4

Add tests ensuring, spark-rapids works with connect.
@gerashegalov gerashegalov added audit General label for audit related tasks Spark 3.5+ Spark 3.5+ issues ? - Needs Triage Need team to review and classify audit_3.5.0 labels Oct 4, 2023
@mattahrens mattahrens added test Only impacts tests and removed ? - Needs Triage Need team to review and classify labels Oct 10, 2023
@razajafri
Copy link
Collaborator

@mattahrens @sameerz
How important do you guys think this feature is for 24.04?

@sameerz sameerz added the ? - Needs Triage Need team to review and classify label Mar 11, 2024
@gerashegalov
Copy link
Collaborator Author

I presume the main driver for connect will be support for an LTS of Databricks 14 https://docs.databricks.com/en/release-notes/runtime/14.0.html. 14.3 LTS was released on Feb 1 https://docs.databricks.com/en/release-notes/runtime/index.html

@razajafri
Copy link
Collaborator

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

@gerashegalov
Copy link
Collaborator Author

What about Apache Spark 3.5.1? Since 14.3 is branching off of it.

Upstream users have a choice, whereas it does not look like Shared Cluster users can opt out https://docs.databricks.com/en/release-notes/runtime/14.0.html#introducing-spark-connect-in-shared-cluster-architecture

@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Mar 12, 2024
@sameerz
Copy link
Collaborator

sameerz commented Mar 15, 2024

We can revisit when we work on 4.0 or Databricks 14.x shims. For 3.5.1 it is not critical.

@gerashegalov
Copy link
Collaborator Author

Related user question #10611

@gerashegalov gerashegalov mentioned this issue Jun 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
audit General label for audit related tasks audit_3.5.0 Spark 3.5+ Spark 3.5+ issues test Only impacts tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@gerashegalov @mattahrens @sameerz @razajafri