Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync Spark Connect pb from upstream (pre Spark 4.0.0) #52

Closed
wants to merge 1 commit into from

Conversation

pan3793
Copy link
Member

@pan3793 pan3793 commented Jan 23, 2025

Why are the changes needed?

Sync pb file from Spark latest branch-4.0

$ git log --oneline 5b2d2149b6..8a1f4acead -- sql/connect/common/src/main/protobuf/spark/connect
ad83f71d78d [SPARK-50893][CONNECT] Mark UDT.DataType optional
f08319aab5f [SPARK-49273][CONNECT][SQL] Origin support for Spark Connect Scala client
57f33ee34e3 [SPARK-50869][ML][CONNECT][PYTHON] Support evaluators on ML Connet
93e198d2fb8 [SPARK-50851][ML][CONNECT][PYTHON] Express ML params with `proto.Expression.Literal`
15173157c33 [SPARK-49907][ML][CONNECT][FOLLOWUP] Remove unused `expressions.proto` import
fafe43c87d8 [SPARK-49907][ML][CONNECT] Support spark.ml on Connect
b210f422b00 [SPARK-49087][SQL][CONNECT] Distinguish UnresolvedFunction calling internal functions
a3a8d1ca668 [SPARK-50715][PYTHON][CONNECT] `SparkSession.Builder` sets the configs in batch
9087d84b512 [SPARK-50710][CONNECT] Add support for optional client reconnection to sessions after release
3f333a0820a [SPARK-50642][CONNECT][SS] Fix the state schema for FlatMapGroupsWithState in spark connect when there is no initial state
aac494e74c6 [SPARK-50134][SPARK-50130][SQL][CONNECT] Support DataFrame API for SCALAR and EXISTS subqueries in Spark Connect
1f6cb60e3c7 [SPARK-50544][PYTHON][CONNECT] Implement `StructType.toDDL`
48efe3f57f9 [SPARK-50134][SPARK-50132][SQL][CONNECT][PYTHON] Support DataFrame API for Lateral Join in Spark Connect
cb5938363ff [SPARK-50075][SQL][PYTHON][CONNECT] Add DataFrame APIs for table-valued functions
b1ff7672cba [SPARK-49857][SQL] Add storageLevel to Dataset localCheckpoint API
23bea28acfc [SPARK-49383][SQL][PYTHON][CONNECT] Support Transpose DataFrame API
0895471ad3b [MINOR] Fix spelling mistake in comment

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before make a pull request

@pan3793 pan3793 requested a review from cxzl25 January 23, 2025 06:41
@pan3793 pan3793 closed this in 24e73a2 Feb 7, 2025
@pan3793
Copy link
Member Author

pan3793 commented Feb 7, 2025

Merged to master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants