-
Notifications
You must be signed in to change notification settings - Fork 29k
[WIP][SPARK-43611][SQL][PS][CONNCECT] Fix unexpected AnalysisException from Spark Connect client
#42040
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
In #39925, we introduced a new mechanism to resolve expression with specified plan. However, sometimes the plan ID might be eliminated by the analyzer, and then some expressions can not be correctly resolved, this issue is the No.1 blocker of PS on Connect. Currently, I investigate the two examples in the ticket and check each rule applied to them. example 1: example 2: In the draft, I modify two rules to retain the plan id. (actually, I modified ResolveNaturalAndUsingJoin in 167bbca) I am wondering whether is there some graceful approach to fix this issue? Otherwise, I'm afraid I will touch more rules. |
| val newProject = Project(finalProjectList, withWindow) | ||
|
|
||
| // retain the plan id used in Spark Connect | ||
| planId.foreach(newProject.setTagValue(LogicalPlan.PLAN_ID_TAG, _)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is to fix example 2.
| } | ||
|
|
||
| // retain the plan id used in Spark Connect | ||
| planId.foreach(newPlan.setTagValue(LogicalPlan.PLAN_ID_TAG, _)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is to fix example 1
|
not sure whether this works:
|
|
Thanks for your time on investigating this issue! 👍 Since this issue is also blocking numerous test cases other than the two examples provided in the ticket, I have created a separate temporary test PR here: #42041 that cherry-picks this fix and enables all tests related to this issue to check if the fix is can be applied to all other test cases as well. P.S. If the cause of failure for every test is different and difficult to fix in one PR, we may need to break this issue into multiple sub-tasks. |
|
close this one in favor of #42086 |
What changes were proposed in this pull request?
Fix unexpected
AnalysisExceptionfrom Spark Connect clientWhy are the changes needed?
to make PS work on Connect
Does this PR introduce any user-facing change?
yes
How was this patch tested?
manually check