[SPARK-46640][FOLLOW-UP] Consider the whole expression tree when excluding subquery references

[nikhilsheoran-db]

What changes were proposed in this pull request?

• Context: In RemoveRedundantAliases, we exclude outerAttrs referenced by subquery expressions when trying to identify aliases that can be removed. Failure to do so could lead to scenarios where certain part of the plans (for example: the join conditions in exists subquery) would end up with duplicated expression ID(s). A fix for this was merged here in the PR here - [SPARK-46640][SQL] Fix RemoveRedundantAlias by excluding subquery attributes #44645.

• The fix there only accounted for the plan.expressions to be a SubqueryExpression.

• We can have scenarios where the SubqueryExpression is wrapped in other expressions -- for example, Alias < CaseWhen < (SubqueryExpression).

• To correctly account for these, need to traverse the whole expression tree and collect outer references from these.

• Consider plans of the form:

Project [CASE WHEN exists#2 [a#1 && (a#1 = a#0)] THEN 1 ELSE 2 END AS result#3]
:  +- LocalRelation <empty>, [a#0]
+- Project [a#0 AS a#1]
  +- LocalRelation <empty>, [a#0]

• The current rule would have rewritten this as follows. Note the join condition with conflicting expression ID(s) a#0 = a#0

Project [CASE WHEN exists#2 [a#0 && (a#0 = a#0)] THEN 1 ELSE 2 END AS result#3]
:  +- LocalRelation <empty>, [a#0]
+- LocalRelation <empty>, [a#0]

Why are the changes needed?

• Queries that would have run successfully with this rule disabled would fail.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Added unit tests

Was this patch authored or co-authored using generative AI tooling?

No

[nikhilsheoran-db]

cc: @agubichev @cloud-fan to take a look.

[cloud-fan]

good catch!