-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-29281][SQL] Correct example of Like/RLike to test the origin intention correctly #25957
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
…ntention correctly
|
Relevant tests could fail as we discovered flaky tests, and the fix seems to be submitted via #25956 |
|
Test build #111524 has finished for PR 25957 at commit
|
|
Test build #111525 has finished for PR 25957 at commit
|
|
I think of ignoring sql commands in examples by prefix |
|
Thanks, yes I had to exclude Like as well in other test as same reason RLike was excluded. Just updated. |
|
Test build #111529 has finished for PR 25957 at commit
|
|
Merged to master. |
|
Thanks all for reviewing and merging! |
What changes were proposed in this pull request?
This patch fixes examples of Like/RLike to test its origin intention correctly. The example doesn't consider the default value of spark.sql.parser.escapedStringLiterals: it's false by default.
Please take a look at current example of Like:
spark/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
Lines 97 to 106 in d72f398
If spark.sql.parser.escapedStringLiterals=false, then it should fail as there's
\Uin pattern (spark.sql.parser.escapedStringLiterals=false by default) but it doesn't fail.For the query
SQL parser removes single
\(not sure that is intended) so the expressions of Like are constructed as following (I've printed out expression of left and right for Like/RLike):which are no longer having origin intention (see left).
Below query tests the origin intention:
Note that
\\\\is needed in pattern asStringUtils.escapeLikeRegexrequires\\to represent normal character of\.Same for RLIKE:
which is OK, but
which no longer haves origin intention.
Below query tests the origin intention:
Why are the changes needed?
Because the example doesn't test the origin intention. Spark is now running automated tests from these examples, so now it's not only documentation issue but also test issue.
Does this PR introduce any user-facing change?
No, as it only corrects documentation.
How was this patch tested?
Added debug log (like above) and ran queries from
spark-sql.