[SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R #33992

yoda-mon · 2021-09-14T07:07:09Z

What changes were proposed in this pull request?

octet_length: caliculate the byte length of strings
bit_length: caliculate the bit length of strings
Those two string related functions are only implemented on SparkSQL, not on Scala, Python and R.

Why are the changes needed?

Those functions would be useful for multi-bytes character users, who mainly working with Scala, Python or R.

Does this PR introduce any user-facing change?

Yes. Users can call octet_length/bit_length APIs on Scala(Dataframe), Python, and R.

How was this patch tested?

unit tests

into add-bit-octet-length

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala

sarutak · 2021-09-14T07:41:43Z

ok to test.

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala

python/pyspark/sql/tests/test_functions.py

python/pyspark/sql/functions.py

HyukjinKwon · 2021-09-14T08:48:16Z

the change looks fine otherwise.

SparkQA · 2021-09-14T08:49:20Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47752/

SparkQA · 2021-09-14T11:33:51Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47756/

SparkQA · 2021-09-14T11:43:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47756/

yoda-mon · 2021-09-14T11:59:23Z

@HyukjinKwon @sarutak
Thank you for your comments, I fixed the commented points.

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala

SparkQA · 2021-09-14T12:27:29Z

Test build #143259 has finished for PR 33992 at commit 545fbe2.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-14T12:35:45Z

retest this please.

SparkQA · 2021-09-14T12:38:24Z

Test build #143249 has finished for PR 33992 at commit c0ffdce.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-14T13:02:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/

SparkQA · 2021-09-14T13:12:40Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47762/

SparkQA · 2021-09-14T13:40:23Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47763/

SparkQA · 2021-09-14T13:55:24Z

Test build #143253 has finished for PR 33992 at commit 3a07337.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

LGTM otherwise. Let me leave it to @sarutak

R/pkg/R/functions.R

python/pyspark/sql/functions.py

python/pyspark/sql/tests/test_functions.py

python/pyspark/sql/functions.pyi

SparkQA · 2021-09-14T17:35:23Z

Test build #143260 has finished for PR 33992 at commit 545fbe2.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T01:58:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/

SparkQA · 2021-09-15T02:08:27Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47791/

SparkQA · 2021-09-15T02:35:46Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/

SparkQA · 2021-09-15T02:44:45Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47793/

SparkQA · 2021-09-15T05:39:06Z

Test build #143288 has finished for PR 33992 at commit afa1700.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-09-15T06:22:32Z

Test build #143290 has finished for PR 33992 at commit 41656f4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sarutak · 2021-09-15T07:27:37Z

Merged to master. Thanks @yoda-mon and @HyukjinKwon !

AmplabJenkins · 2021-09-15T13:14:36Z

Can one of the admins verify this patch?

…or Examples in docstring ### What changes were proposed in this pull request? This PR is a minor followup of #33992 to fix the warnings during PySpark documentation build: ``` /.../spark/python/pyspark/sql/functions.py:docstring of pyspark.sql.functions.bit_length:19: WARNING: Unexpected section title or transition. ------- /.../spark/python/pyspark/sql/functions.py:docstring of pyspark.sql.functions.octet_length:19: WARNING: Unexpected section title or transition. ------- ``` We should always have the same length of hyphens with the title. ### Why are the changes needed? To remove warnings during the documentation build and show the HTML pages correctly. ### Does this PR introduce _any_ user-facing change? This is not released yet, and only in master branch. So, no to end users. ### How was this patch tested? Manually built the docs via `make clean html` at `python/docs` directory. Closes #34196 from HyukjinKwon/SPARK-36751. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

yoda-mon added 15 commits September 10, 2021 18:23

Implement scala/python octet_length

1df379b

Implement R octet_length

6df4668

Implement bit-length functions

eb2456b

Rearange the order of octet_lengh

3e149fa

Disable scalastyle to use non-ascii charactor on tests

ff2f031

Insert newline to python docstrings to keep pylint rules

5b9c98b

add space after comma to pass lint-R

848f224

Implement scala/python octet_length

4edc918

Implement R octet_length

5370a51

Implement bit-length functions

72c9398

Rearange the order of octet_lengh

761f8b6

Disable scalastyle to use non-ascii charactor on tests

f7ace95

Insert newline to python docstrings to keep pylint rules

2ed29f5

add space after comma to pass lint-R

0858001

Merge branch 'add-bit-octet-length' of https://github.com/yoda-mon/spark

c0ffdce

into add-bit-octet-length

github-actions bot added CORE PYTHON R SQL labels Sep 14, 2021

sarutak reviewed Sep 14, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 14, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 14, 2021

View reviewed changes

python/pyspark/sql/tests/test_functions.py Show resolved Hide resolved

HyukjinKwon reviewed Sep 14, 2021

View reviewed changes

python/pyspark/sql/tests/test_functions.py Outdated Show resolved Hide resolved

HyukjinKwon reviewed Sep 14, 2021

View reviewed changes

python/pyspark/sql/functions.py Show resolved Hide resolved

yoda-mon added 2 commits September 14, 2021 18:42

Delete unnecessary line in the scala test.

327d7d5

Add a short description of the test

387a92f

sarutak mentioned this pull request Sep 14, 2021

[SPARK-36683][SQL] Add new built-in SQL functions: SEC and CSC #33988

Closed

sarutak reviewed Sep 14, 2021

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala Outdated Show resolved Hide resolved

differentiate octet_length and bit_length test name

545fbe2

HyukjinKwon approved these changes Sep 14, 2021

View reviewed changes

HyukjinKwon reviewed Sep 14, 2021

View reviewed changes

python/pyspark/sql/functions.pyi Show resolved Hide resolved

yoda-mon added 3 commits September 15, 2021 10:06

Formatting decstrings

afa1700

differentiate test comments

127da47

Add bit/octet length to the api document

41656f4

sarutak approved these changes Sep 15, 2021

View reviewed changes

sarutak closed this in 0666f5c Sep 15, 2021

HyukjinKwon mentioned this pull request Oct 6, 2021

[SPARK-36751][PYTHON][DOCS][FOLLOW-UP] Fix unexpected section title for Examples in docstring #34196

Closed

[SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R #33992

[SPARK-36751][SQL][PYTHON][R] Add bit/octet_length APIs to Scala, Python and R #33992

Uh oh!

Conversation

yoda-mon commented Sep 14, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HyukjinKwon commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

yoda-mon commented Sep 14, 2021

Uh oh!

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

sarutak commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Sep 14, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

SparkQA commented Sep 15, 2021

Uh oh!

sarutak commented Sep 15, 2021

Uh oh!

AmplabJenkins commented Sep 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development