Support concat with separator on GPU #2479
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@apache.org>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
when its the separator
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
|
build |
|
CUDF changes all merged in now. |
|
build |
jlowe
changed the title
jlowe
reviewed
May 27, 2021
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
revans2
reviewed
May 27, 2021
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Thomas Graves <tgraves@apache.org>
|
build |
|
oops, still need to remove one thing |
|
ok, I think I addressed everyone's comments, if I missed something let me know. |
revans2
previously approved these changes
May 27, 2021
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
andygrove
reviewed
May 27, 2021
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
jlowe
previously approved these changes
May 27, 2021
|
build |
andygrove
reviewed
May 27, 2021
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
andygrove
reviewed
May 27, 2021
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala
Outdated
Show resolved
Hide resolved
…nctions.scala Co-authored-by: Andy Grove <andygrove73@gmail.com>
|
build |
andygrove
approved these changes
May 27, 2021
revans2
approved these changes
May 27, 2021
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* start concat_ws Signed-off-by: Thomas Graves <tgraves@apache.org> * more stuff * call into stringConcateWs Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes and more functionality for concat_ws * Test and debug 1 column Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Handle arrays and add more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix type checks * TEsts working and running on gpu * Fix empty str scalar and debug * debug * Update to latest cudf changes, tests all passing Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Try using scalar separator but cudf doesn't handle null scalars the same when its the separator * more changes * handle 1 column with scalar sep * All working with no leaks Signed-off-by: Thomas Graves <tgraves@nvidia.com> * updates * Fix line lengths * reogranize * refactor code to be more readable, all tests pass, no leaks Signed-off-by: Thomas Graves <tgraves@nvidia.com> * cleanup * refactor the separator handling * Update docs Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Add another test for all null col separator * Change to use resource if allowed and clean up scalar separator Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Review comments * rework to remove finally and review comments Signed-off-by: Thomas Graves <tgraves@apache.org> * replace nullable with _ and simplify case * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* start concat_ws Signed-off-by: Thomas Graves <tgraves@apache.org> * more stuff * call into stringConcateWs Signed-off-by: Thomas Graves <tgraves@apache.org> * Fixes and more functionality for concat_ws * Test and debug 1 column Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Handle arrays and add more tests Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Fix type checks * TEsts working and running on gpu * Fix empty str scalar and debug * debug * Update to latest cudf changes, tests all passing Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Try using scalar separator but cudf doesn't handle null scalars the same when its the separator * more changes * handle 1 column with scalar sep * All working with no leaks Signed-off-by: Thomas Graves <tgraves@nvidia.com> * updates * Fix line lengths * reogranize * refactor code to be more readable, all tests pass, no leaks Signed-off-by: Thomas Graves <tgraves@nvidia.com> * cleanup * refactor the separator handling * Update docs Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Add another test for all null col separator * Change to use resource if allowed and clean up scalar separator Signed-off-by: Thomas Graves <tgraves@nvidia.com> * Review comments * rework to remove finally and review comments Signed-off-by: Thomas Graves <tgraves@apache.org> * replace nullable with _ and simplify case * Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala Co-authored-by: Andy Grove <andygrove73@gmail.com> Co-authored-by: Andy Grove <andygrove73@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This depends on CUDF PR: rapidsai/cudf#8289
fixes #63
Note the issue has a bunch of Spark behavior for concat_ws as well as some examples if you want to see them. (#63 (comment)).
You can pay attention to the null handling as that is what we found some incompatibilities with in CUDF.
At a high level we add a new GpuConcatWs expression that supports taking Strings or Arrays or Strings and returns a String column. If only a separator is specified we fall back to the CPU, otherwise as long as 1 other column with the separator we process it. The code handles 2 types of separators, scalar and the spark sql api allows a column of separators. We use the corresponding CUDF api based on the separator passed in. concat_ws also allows an Array of Strings, so the strings within a single Array[String] column have to be concatenated first using a different CUDF api and then those are combined with any other columns to get the final output.
I ran a test on larger customer dataset and it helped mostly due to not going off gpu and then coming back. 80million rows, concatenating 2 sting columns, gpuconcatws took around 200ms.