-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-34059][SQL][CORE] Use for/foreach rather than map to make sure execute it eagerly #31110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
cc @srowen FYI |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
This comment has been minimized.
This comment has been minimized.
f3e864e to
74bc183
Compare
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #133890 has finished for PR 31110 at commit
|
|
Thanks! Merging to master. |
|
Thanks for fixing this @HyukjinKwon ! Surprised these did not cause problems earlier. |
|
Sure I'll prepare backport |
|
It will definitely have the same effect if the collection is not lazy, so I doubt it currently is the cause of a bug. But yeah it'd be fine to backport in case there is any chance the collection that something returns suddenly gets lazy. |
… sure execute it eagerly ### What changes were proposed in this pull request? This is a backport of #31110. I ran intelliJ inspection again in this branch. This PR is basically a followup of #14332. Calling `map` alone might leave it not executed due to lazy evaluation, e.g.) ``` scala> val foo = Seq(1,2,3) foo: Seq[Int] = List(1, 2, 3) scala> foo.map(println) 1 2 3 res0: Seq[Unit] = List((), (), ()) scala> foo.view.map(println) res1: scala.collection.SeqView[Unit,Seq[_]] = SeqViewM(...) scala> foo.view.foreach(println) 1 2 3 ``` We should better use `foreach` to make sure it's executed where the output is unused or `Unit`. ### Why are the changes needed? To prevent the potential issues by not executing `map`. ### Does this PR introduce _any_ user-facing change? No, the current codes look not causing any problem for now. ### How was this patch tested? I found these item by running IntelliJ inspection, double checked one by one, and fixed them. These should be all instances across the codebase ideally. Closes #31137 from HyukjinKwon/SPARK-34059-3.1. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
… sure execute it eagerly ### What changes were proposed in this pull request? This is a backport of #31110. I ran intelliJ inspection again in this branch. This PR is basically a followup of #14332. Calling `map` alone might leave it not executed due to lazy evaluation, e.g.) ``` scala> val foo = Seq(1,2,3) foo: Seq[Int] = List(1, 2, 3) scala> foo.map(println) 1 2 3 res0: Seq[Unit] = List((), (), ()) scala> foo.view.map(println) res1: scala.collection.SeqView[Unit,Seq[_]] = SeqViewM(...) scala> foo.view.foreach(println) 1 2 3 ``` We should better use `foreach` to make sure it's executed where the output is unused or `Unit`. ### Why are the changes needed? To prevent the potential issues by not executing `map`. ### Does this PR introduce _any_ user-facing change? No, the current codes look not causing any problem for now. ### How was this patch tested? I found these item by running IntelliJ inspection, double checked one by one, and fixed them. These should be all instances across the codebase ideally. Closes #31138 from HyukjinKwon/SPARK-34059-3.0. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
… sure execute it eagerly ### What changes were proposed in this pull request? This is a backport of #31110. I ran intelliJ inspection again in this branch. This PR is basically a followup of #14332. Calling `map` alone might leave it not executed due to lazy evaluation, e.g.) ``` scala> val foo = Seq(1,2,3) foo: Seq[Int] = List(1, 2, 3) scala> foo.map(println) 1 2 3 res0: Seq[Unit] = List((), (), ()) scala> foo.view.map(println) res1: scala.collection.SeqView[Unit,Seq[_]] = SeqViewM(...) scala> foo.view.foreach(println) 1 2 3 ``` We should better use `foreach` to make sure it's executed where the output is unused or `Unit`. ### Why are the changes needed? To prevent the potential issues by not executing `map`. ### Does this PR introduce _any_ user-facing change? No, the current codes look not causing any problem for now. ### How was this patch tested? I found these item by running IntelliJ inspection, double checked one by one, and fixed them. These should be all instances across the codebase ideally. Closes #31139 from HyukjinKwon/SPARK-34059-2.4. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>
What changes were proposed in this pull request?
This PR is basically a followup of #14332.
Calling
mapalone might leave it not executed due to lazy evaluation, e.g.)We should better use
foreachto make sure it's executed where the output is unused orUnit.Why are the changes needed?
To prevent the potential issues by not executing
map.Does this PR introduce any user-facing change?
No, the current codes look not causing any problem for now.
How was this patch tested?
I found these item by running IntelliJ inspection, double checked one by one, and fixed them. These should be all instances across the codebase ideally.