-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-12404] [SQL] Ensure objects passed to StaticInvoke is Serializable #10357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Test build #47922 has finished for PR 10357 at commit
|
|
retest this please. |
|
Test build #47923 has finished for PR 10357 at commit
|
|
Regression test please, for any patch that includes a bug fix. |
|
So we can avoid delay cutting RC4, I've opened #10360 with tests added. |
|
Test build #47947 has finished for PR 10357 at commit
|
|
Test build #47948 has finished for PR 10357 at commit
|
|
Test build #47954 has finished for PR 10357 at commit
|
|
Thanks, merging to master and 1.6 |
Now `StaticInvoke` receives `Any` as a object and `StaticInvoke` can be serialized but sometimes the object passed is not serializable. For example, following code raises Exception because `RowEncoder#extractorsFor` invoked indirectly makes `StaticInvoke`. ``` case class TimestampContainer(timestamp: java.sql.Timestamp) val rdd = sc.parallelize(1 to 2).map(_ => TimestampContainer(System.currentTimeMillis)) val df = rdd.toDF val ds = df.as[TimestampContainer] val rdd2 = ds.rdd <----------------- invokes extractorsFor indirectory ``` I'll add test cases. Author: Kousuke Saruta <sarutak@oss.nttdata.co.jp> Author: Michael Armbrust <michael@databricks.com> Closes #10357 from sarutak/SPARK-12404. (cherry picked from commit 6eba655) Signed-off-by: Michael Armbrust <michael@databricks.com>
Now
StaticInvokereceivesAnyas a object andStaticInvokecan be serialized but sometimes the object passed is not serializable.For example, following code raises Exception because
RowEncoder#extractorsForinvoked indirectly makesStaticInvoke.I'll add test cases.