You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some versions of spark-testing-base are not compatible with sscheck, resulting in non serializable exceptions for tests using checkpointing. The problem is that some versions of spark-testing-base use a version of "io.github.nicolasstucki" %% "multisets" defining an implicit value config that causes this serialization problem (see e.g. https://github.com/holdenk/spark-testing-base/blob/7d4077efacd5ed454aededcf9278ddd7953dad80/src/main/1.3/scala/com/holdenkarau/spark/testing/JavaStreamingSuitebase.scala). This is weird because we only use com.holdenkarau.spark.testing.TestInputStream as a transient val, maybe it has something to do with the fact that that value is implicit.
Anyway a simple solution is copying the classes to this project, thus removing the dependency and effectively freezing it
The text was updated successfully, but these errors were encountered:
That variable config should be transient. The multisets library has known problems with serialization nicolasstucki/multisets#9, Note Batch.toBag and PDStream.toBagSeq are not used anyway, so I'll remove them too
Some versions of spark-testing-base are not compatible with sscheck, resulting in non serializable exceptions for tests using checkpointing. The problem is that some versions of spark-testing-base use a version of "io.github.nicolasstucki" %% "multisets" defining an implicit value
config
that causes this serialization problem (see e.g. https://github.com/holdenk/spark-testing-base/blob/7d4077efacd5ed454aededcf9278ddd7953dad80/src/main/1.3/scala/com/holdenkarau/spark/testing/JavaStreamingSuitebase.scala). This is weird because we only usecom.holdenkarau.spark.testing.TestInputStream
as a transient val, maybe it has something to do with the fact that that value is implicit.Anyway a simple solution is copying the classes to this project, thus removing the dependency and effectively freezing it
The text was updated successfully, but these errors were encountered: