Make IO serializable

[schernichkin]

Sometimes it's useful to serialize IO action and pass it over wire. Monix and ZIO support that by extending Serializable trait, cats-effect - not.

[djspiewak]

Oy, I hate Serializable. You're right that this is probably important though, particularly for Spark

[schernichkin]

Oy, I hate Serializable.

Me too, but this is how Scala implemented:) In fact the only reason why I'm currently using Monix in my project is because Monix's Task is serializable.

[schernichkin]

I investigated this problem in more detail, looks like it can be solved without redesign of IO. Currently IO instances capture ContextShift[IO] and hence Scala's ExecutionContext, which is not serializable.

Monix do not suffer from this problem, because Task instances do not require ExecutionContext on creation site. Instead Monix's Scheduler (which is analog of ContextShift in some sense) passed to run... functions.

I find Monix's approach more appropriate, because calculation is separated from execution context, it can be serialized, passed over passed over wire and executed one or more times in different context (with different schedulers). However implementing this approach in cats-effect will lead to major redesign of public API and internal structure.

This is definitely not "good first issue". It should be done by someone familiar with cats-effect architecture, someone who can assess consequences of such changes.

[SystemFw]

@schernichkin the approach you describe is roughly what we plan to do for IO in 3.0 (not because of serialisable, but because of many other reasons).

---

On another note, I do wonder how well does serialisation of Task work in practice since you have lambdas in FlatMap nodes that can close over arbitrary things

[schernichkin]

@SystemFw Scala supports lambda serialization out of the box, this is how Spark works. Obviously, you should be extremely careful not to capture something what can't be serialized or something too big (quite common mistake made by novice Spark programmers).

I met two more approaches: "static closures" which used in distributed Haskell and require language support and "recorder monad" which allow to store intermediate values in log, serialize it and replay it later on remote machine. This approach was taken in transient and it does not require specific language features aside of serialization.

However I think using Serializable and build-in closure serialization is more Scala-way, it requires less effort and relays on already implemented industry-strength serialization libraries.

[SystemFw]

I met two more approaches: "static closures" which used in distributed Haskell and require language support and "recorder monad" which allow to store intermediate values in log, serialize it and replay it later on remote machine. This approach was taken in transient and it does not require specific language features aside of serialization.

Yeah, Cloud Haskell and transient are the approaches I'm more familiar with (Quasar also takes a clever approach)

Scala supports lambda serialization out of the box, this is how Spark works. Obviously, you should be extremely careful not to capture something what can't be serialized

Sure, my question wasn't theoretical, but just like, in practice, is this actually workable for something like IO? (completely opaque, you might get one that closes over something like a connection from a library and you don't even know). Anyway, I'm not opposed to this change so it's fine

[djspiewak]

Sure, my question wasn't theoretical, but just like, in practice, is this actually workable for something like IO?

In some cases, not in others. Basically just like serialization in general. :-/ I don't see any particular harm in making IO serialization work in all cases that general closure serialization works. That's basically all we can do anyway, and it makes cats-effect much more Spark-friendly.

I investigated this problem in more detail, looks like it can be solved without redesign of IO. Currently IO instances capture ContextShift[IO] and hence Scala's ExecutionContext, which is not serializable.

Ooooooh, I forgot we close over ContextShift instances. You're right, this is far from a good first issue. This is actually a redesign of a component of the run loop. We already have this planned for CE 3.0, but we can't do it until then because it's a major change to not only the public API, but also the runtime guarantees.

[schernichkin]

@SystemFw Btw which approach Quasar takes? Is where any articles describing it? Currently I'm prototyping framework for writing applications for distributed graph processing. It relies on the idea of strong mobility (particularly has moveTo operation which can transfer running process to remote node), run on Akka cluster and uses ContT transformer to represent distributed process. But I also interested in other approaches.

[kubukoz]

@djspiewak 👀 👀 👀

[Daenyth]

I'd love to see this as a goal for CE3

[djspiewak]

This should be doable for ce3. EvalOn nodes are unlikely to properly serialize, but as long as you don't touch them, everything else should be okay.