Scala.JS build with FS2?

[fiadliel]

What do you think of having a (possibly experimental) Scala.JS build of FS2?

With Node optimization, rough ball-park speed reduction seems around 20x compared with running on the JVM :) But it does work (mostly).

Due to the lack of async support in scalacheck, it seems somewhat difficult to do any async testing, but I got the pure tests working using a non-task, non-async, Trampoline-based monad.

Task itself works in a browser/JS context, as long as you avoid run/attemptRun/etc., for obvious reasons.

[pchlupacek]

@fiadliel can you perhaps try performance of processing with and w/o fs2 on node.js before we decide? I just tried that about like 6 months ago with 0.7 and performance was so poor that we had decided to trash idea. Can you do tests with fs2._ now?

The idea is to compare js/js benchmark instead of js/jvm benchmark :-)

[fiadliel]

Sure! I'll have a look at which tests either are, or can be made to be equivalent.

Having some performance tests that have approximately the same effects/results across 0.8 and 0.9 would be useful for comparisons even without considering the JS aspect.

[japgolly]

On the topic of (Scala.)JS benchmarking, I have a library scalajs-benchmark (live demo) which you might find useful.

[ghost]

A big 👍 if this could work on scala.js! Main reason would be that it would be safe for many libraries to use FS2 knowing that it won't break the stack re multi-platform.

Out of curiosity, how does fs2 work on scala.js without multi-threading, etc? Or perhaps rather, what won't work?

[pchlupacek]

@inthenow hence we mostly use asynchronous code for concurrency, this will work in scala.js out of the box, but, obviously not concurrently, as js does not have multi-threading model.

[japgolly]

If you're interested you might be able to cook something up with WebWorkers. That'll give you real threads in JS albeit with limited access.

[fiadliel]

Sorry, Christmas/New Year was a somewhat extended downtime this year. But I'll get back to this over the next week.

@inthenow you can't do anything asynchronously, and then wait for results - it has to be a callback. So fs2.util.Task.run will not run (cannot compile), but .runAsync is fine.

Everything else from Task should work, but long-running computations without any callbacks (e.g. lots of repeated Task.delay/Task.now) would, I think, hog the JS thread.

The specifics are down to what monadic effect is being used; I could imagine something other than fs2.util.Task that is specialised for the JS environment (including, perhaps, WebWorker support), but something that works on both the JVM and JS is quite convenient.

[japgolly]

FYI scala-js/scala-js#186

[ghost]

Thanks guys....I'm going to cut straight to the chase, as there is some more background:

If you look at https://github.com/non/cats/issues/32 and typelevel/general#4, the general question/issues/problems of having a cool Task that works on JS/JVM is not new. As @japgolly points out, it's Await.result that's the killer, and what seems to work well on JS seems to upscale badly onto JVM, and what works on JVM seems to downscale badly onto JS. (eg on JS everything ends up synchronized, WebWorkers are more like processes so useful for microservices such as an email server, etc)

I'm perhaps beginning to think that a single api will never work...but I would love to be wrong.

Anyway I don't want to suggest that cats/monifu/fs2 "merge" any work, but there's scope here for some collaboration. What do you think?

CC: @stew, @tpolecat, @alexandru, @ceedubs, @milessabin + anyone else....

[alexandru]

Had been CC-ed, got email :-)

If you're interested in my Task implementation (see design document) which was built with Scala.js in mind, then do note that I'm thinking about splitting the project (monifu/monix) in a way that makes sense, such that one can import only the Task and its dependencies. The design/changes are not final, but the target for the new version is the beginning of March.

[pchiusano]

I'm not really sure what the current status of this, but I'd like to punt on it for now. We can revisit after 0.9 is out. Assigning the 'Later' milestone.

[mdedetrich]

The only real things you need to know regarding Scala.js and Async is that you cant wait on a Future, i.e. there is no Await.result, and there is only "one thread" which you have no control over. The former may have an issue on some things, as @fiadliel mentioned earlier but the latter should only have issues in regards to performance.

There are some hacks for Await.result, but none of them are entirely correct (they don't follow the same semantics as the JVM version). This is the reason why Await.result doesn't exist in Scala.js.

[alexandru]

Personally I think that whenever you've got a conversion like Future[A] => A, it always relies on platform specific tricks that aren't portable (threads blocked, CPS, etc.) and that ends up backfiring. Even on the JVM, when blocking a thread you need to know whether the underlying thread-pool supports it, as with a max-sized thread-pool you can end up blocking all threads and have your application freeze because there are no threads left for progress to happen.

And on the Javascript side, even with web-workers, you lack the semantics of AtomicReference.compareAndSet or of wait/notify sharable between workers, which means synchronization would be very expensive and absolutely awful. Basically Await.result is not possible on Scala.js.

[tpolecat]

Is anyone actually arguing that blocking is a fundamental operation? If you have .unsafeRunAsync(f: Throwable \/ A => Unit) you can build a trap to make it synchronous on platforms that support it, but it's fine with me to say you're on your own.

[fiadliel]

Blocking is part of the API currently, but only the JVM is officially supported. I think it would be untidy to have methods which you realize don't work on JS because there are linkage errors. So I think the sync calls should be separated out somehow (whether via a comonad or otherwise).

[mpilquist]

I'm not really keen on the idea of separating the blocking API from the non-blocking API. Seems like we'd be making the by far common use case of running FS2 on the JVM worse to support Scala.js.

[mdedetrich]

@alexandru

Personally I think that whenever you've got a conversion like Future[A] => A, it always relies on platform specific tricks that aren't portable (threads blocked, CPS, etc.) and that ends up backfiring. Even on the JVM, when blocking a thread you need to know whether the underlying thread-pool supports it, as with a max-sized thread-pool you can end up blocking all threads and have your application freeze because there are no threads left for progress to happen.

Yeah I think this is true, per say. Any reasonable platform with a detailed bytecode can implement Await.result (even if it doesn't support multiple threads). Its just that since Javascript isn't really a bytecode, but an actual language, you get limited in a lot of ways (another good example is not having a Char in Javascript which also effects Scala.js). The not having multiple threads isn't really a big issue, as you can just use a global threadpool which has a fixed thread size of one.

So at least personally, its more of a problem with Scala.js (due to Javascript being Javascript) rather than the merit of Await.result itself. Although Await.result does have issues, it is needed in certain rare situations (one that actually always comes up a lot is in testing)

[ghost]

There is a hack for Await.result in scala.js, no idea if it is useful here or not. It's due to moved out of cats, but for the moment see this.

Used with an " execute immediate context" that used to be the default for scala.js, this "works" as the underlying future must have completed to get to the await. Of course, there is no guarantee that it completes before the await but that's why it a hack ;)

[ghost]

Right. This is a bit whacky, but here goes:

If await.result is the big killer for scala.js, why not use the traceur-compiler that has async support

[alexandru]

@inthenow because it's not going to work for the purposes of Await.result. Scala also has async support. Amongst the limitations is the inability to await inside a catch clause or inside of anonymous functions / closures. The catch clause limitation can probably be resolved (the .NET / C# folks eventually did it), but you can't await in a closure, because that closure can then be executed either synchronously or asynchronously or never, so the compiler can't know what to make of it and wouldn't be able to transform it in a state machine.

For a Task implementation there is a legitimate case for computations that can be evaluated synchronously, or that have been evaluated already. The approach I'm taking is to try running it and return either a value or a future.

def coeval[A](task: Task[A])(implicit s: Scheduler): Coeval[Either[CancelableFuture[A], A]] = ???

[ghost]

@alexandru A while after posting the comment, I went through all the ECMA docs and came to the general conclusion that it wouldn't help either 😒

[fiadliel]

I don't think this is a big problem -- I would not expect many pipelines using an IO-like effect type to be shareable between JS and JVM (the available libraries, expected functionality, etc.) are completely different.

People on JS just have to avoid a small set of methods, which wouldn't link anyway.

Fundamentally, once a possibly asynchronous operation is actually asynchronous, we can't both handle the callback, and wait for the result, with just a single thread - impossible. And in the case of Task, the type does not (currently) represent whether the operation is async, so we can't track whether it's possible or not based on the type.

Since we can't do impossible things, I think the biggest part of the job here to have something useful (if not tidy) is to choose a test framework which actually lets us run tests asynchronously. uTest looks interesting, but it doesn't have any support for scalacheck (yet).

[mpilquist]

ScalaTest has Scala.js support as of 3.0. I use it with Scalacheck on scodec.

[alexandru]

@fiadliel, folks, sorry for the spam. I like thinking about this stuff and this is a really narrow domain that other people aren't interested in :-) What follows is a brain dump :-P

For testing Monix I've built Minitest which does have both integration with ScalaCheck and support for testing asynchronous computations working in Scala.js, something I believe ScalaTest still doesn't do. It's very light, which is a feature I think.

Of course, in Monix I haven't really needed actual asynchronicity, because time itself is provided by the Scheduler and so you can fake it, non-determinism included.

And in the case of Task, the type does not (currently) represent whether the operation is async, so we can't track whether it's possible or not based on the type.

The problem is that Task is being used with a mixed personality. But I think Task should be considered asynchronous, always, even if as an optimization it might evaluate synchronously on the current thread. In Monix's Task implementation the only way to get a result out is execute runAsync(scheduler) on it. It has no run: A. Even more, asynchronous computations start things like registering runnable in a thread-pool, maybe opening file-handles and so on, so for me it's important for them to provide on execution a way to cancel things. Because of this reason, my chooseFirstOf implementation is actually safe, see how the timeout operator is implemented.

The mistake, in my opinion, is that people think of Task as being a sort of IO type. Which means computations that are being executed on a trampoline, available immediately because of the signature of unsafePerformIO. Except that Task is more than a trampoline, because it always has some sort of Async state which can only be completed by calling a provided callback. If you think in terms of signatures it's () => A versus A => (). It's almost as if async is the dual of sync :-) And a trampoline cannot complete with the result of a computation ending with a callback, unless it blocks for that result.

Blocking for a result is of course bad, because it's unsafe, as blocking, just like intrinsic locks, are breaking the encapsulation of the underlying platform. The Internet is filled with the misery of people experiencing deadlocks or sudden slowdowns and not knowing what's going on. Did you know that Scala's own global has an absolute upper limit which means it's totally possible for the application to freeze with perfectly "valid" code?

Back to Monix's Task, you can still block for the result, but you'd have to do Await.result(task.runAsync, 1.second) on it, piggybacking on Scala's standard library and acknowledging the fact that the current thread is getting blocked, with a mandatory timeout, without which blocking would be totally insane. Or in case you know that the Task should more or less execute immediately, then you can transform it into a Coeval[A \/ CancelableFuture[A]].

In Monix a Coeval is the Task-like type that always executes synchronously, as an added restriction to Task. Sort of like the Cats Eval, though I took error handling more seriously. I guess you could say that Coeval is the IO type. And if you're inclined, you can abstract over both because they are Evaluable instances. In Monix, all the exposed types Coeval, Task and Observable are shareable between the JVM and JS, with no exceptions. And I think that all of them are very useful, especially on top of Javascript where otherwise you'd have to deal with callback-hell.

[mpilquist]

I started this work on the topic/sjs branch. No promises at this point, but I'm going to try to get this in before 0.9.0 final.

[mpilquist]

Quick progress update on the topic/sjs branch:

• The toList and toVector methods on pure streams were internally implemented by lifting the stream to a Stream[Task,A] and then calling runLog. This is problematic in SJS because we can't block for a result so I introduced a Catchable[Either[Throwable,?]] instance and changed toList/toVector to evaluate in that effect instead.
• I added some new run methods to Task that are inspired by Monix's approach -- unsafeRunSync, unsafeAttemptRunSync, unsafeValue, and unsafeAttemptValue. These all run a Task up until the first async boundary, making them safe to be called from Scala.js code.
• I moved the various blocking run methods on Task to a JVM-specific syntax extension -- unsafeRun, unsafeAttemptRun, unsafeRunFor, and unsafeAttemptRunFor. This is a source compatible change but has the benefit of creating a compilation error if these methods are called from a Scala.js build.
• I opted to leave the fs2.internal.Future class alone instead of bifurcating the code in to JVM-safe / JS-safe / both-safe. I figure that since this type is private to fs2, any Scala.js linkage errors should be found by our test suite, and hence, spreading its implementation across 3 source files didn't have a good payoff.
• I added Scala.js implementations of Strategy and Scheduler along with implicit defaults. I think this is safe because there's really only 1 reasonable implementation of each of these interfaces in Scala.js. Correct me if I'm wrong please -- if so, we can drop the implicit modified from these instances.

At this point, pure streams and basic effectful streams appear to be working fine under Scala.js.

I'm going to start porting the core test suite over next. This will take a while, because all the tests will need to be changed to return a Future instead of Unit. This should expose other places we are using java.util.concurrent or other blocking constructs.