Scala Native #3057

armanbilge · 2022-06-27T23:35:18Z

Closes #1302.

armanbilge · 2022-06-28T00:57:40Z

std/native/src/main/scala/cats/effect/std/SecureRandomCompanionPlatform.scala

+@extern
+@nowarn
+private[std] object sysrandom {
+  def getentropy(buf: Ptr[Byte], buflen: CSize): Int = extern
+}


This works on Linux and macOS. I reckon that's plenty good for now. Anyone can implement SecureRandom for themselves to suit their requirements.

We definitely need to write up some serious words about the limitations of the current implementation. :-)

Sure. But note that anyone who tries to use this on an unsupported platform will get an error at linking time, before runtime. So it's not evil, just annoying.

armanbilge · 2022-06-28T00:58:58Z

build.sbt

+  .nativeSettings(
+    libraryDependencies += "io.github.cquiroz" %%% "scala-java-time" % "2.4.0"
+  )


I don't think Scala Native has the same artifact size concerns as Scala.js, so I reckon we should ship java.time support out-of-the-box.

armanbilge · 2022-09-04T02:11:36Z

core/native/src/main/scala/cats/effect/IOCompanionPlatform.scala

  def blocking[A](thunk: => A): IO[A] =
-    apply(thunk)
+    // do our best to mitigate blocking
+    IO.cede *> apply(thunk) <* IO.cede


Downstream libraries are riddled with blocking calls (e.g. DNS resolution in ip4s, file I/O in fs2). Actually even IO.println is blocking! Since we don't have a blocking threadpool to run them on, this seems like the best we can do to mitigate their effects.

Actually we can do even better here: this should be apply(thunk).guarantee(IO.cede). We don't want to skip that cede even if we errored.

…tive

Now behaves like `ScheduledExecutorService#setRemoveOnCancelPolicy(true)` used on JVM

This reverts commit 7efbac0.

djspiewak · 2022-09-19T07:06:59Z

So this is all in series/3.3.x now right? How do we untangle the git gordian knot?

armanbilge · 2022-09-19T09:41:04Z

No, this is not really in series/3.3.x. Actually pulling that off would have been too painful because Scala Native requires Scala 3.1+ and the JVM/JS series/3.3.x is stuck on Scala 3.0. Hence my "backport" PR #3138 really started a parallel branch series/3.3.x-native. That branch can simply be abandoned.

Yes, sorry, not ideal. But I wasn't prepared to let this die-on-the-vine like my Env PR from *checks watch* ...

djspiewak · 2022-09-19T11:54:20Z

Yes, sorry, not ideal. But I wasn't prepared to let this die-on-the-vine like my Env PR from checks watch ...

whistles innocently

Either way, given that this has now been released, I think it's best if we get it into series/3.x to form a part of 3.4. :-)

…tive

armanbilge · 2022-09-19T20:10:13Z

Ok, @djspiewak, let's land this one :) tbh I was quite sad to ship this out in 3.3.14 without your review, so I'm looking forward to that 😇

Question: do we want to add a macOS CI job? There a couple places where we directly call native APIs (high-precision time and cryptographically strong random numbers). They are all supported on Linux/macOS, but in theory we should be careful.

djspiewak · 2022-09-19T20:26:04Z

Question: do we want to add a macOS CI job? There a couple places where we directly call native APIs (high-precision time and cryptographically strong random numbers). They are all supported on Linux/macOS, but in theory we should be careful.

I hate to say it, but we should be careful. :-( So that means one more matrix entry.

armanbilge · 2022-09-20T06:18:03Z

Something I haven't done in this PR is setup the test harness so we can run the IOAppSpec with Native executables. It's possible by the same trick we use to run the IOAppSpec for JS. So just a yak I'd be happy to put off 😆

djspiewak

Mostly just worried about the lack of documentation and caveats!

djspiewak · 2022-09-20T17:54:33Z

core/native/src/main/scala/cats/effect/IOApp.scala

+    lazy val keepAlive: IO[Nothing] =
+      IO.sleep(1.hour) >> keepAlive


Do we actually need this with Scala Native?

I believe so! The whole point of this is to keep an IO.never program alive. In the IO.never case the EC queue will be empty. In theory we could just have the main thread go into a permanently parked state in this situation.

But in practice the EC actually has to stop running if its queues are empty so that it can yield to the Scala Native global EC, to make test frameworks work correctly. See also the doc comment on poll(...).

Fascinating. It's identical code but actually handled in an entirely different way.

djspiewak · 2022-09-20T17:56:01Z

core/native/src/main/scala/cats/effect/IOCompanionPlatform.scala

+  def blocking[A](thunk: => A): IO[A] =
+    // do our best to mitigate blocking
+    IO.cede *> apply(thunk).guarantee(IO.cede)


djspiewak · 2022-09-20T20:59:27Z

core/native/src/main/scala/cats/effect/IOApp.scala

+import scala.concurrent.CancellationException
+import scala.concurrent.duration._
+
+/**


Can we write some words in here which clarifies 1) the experimental nature of things, and 2) the limitations imposed by this particular IOApp? It's fair to call out to EpollCat for now since that's the only usable runtime.

Yep can write some doc. Besides the blocking issue I wouldn't really say IOApp is limited. Anything you can do in Scala Native today works fine with this IOApp. If you want to do async I/O with JDK APIs, tough, because they are not implemented anyway, but that's not this IOApps fault.

It's fair to call out to EpollCat for now since that's the only usable runtime.

It's not actually. It's only necessary for fs2.io.net and thus ember or skunk etc.

I have a CurlApp in http4s-curl that's perfectly good if the only non-blocking I/O you need are HTTP requests (this is sufficient to implement a Lambda runtime btw). Also that runtime is theoretically capable of polling on arbitrary file descriptors, but I have not exposed that.

I also just wrote an io_uring-based runtime in armanbilge/fs2-io_uring#2. But that's still a bit premature :)

Finally, see also #3153 (comment) where we discussed some interesting native applications built with vanilla IOApp, including CLIs and an LSP.

djspiewak · 2022-09-20T21:00:11Z

core/native/src/main/scala/cats/effect/tracing/TracingPlatform.scala

+  @nowarn("msg=never used")
+  def calculateTracingEvent(key: Any): TracingEvent = null


This will be fun™…

djspiewak · 2022-09-20T21:03:33Z

std/native/src/main/scala/cats/effect/std/SecureRandomCompanionPlatform.scala

+@extern
+@nowarn
+private[std] object sysrandom {
+  def getentropy(buf: Ptr[Byte], buflen: CSize): Int = extern
+}


We definitely need to write up some serious words about the limitations of the current implementation. :-)

djspiewak · 2022-09-20T21:04:29Z

tests/native/src/test/scala/cats/effect/DetectPlatform.scala

+package cats.effect
+
+trait DetectPlatform {
+  def isWSL: Boolean = System.getProperty("os.version").contains("-WSL")


How are system properties handled on Scala Native?

No idea what this would say for WSL. But I believe it does attempt to populate them as the JVM would.

armanbilge added 25 commits June 18, 2022 13:43

kernel.native

47bf28d

kernel-testkit.native

5096ca3

laws.native

38c3a25

std.native

6f509f2

core.native compiles

d1afc95

example.native

689dcc7

tests.native compiling

f588289

Add missing javalib

ae44a3b

Fix FiberMonitor

c36ad50

Fix file name

9040aa7

Use busy-wait scheduler for tests

2336b7e

fix busy wait scheduler

557da3c

All tests pass

8fe1977

Setup snapshotting

2b96471

Weird scaladoc issue on native

50ff72d

Fix Scala 3 compile

904a40b

Merge branch 'typelevel:series/3.x' into experiment/native

2db9428

Updates

13999c2

Try setting up CI

e063f5c

scalafmt+scalafix

a620bef

Add ciNative alias

280a314

Explicit types for implicits

29df46b

De-boot-leg

d0cd453

Fix MiMa

2e2ae42

Fix workflow

1ff75b5

armanbilge marked this pull request as draft June 27, 2022 23:35

armanbilge mentioned this pull request Jun 27, 2022

Experiment with scala native #1302

Closed

Implement SecureRandom for Native

c1c5b17

armanbilge commented Jun 28, 2022

View reviewed changes

Surround blocking calls with cede

bbb5d0d

armanbilge commented Sep 4, 2022

View reviewed changes

armanbilge added 5 commits September 5, 2022 22:24

Update nowMicros with SN 0.4.7 APIs

8c8ae32

More cleanup for nowMicros

989ad9c

guarantee(cede) after blocking thunk

2b3bc18

Add failing test for SecureRandom#nextInt

ff197d1

Correctly override SecureRandom#next

6ab05af

armanbilge mentioned this pull request Sep 9, 2022

Backport Scala Native support to 3.3.x #3138

Merged

armanbilge added 9 commits September 11, 2022 11:16

Gracefully handle negative sleeps

82b2665

Merge remote-tracking branch 'upstream/series/3.x' into experiment/na…

6075d85

…tive

Add pollEvery parameter for fairness, default 64

7d9b802

try/catch like the WSTP

ea87873

ScheduledTask -> SleepTask

9e27023

Refactor sleep cancelation

39cdd77

Now behaves like `ScheduledExecutorService#setRemoveOnCancelPolicy(true)` used on JVM

Rip out cachedNow stuff

1221506

Revert "Make installGlobal public"

a14be23

This reverts commit 7efbac0.

Fix suspend impl

0f3ea9c

Merge remote-tracking branch 'upstream/series/3.x' into experiment/na…

e442f8f

…tive

Add native+macOS job

989e648

djspiewak added this to the v3.4.0 milestone Sep 20, 2022

djspiewak reviewed Sep 20, 2022

View reviewed changes

armanbilge added 2 commits September 21, 2022 01:16

Add implementation note/limits for SecureRandom

dd56b2a

Document limitations for IOApp on Native

2a3c545

djspiewak merged commit bda6ac6 into typelevel:series/3.x Sep 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala Native #3057

Scala Native #3057

armanbilge commented Jun 27, 2022

armanbilge Jun 28, 2022

djspiewak Sep 20, 2022

armanbilge Sep 20, 2022

armanbilge Jun 28, 2022

armanbilge Sep 4, 2022 •

edited

Loading

armanbilge Sep 6, 2022

djspiewak commented Sep 19, 2022

armanbilge commented Sep 19, 2022 •

edited

Loading

djspiewak commented Sep 19, 2022

armanbilge commented Sep 19, 2022

djspiewak commented Sep 19, 2022

armanbilge commented Sep 20, 2022

djspiewak left a comment

djspiewak Sep 20, 2022

armanbilge Sep 20, 2022

djspiewak Sep 21, 2022

djspiewak Sep 20, 2022

djspiewak Sep 20, 2022

armanbilge Sep 20, 2022 •

edited

Loading

djspiewak Sep 20, 2022

djspiewak Sep 20, 2022

djspiewak Sep 20, 2022

armanbilge Sep 20, 2022

		lazy val keepAlive: IO[Nothing] =
		IO.sleep(1.hour) >> keepAlive

		@nowarn("msg=never used")
		def calculateTracingEvent(key: Any): TracingEvent = null

Scala Native #3057

Scala Native #3057

Conversation

armanbilge commented Jun 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Sep 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djspiewak commented Sep 19, 2022

armanbilge commented Sep 19, 2022 • edited Loading

djspiewak commented Sep 19, 2022

armanbilge commented Sep 19, 2022

djspiewak commented Sep 19, 2022

armanbilge commented Sep 20, 2022

djspiewak left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armanbilge Sep 4, 2022 •

edited

Loading

armanbilge commented Sep 19, 2022 •

edited

Loading

armanbilge Sep 20, 2022 •

edited

Loading