Cross-build http4s for Scala.js

[armanbilge]

tl;dr try it out with

val http4sV = "1.0-377-020cf9e"
libraryDependencies ++= Seq(
  // Node.js servers, not for serverless
  "com.armanbilge" %%% "http4s-ember-server" % http4sV,

  // Any Node.js environment, including servers, AWS Lambda, and CLI app
  "com.armanbilge" %%% "http4s-ember-client" % http4sV,

  // Any Node.js serverless environment, such as AWS Lambda or Google Functions
  "com.armanbilge" %%% "http4s-node-serverless" % http4sV

  // Browsers and also Cloudflare Workers
  "com.armanbilge" %%% "http4s-dom-fetch-client" % http4sV,

  // Browsers supporting service workers and also Cloudflare Workers
  "com.armanbilge" %%% "http4s-dom-service-worker" % http4sV,

  // any other modules from the list below :)
)

This PR introduces cross-builds to Scala.js for several of the core http4s modules per #956:

• core
• laws
• testing
• tests
• server
• client
• ember-core
• ember-server
• ember-client
• node-serverless new!
• dom-core new!
• dom-fetch-client new!
• dom-service-worker new!
• dsl
• boopickle
• jawn
• circe

It's still a bit rough around the edges, but very nearly there I think (tests are passing on my machine). The main gaps for Scala.js are file- and crypto-related stuff, not sure how important these are.

At this time, there is no server or client implementation for Scala.js but I think that is best done in follow-up work :) I've implemented a fetch-based client for browsers, and ember is on the way via here! typelevel/fs2#2453.

Admin stuff: this is my first PR to this repo so that means every commit I push will have to be manually approved for CI, which makes it tricky to work independently. Could I possibly get manually whitelisted, or alternatively if there's another small PR I can get merged right away (e.g., fix typo etc.) that would also get me on the whitelist. Thanks!

[armanbilge]

All tests are passing!! Besides the occasional hiccup with org.http4s.server.middleware.ThrottleSuite which seems flaky on Firefox. I updated the test matrix following after CE3's, which tests the Scala.js modules on both Node.js and Firefox.

Now, I seek guidance on a couple remaining issues so we can hopefully get this merged :) Some TODO's I scattered in the code:

• CirceInstancesPlatform for JS: the CirceInstances API offers several methods jsonDecoderIncremental, jsonDecoderAdaptive, streamJsonArrayDecoder, jsonDecoderByteBufferImpl. To keep parity across JVM and JS I implemented all these methods for JS, however they're just using Javascript's String-based JSON parser without doing anything optimizations as suggested by the method names. Ideas:

  1. Leave as-is.
  2. Drop API-parity, and offer only one (or two) honestly-named methods for JS.
  3. Try to improve the implementations to match the method names. ✅

• Crypto for JS: oauth1, Websockets, CSRF all rely on crypto functions not available in Scala.js. Ideas:

  1. Drop these for JS.
  2. Implement with the Web Crypto API also experimentally available for Node.js. ✅
  3. Create several implementations using JS-platform-specific APIs for web, Node.js, etc. Provide these as separate modules or maybe try to select dynamically at runtime based on what's available.
  4. Find/roll-our-own crypto implementations in pure Scala??? 😬

A couple other more minor things:

• No file system stuff for JS, but we could support this with a separate module targeting Node.js.
• There's something strange with floats on Scala.js (try 1.2f.toString and count the decimal places!) so I disabled these tests on JS.
• DecodeSpec uses a bunch of weird charsets not available on JS, not sure if these are critical to the tests/can be rewritten.
• I think I messed up the sbt files by running scalafmtSbt and also a bunch of file headers. Big deal?

Thanks in advance for your feedback/reviews!!

[armanbilge]

After chatting with @ChristopherDavenport on Discord, my understanding is the thinking is to prioritize support for browser-based users without locking out other JS platforms. So, I'm going to use the Web Crypto APIs to plug the crypto hole for now, and we can hopefully take advantage of its experimental presence in Node.js and/or provide a fallback implementation at a later point.

[armanbilge]

@rossabaker I think I've addressed most of your review comments, let me know if I've missed anything. I think I restored all of the headers and I did my best undoing at least some of the build formating. See my comments above re the Node.js/python server scripts and the crypto library.

We'll be getting fs2 3.1 very soon which will let me make a final pass of cleanups, it doesn't necessarily have to be this PR.

[rossabaker]

I wonder whether we even need this hack if sbt-header is leaving the attributions alone now? That would be lovely, because you just experienced firsthand how fragile this is.

[armanbilge]

We don't (assuming you are happy with the dual-headers), I can remove it.

[rossabaker]

Yeah, what you did looks great. People have tried to tear this project down for stealing work over the years, but I don't think many projects go to our lengths to credit their sources. Thanks for persevering on this to keep that tradition alive.

[armanbilge]

I had no idea about that history, really sad to hear because it's readily apparent you've gone above and beyond here.

[rossabaker]

I'd say privatize whatever you see fit in the JS package, and then we'll land this and cut a milestone.

[armanbilge]

Hooray!! I'll take a careful pass through the JS package and let you know when it's ready.

then we'll land this and cut a milestone.

Do you still want to wait for a proper scala 3 artifact for scala-js-dom before the milestone?

[armanbilge]

Actually, I have an idea: scala-js-dom (where I got the JS crypto stuff from) is hopefully releasing their 2.0 with Scala 3 artifacts very soon. Maybe I can get in a PR to fix the issue that prevented using their facades in Node.js and forced me to copy their code into http4s. That would be a huge win.

Edit: on the other hand, that would still be adding a dom dependency to http4s core ... which is just weird. This is a tricky one.

[rossabaker]

I don't want to keep this PR open longer than necessary. I would exclude the DOM packages from publishing on Scala 3 for this round, and we can reintroduce later.

I don't think it's terrible if that DOM dependency in core is JS only, if it's necessary to make the core run on JS? Maybe I'm misunderstanding.

[rossabaker]

Getting a ton of warnings like this. Maybe we're stuck with them:

[warn] /Users/ross.baker/src/http4s/core/js/src/main/scala/org/http4s/js/crypto/Crypto.scala:863:42: dead code following this construct
[warn]   var hash: HashAlgorithmIdentifier = js.native

[armanbilge]

I think 2.13 is better with this, because I tried @nowarning but then 2.13 started complaining there was nothing to @nowarn. This could probably be solved with a magic incantation of sbt settings.

[rossabaker]

I'm not terribly worried about it, but there's enough scala.JS only code, it would be a nice mopup later so we don't miss bugs there by tolerating warnings.

[armanbilge]

I don't want to keep this PR open longer than necessary. I would exclude the DOM packages from publishing on Scala 3 for this round, and we can reintroduce later.

Me neither, let's close this out. So it's okay if we do a breaking bump for scala-js-dom from 1.0 to 2.0 for the next milestone?

I don't think it's terrible if that DOM dependency in core is JS only, if it's necessary to make the core run on JS? Maybe I'm misunderstanding.

Yes, just talking JS here. A DOM (browser) dependency for core is weird because core also runs on non-browser environments such as Node.js. So it should be agnostic.

[rossabaker]

I took the liberty of drafting a changelog here. My JavaScript and frontend knowledge is about 15 years out of date, so please don't hesitate to correct anything silly that I said.

So it's okay if we do a breaking bump for scala-js-dom from 1.0 to 2.0 for the next milestone?

For sure. 1.0 is very much under development.

A DOM (browser) dependency for core is weird because core also runs on non-browser environments such as Node.js. So it should be agnostic.

And it gets pulled into http4s-core because it provides the JS crypto, otherwise it would be in http4s-dom-core? Seems like maybe that scala-js-dom is missing some modularity?

[armanbilge]

I took the liberty of drafting a changelog here. My JavaScript and frontend knowledge is about 15 years out of date, so please don't hesitate to correct anything silly that I said.

I took a brief look, looks good! 😄

For sure. 1.0 is very much under development.

Then FWIW, there's no harm to release Scala 3 artifacts using for3use2_13 for scala-js-dom, because there will never be a scala-js-dom v1.x with a Scala 3 artifact. When we get the Scala 3 artifact we will also be bumping up to 2.0 anyway. Totally up to you.

Seems like maybe that scala-js-dom is missing some modularity

I'd say it's a Scala.js problem as a whole, because they are trying to use a single artifact suffix _sjs to support two very different platforms (browsers and Node.js).

I get the feeling your itching to release, say tomorrow? I'll wrap up everything that's left tonight.

[rossabaker]

there's no harm to release Scala 3 artifacts using for3use2_13 for scala-js-dom, because there will never be a scala-js-dom v1.x with a Scala 3 artifact.

"Never" is a strong word, and I'd rather not normalize for3use2_13 in published dependencies. We'll have a proper fix soon enough.

I'd say it's a Scala.js problem as a whole, because they are trying to use a single artifact suffix _sjs to support two very different platforms (browsers and Node.js).

I'm not sure the solution to that problem is copying massive amounts of code. What is the real downside to having all this DOM support in an http4s-core for Node.js? I don't have a good grasp of what the optimizing compiler does for us here.

I get the feeling your itching to release, say tomorrow? I'll wrap up everything that's left tonight.

Would love to merge at least. Release would be great. It would be a bit cleaner if we had an fs2 milestone to keep like qualifiers with our dependencies. But not strictly necessary, I guess. /cc @mpilquist

[armanbilge]

"Never" is a strong word

I can say so with certainty because of scala-js/scala-js-dom#451. Basically, the for3use2_13 has been the designated version for Scala 3, which is why they are forced to do a major version bump to release a real Scala 3 artifact. But, I agree better not to normalize this practice!!

What is the real downside to having all this DOM support in an http4s-core for Node.js?

The risk is that that library evolves in ways that strongly diverge from our needs. For example, attempting to use their crypto facade on a non-DOM environment i.e. Node.js is currently throwing a fatal exception, because of a package object initializer that assumes DOM stuff. This is why I copy-pasted the code in the first place. Theoretically, we can try and patch these (and pray that they accept our PRs) but at the end of the day our goals are just not well-aligned.

After crossing many many projects, I do think we need a cross-platform crypto library that supports browsers, Node.js, and maybe even JVM. Http4s, Skunk, and scodec all have this common need.

Would love to merge at least. Release would be great. It would be a bit cleaner if we had an fs2 milestone to keep like qualifiers with our dependencies.

Yes, let's merge today please. I would also prefer to wait for fs2 to release, but I think that's still be going to be a little while.

Last night and one more time this morning I took a pass through the JS cryptography facade. Unfortunately, I don't think we can make much of it private without being surgical. CryptoKey is the essential public interface there, but for users to obtain a CryptoKey we have to provide the algorithms and the SubtleCrypto API. Hope that's okay.

I just pushed a fix for the build but otherwise I'm 👍 to merge. Thanks!

[mpilquist]

Hoping for fs2 3.1.0 release tonight

[rossabaker]

Okay. I can start a release train this afternoon then, since this will come third.

Before that happens, someone needs to explain the security alerts to me like I'm a dumb backend developer: #5031