Requirements for Building a new Streaming JavaScript Environment (Node.js alternatives)

[sebmarkbage]

I just wanted to drop some notes for environments implementing streaming other than Node.js (like CloudFlare Workers, Deno or custom middleware on top of Node.js). We do support Web Streams but I wouldn't consider that implementation by itself "production ready". Only the Node.js environment - without any unknown proxies. Web Streams - by themselves - lack a few capabilities that you may or may not be able to work around in your environment.

This is intended more as a guide for implementators of new JavaScript environments. The same applies for both Streaming SSR and Server Components.

The way we use Node.js is unconventional and based on a subtle balanced set of using the API that might not be obvious from reading the code. There are a few requirements that are not commonly required for other streaming use cases that now become much more important. They basically boil down to that different bytes in the stream have different priority and don't always render in the same order.

Backpressure

Properly managing backpressure becomes important for reaching optimal performance. Backpressure can be seen as purely a CPU based optimization and where you don't have any such constraints you might be compelled to not apply as much backpressure. This can create suboptimal behavior for a React stream.

The way React streaming works is that we compute what we can and generate a priority queue. We then write the highest priority content we have ready at the time in order of priority. Then we try to compute some more and then we do the whole thing again.

For example, we try to compute A, B and C which has highest, medium and lowest priority. Let's say that we're able to compute A and C but B is still suspended on more data needing to load. Then we'll write A since it's the highest priority. If we see that we don't have any backpressure we'll ALSO write C at this point. That way we can utilize the idle time on the network to send some more bytes down. Then when we finish B later we can send those down the network.

If those bytes actually was handed off to the network layer and it was able to send it down the wire, this can be a really good way to utilize the available bandwidth instead of letting it sit idle.

However, if there was something else like a middleware or proxy down the wire that was too busy to handle it, it might get buffered instead. Then it gets sent later in the order of ACB when it could've been sent in the order of ABC instead which would've been better in that case.

Therefore it's important that every part of the pipe is able to provide backpressure because if it doesn't, it'll buffer too much and end up sending the response with lower priority before higher priority.

If you also apply too much backpressure then you might instead stall waiting on more content to be written. So finding the right balance here is important. This is already the case for maximizing a network stack but often you can get away without it, this architecture just makes it more important.

Flushing

Similar to lack of backpressure, GZIP, other compression algorithms and middlewares can cause buffers to fill up. For example GZIP uses a fixed sized window to do compression on each chunk. Before that window is full, it'll buffer the content. That means that if there are bytes that could be sent to the client and displayed to the user, they won't get there until the window gets filled up. However, if React is suspended on some slow I/O, we have nothing to write and it just sits there until later. Effectively defeating the purpose of streaming for the use case of managing slow data sources.

In Node.js WritableStreams (and in many other platforms before it), there's a flush() method that React calls to force a signal to any GZIP middleware that it now has to flush its buffer even if it's not yet full. This doesn't exist on ReadableStreams and that's why we don't support ReadableStream.

Web Streams primarily use ReadableStreams as the return value of things like responseWith. You can use WritableStreams (but not in Firefox). However, there's no convention for flush() even in a WritableStream. I'm sure there will be solution eventually. However, if you're trying to build a production server on top of WebStreams with compression support, you'll need to invent an API convention for this that we can add support for.

Corking/Uncorking

The following section isn't really an issue if you use React for the entire document but currently, in practice, a lot of frameworks wants to do richer custom stuff outside what React emits.

If you're trying to prepend data to React's HTML stream, you can easily do that using an intermediate stream. You can also add stuff to the end of the stream after React is done with everything (although this can take a while if it's a slow backing data store). However, if you're trying to add stuff in between React's chunked streaming it's not so easy.

By default, React writes small chunks/buffers to the underlying stream. Currently as little as a single byte. We'll likely start batching them more to avoid overhead downstream but there's no contract for how big or small these chunks might be. There is no guarantee that these buffers contain coherent chunks of HTML. They can be fractions of HTML. So injecting more HTML like style tags, extra scripts, or meta data wouldn't work.

Node.js has a function for cork/uncork which allows an underlying stream to start buffering for a while to create larger chunks. React (ab)uses this to tell the underlying stream that it's done with the current chunk of HTML. In a custom Node.js WritableStream you can use these methods as signals that it's now safe to inject more HTML between the chunks.

There doesn't exist an equivalent convention for Web Streams (and arguably we're abusing it even in Node.js anyway). A hacky workaround is to use setImmediate, postTask or setTimeout(0) to schedule a task. Because React's stream will always resolve in a sequence of micro-tasks, this tasks will always be between one of those. So that's an opportunity to emit more content. Albeit not super efficient.

setImmediate (or postTask)

Another case where we use setImmediate or postTask is to batch work across incoming I/O.

Let's say you fetch(A), fetch(B) and fetch(C). They come back in the order of A, C and B. These then schedule a callback on a queue to worked through by the JS thread. If there's no relative priority between them, you can just invoke the callbacks and runtime computation of each one. That's how most async I/O systems work today.

In the case where CPU is really small and there's no backpressure, this doesn't matter. Let's say computing A takes a little bit of time. In the meantime C gets added to the queue and then later B gets added to the queue - before computing A is done. One computing A is done we have the data for both C and B. Node.js is not a priority queue, so C will be called first and then processed. Even though B was already done.

Additionally, React can often produce a better result by having access to more data. E.g. instead of generating separate HTML segments for B and C that gets pieced together, it's sometimes valuable to do them both at once if both are available.

To solve both of these problems, we don't immediately do work when a Promise is resolved. Instead, we call setImmediate to schedule a callback when the first Promise resolves. What that does is schedule a callback last in the queue. That ensures that we first collect all the Promises that were already ready. Then we work on the highest priority among those.

We don't use setImmediate because it's the ideal architecture but as a hack to workaround the async design of this environment. At a high level what we want to do is:

1. do "CPU work"
2. look at the list of things that completed during the time it took to do the CPU work.

Another way to implement this would be if Node.js had support for the prioritized postTask API. That would help in the sense the work of resolving the Promises would also be prioritized. However, it doesn't help with the batching by itself but if React also scheduled its own work with postTask it would help with that problem too.

There are various other ways to implement async I/O systems that.

Synchronous I/O

We don't take advantage of this yet but we plan on the built-in React I/O APIs like react-fetch, react-fs etc. to sometimes do blocking calls behind the scene internally. This would be mostly applicable when the system is running a JS VM and an OS thread per request. This is common in simplified "CommonJS" environments.

In Node.js there are a lot of synchronous options of existing APIs that are beneficial for this case. They also exist in some Web Workers.

These are commonly highly discouraged in modern JS but there are ways to use them right to great advantage. If you're building a new JS environment - I wouldn't dismiss these APIs. I wouldn't assume you should never add them.

[jplhomer]

This is a phenomenal writeup — thank you for sharing.

For the time being, should we expect to be able to use pipeToReadableStream to stream SSR responses on platforms like Cloudflare Workers?

I've been giving this a shot recently and running into issues: facebook/react#22772 and I'm curious if this is an expected limitation of using the browser version of this API.