stream: implement WHATWG streams

[jasnell]

This is an initial Work in progress Experimental implementation of the WHATWG streams standard.

The intent is for this to be a complete spec compliant implementation. This PR includes the web platform tests.

It is my intention to...

1. Use this as part of the QUIC/HTTP3 implementation, and eventually as part of a larger revamped net subsystem API.
2. Integrate this into the fs/promises FileHandle API.
3. Integrate this into the Blob API
4. Implement adapters and magic for existing node.js streams APIs

Once compiled, the module is available via require('stream/web'). This does not expose the globals, and I don't intend to expose the globals until after the implementation is ready to graduate from experimental.

For an example of how the QUIC API intends to use this... here's a snippet:

session.onstream = async ({stream, respondWith}) => {
  console.log(await stream.readable.getReader().read());  // reads a single chunk

  respondWith({ body: getReadableStreamFromSomewhere() });
};

TODOS:

• Necessary to land initial implementation
  • Finish implementation
    • ReadableStream
    • WritableStream
    • TransformStream
    • Async iteration of ReadableStream
    • pipeTo and pipeThrough
    • postMessage() transfer of ReadableStream, WritableStream, and TransformStream
    • Validate passing WPT tests
  • Add appropriate WPT tests
  • Finish initial documentation
  • Split into multiple files
  • Move into lib/internal/webstreams
• Necessary for coming out of experimental
  • Revisit error codes strategy
  • Performance profiling and optimization
  • Potentially split into separate files
  • Potentially refactor to simplify code
  • Implement Node.js streams adapters
    • stream.Readable <-> ReadableStream Adapters
    • stream.Writable <-> WritableStream Adapters
    • stream.Transform <-> TransformStream Adapters
    • In pipeTo / pipeThrough, if the destination and source are adapters of stream.Readable and stream.Writable, fallback to Node.js pipeline() instead if possible. If source and destination are both backed by native StreamBase, fallback to native pipeline.
    • stream.pipeline integration if possible.
    • Implement FileHandle.prototype.readable to get a ReadableStream for the FileHandle
• Additional
  • Implement Blob.prototype.readable to get a ReadableStream for the Blob
  • Implement QUIC Stream.prototype.readable

[XadillaX]

A deadly simple way is just using Deno's. (just joking

[legendecas]

A deadly simple way is just using Deno's.

IIUC, like said in the OP, it is planned to be "working with existing require('stream') API through low level adapters and magic". I don't believe it's a good idea to "adopt" Deno's implementation as it is not designed to be work with Node.js stream API in the first place.

[ljharb]

It seems kind of odd to have the specifier include a standards body’s name.

[jasnell]

It seems kind of odd to have the specifier include a standards body’s name.

That's something we can worry about later once the basic implementation is done.. but... Alternative suggestions?

[ljharb]

I agree the bikeshed can wait :-) just pointing it out.

As for suggestions: what are the long-term plans in node here? Are "node streams" going to eventually be deprecated/replaced by "whatwg streams"? Or will node likely have two streams implementations for awhile? What's notable about "whatwg streams" besides that they're what's in browsers? Would "web streams" make more sense?

[jasnell]

Existing streams aren't going anywhere. At least not for the foreseeable future. I have no intent here to impact existing streams at all.

Perhaps ... require('streams/standard')?

[ljharb]

Not stoked on implying that web specs are somehow “the” standard, but it’s probably better than “whatwg”, if “web” isn’t viable :-)

[ronag]

I’d very much like to have a discussion about our overall strategy and plan regarding streams in node before landing this. I’m not against it just want to make sure we have a reasonable plan we can communicate.

[jasnell]

I’d very much like to have a discussion about our overall strategy and plan regarding streams in node before landing this

Ok, but I would ask that that discussion not happen in this PR. Can I ask you to open either an issue or discussion topic seeded with your thoughts?

[ronag]

Any preliminary performance figures relative to node streams?

Is stream.pipeline support in scope for this PR or is that future work?

[jasnell]

Any preliminary performance figures ...

No. Not yet. I won't be working on performance optimizations until I'm sure the implementation is correct per the spec.

Is stream.pipeline support in scope for this PR ...

No, the adapters to existing streams API will come in separate PRs

[jimmywarting]

• Implement FileHandle.prototype.readable to get a ReadableStream for the FileHandle

I was like: Wait what? Dose node have something called a FileHandle? must be something i have missed. But turns out that was not the same as FileSystemFileHandle provided by Native file system access (which i totally think we should implement btw, Deno have considered this too)

I don't know how i feel about FileHandle.prototype.readable i think it should perhaps mimic more how the file system access works (you get a FileHandle, call FileHandle.prototype.getFile() and receive a Blob/File backed up by the filesystem (#37340)

const file = await fileHandle.getFile()
const stream = file.stream()
console.assert(file instancesof Blob) // true
console.assert(stream instancesof ReadableStream) // true

---

• Implement Blob.prototype.readable to get a ReadableStream for the Blob

I hope you meant Blob.prototype.stream()?