Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Move isaacs/readable-stream to iojs and make it the authoratative source #55

Closed
rvagg opened this issue Dec 4, 2014 · 24 comments

Comments

@rvagg
Copy link
Member

rvagg commented Dec 4, 2014

#53 and all of the other obscure streams issues that keep coming up keep on giving me headaches. Even though I "maintain" readable-stream as an extraction of joyent/node streams I still don't have a complete handle on the inner workings of the streams code and I imagine this is the same for many core contributors. I'm willing to bet that even now, nobody understands the code better than @isaacs because it's grown so complex.

As the most obvious source of JavaScript complexity in core, I propose that we move readable-stream to iojs and make it the place to file streams issues and where the code in core is extraced from rather than the other way around. It can also have its own core team, separate but potentially overlapping with iojs TC, that cares about and understands how they work and maybe the tight integration with core can be loosened (and perhaps broken completely) over time. As a separate project we can also pursue better testing (including browserify) of the code.

@rvagg rvagg added the tc-agenda label Dec 4, 2014
@feross
Copy link
Contributor

feross commented Dec 4, 2014

+1 I'm hugely in favor of this.

@rvagg
Copy link
Member Author

rvagg commented Dec 4, 2014

@feross be warned, you'd be on my list of people to pull in to a maintainer team if this goes ahead

@jonathanong
Copy link
Contributor

+1

@isaacs
Copy link
Contributor

isaacs commented Dec 4, 2014

+1. @rvagg do you need me to do the switch?

@fengmk2
Copy link
Contributor

fengmk2 commented Dec 4, 2014

+1

@rvagg
Copy link
Member Author

rvagg commented Dec 4, 2014

@isaacs at this stage I'm more looking for agreement from TC to reverse the flow of code so that all the work is done in readable-stream rather than core, but there's no harm in moving readable-stream over to the org in the meantime.

@isaacs
Copy link
Contributor

isaacs commented Dec 4, 2014

Oh, I see, so the plan would be to have the module be the "upstream" and then iojs consumes it?

If so, that makes a lot of sense, and moving it to the org also seems sensible. I think this should be the exception rather than the norm, however. Stuff like util-debuglog should probably still remain as-is for now.

@ahdinosaur
Copy link

👍

1 similar comment
@tellnes
Copy link
Contributor

tellnes commented Dec 4, 2014

+1

@indutny
Copy link
Member

indutny commented Dec 4, 2014

Finally, +1

@missinglink
Copy link
Contributor

+1 happy to help

@bnoordhuis
Copy link
Member

+1 as long as I don't have to triage bugs. :-)

@chrisdickinson
Copy link
Contributor

I'm a little worried about making readable-stream (the package) the authoritative source for streams in Node/iojs -- though at the moment I'm having difficulty putting the "why" down into words. I'll revisit this later today and see if I can't explain better, in the meantime I'm "-0" for that change (it could just turn out that I'm being silly about this, though.) I'm totally +1 putting the package under the iojs org, though!

@chrisdickinson
Copy link
Contributor

Okay, here goes:

Issues with and PRs against streams will be opened against this repository. The latter can be solved with a big ole' disclaimer comment at the top of the vendored file, but the former splits issues between "those who know where readable-stream is" and "those who don't." Additionally, It could have the side effect of making stream issues less visible to streams maintainers, who would have to look in two places to track current problems (and, due to the gravity-like effect of interest on attention, will likely end up following the readable-stream tracker way more closely than the iojs tracker).

maybe the tight integration with core can be loosened (and perhaps broken completely) over time

As one of node's core primitives, I'm not sure loosening or breaking streams the integration of streams with core is feasible; even if it is feasible, I am not sure that it would be a good thing. Streams are one of the few things that packages must agree on. Anecdotally, js-git is a good example of what a world without standard stream primitives would look like -- where the stream primitive is a local agreement between cooperating packages.

In order to have my streams1-based git packages work with js-git, I rewrote them into min-streams and added translation modules. The min-streams only depended on other min-streams), and exposed streams1 to other consumers on the side. This took a lot of time, and while it made the module usable from min-streams or streams1, it still wouldn't necessarily work with pull-streams, or RxJS; and it only worked in a single package because I had a verbal agreement with the consumers (all two of us) of these packages -- I knew that the min-stream version would be available at <package>/min.js. Were this not the case, there would likely be a streams-package-A and a min-package-A, which is the sort of lock-in effect that streams effectively combatted.

One of the really lovely things about streams as a primitive is that modules are casually pluggable. There's not so much specification as to get in the way of the problems being solved, or so little specification that more time is spent providing shim modules at edges between ecosystems, but just enough agreement to foster cooperation and enable the community to do some truly cool things. My experience integrating streams1 with min-streams for js-git led me to believe that local, ecosystem-based agreements for the stream primitive are far less effective than global agreement. Min-streams aren't bad -- they're really neat, and for the constraints js-git works within they make total sense -- but both streams1 and min-streams lost a lot of effectiveness by not being the globally agreed-upon standard interface.

(And, given all that, it follows that I believe that the readable-stream module and this repo would have to be upgraded in lockstep for most changes to readable-stream, which would create more work, and thus more potential for human error than currently exists.)

I'm willing to bet that even now, nobody understands the code better than @isaacs because it's grown so complex.

Oof, I definitely agree with this. I have a tiny stack of hand drawn state machine diagrams with a lot of frustrated scribbles in the margins that I consult regularly. We need to decrease the bus factor, and forming an impromptu team of people that can field issues with streams is a great idea.
That said, I don't think moving streams out of core is a precondition to forming that team.

I'm a little concerned about having the option to "solely" work in streams. Having streams in core provides JS developers who are aspiring contributors a nice "ladder" to climb up into different subsystems (and C++). The design of streams, on the other hand, benefits from informed use in core -- and I think there are still gains to be made there (uniting resource/handle-backed streams under a common subclass, or otherwise addressing pain points with the existing integrations).

Also of note: stream documentation -- due to the complexity of streams, they hit a lot of pain points with the way documentation is currently done in Node (is it a guide? is it reference material? who is the audience?) Moving the symptom out of Node proper makes it less likely that the underlying problem will get addressed satisfactorily -- when docs are revisited, they should be revisited with the pain of streams docs fresh in mind, and with solutions in hand.

Sorry for the unstructured stream of concerns. To recap, in numbered list form:

  1. Moving streams into a repo doesn't move the issues out of this repo.
    1. But it does make stream maintainers have to look in two repos for pertinent issues.
    2. It adds a step in PR'ing streams changes.
    3. It divides the audience into folks who know where to look and folks who don't.
  2. I do not think that decoupling streams from core will be to either core or stream's benefit.
    1. Streams work best as global agreements between all packages.
    2. Changes to a readable-stream repo will likely have to happen in lock-step with changes to this repo.
      1. This adds manual steps, which increases the potential for human error.
  3. Streams are complex enough to deserve their own repo
    1. I think that having to work with streams in-situ improves the design of streams.
    2. Having streams in core gives new contributors who are familiar with JS a good place to start.
    3. Removing the onus of documenting streams from this repo means that updates to the docs subsystem are less likely to keep the pain points of streams in mind.

@mjackson
Copy link
Contributor

mjackson commented Dec 5, 2014

I'm willing to bet that even now, nobody understands the code better than @isaacs because it's grown so complex.

This is my main concern with using readable-stream. The code is far, far more complex than it needs to be. As one of the core primitives, the code should be as simple and portable as possible.

I have a tiny stack of hand drawn state machine diagrams with a lot of frustrated scribbles in the margins that I consult regularly.

I'm willing to bet that everyone here has spent countless hours digging through that code, searching for obscure bugs and trying to fix them.

Can we please, please simplify? I'm more than happy to help with the effort. I wrote BufferedStream in the days of node 0.4 to normalize behavior of the different "streams" that existed at that time, and I'm familiar with the complexity of this particular problem.

@mikeal
Copy link
Contributor

mikeal commented Dec 5, 2014

As one of the core primitives, the code should be as simple and portable as possible.

This sounds like something obvious but it turns out not to be true. Streams has to straddle old-style and new-style support in one object and be as performant as possible.

It is very common in Node that code is obfuscated in crazy looking ways for performance reasons. Simplicity and clarity is certainly the top priority for API design but I'm afraid it falls below performance and compatibility in the actual implementation.

@mjackson
Copy link
Contributor

mjackson commented Dec 5, 2014

Simplicity and clarity is certainly the top priority for API design but I'm afraid it falls below performance and compatibility in the actual implementation.

I don't see any evidence to suggest that performance must suffer for the sake of clarity and brevity. For some reason node has always assumed that the inverse is true though, and it's an opinion that has severely hampered its usability. You can have good performance and a clean implementation.

@mikeal
Copy link
Contributor

mikeal commented Dec 5, 2014

@mjackson there is a long history of patches from @trevnorris that are designed to hit v8 performance paths and are much less readable than they would be if performance was not a consideration.

@chrisdickinson
Copy link
Contributor

Can we please, please simplify? I'm be more than happy to help with the effort.

I'm +1 on simplifying streams where possible, but it might be best to open up that point up as a separate issue so the conversations don't get intermingled too much.

It is very common in Node that code is obfuscated in crazy looking ways for performance reasons.

Compatibility is a big reason things streams are complex, but performance is not a prerequisite contributor to their complexity. At a micro level, there's all sorts of deoptimizations in streams; at a slightly larger level, some of the backing data structures can be swapped out to improve performance for common cases, like logging.

In any case I'm getting ahead of myself -- it would be good to discuss this in a separate issue where we can discuss concrete pain points with streams at present before putting out solutions.

@mikeal
Copy link
Contributor

mikeal commented Dec 5, 2014

I should have been clearer. I think there is a lot of room for streams to get faster, and that the work that goes in to making them faster may not make them less complex.

targos added a commit to targos/node that referenced this issue Apr 17, 2021
Original commit message:

    Merged: [wasm-simd] Fix loading fp pair registers

    We were incorrectly clearing the high reg from the list of regs to load.
    The intention was to prevent double (and incorrect) loading - loading
    128 bits from the low fp and the loading 128 bits from the high fp.
    But this violates the assumption that the two regs in a pair would be
    set or unset at the same time.

    The fix here is to introduce a new enum for register loads, a nop, which
    does nothing. The high fp of the fp pair will be tied to this nop, so as
    we iterate down the reglist, we load 128 bits using the low fp, then
    don't load anything for the high fp.

    Bug: chromium:1161654
    (cherry picked from commit 8c698702ced0de085aa91370d8cb44deab3fcf54)

    (cherry picked from commit ffd6ff5a61b9343ccc62e6c03b71a33682c6084d)

    Change-Id: Ib8134574b24f74f24ca9efd34b3444173296d8f1
    No-Try: true
    No-Presubmit: true
    No-Tree-Checks: true
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2619416
    Commit-Queue: Zhi An Ng <zhin@chromium.org>
    Reviewed-by: Clemens Backes <clemensb@chromium.org>
    Cr-Original-Commit-Position: refs/branch-heads/8.8@{nodejs#28}
    Cr-Original-Branched-From: 2dbcdc105b963ee2501c82139eef7e0603977ff0-refs/heads/8.8.278@{#1}
    Cr-Original-Branched-From: 366d30c99049b3f1c673f8a93deb9f879d0fa9f0-refs/heads/master@{#71094}
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2649176
    Reviewed-by: Victor-Gabriel Savu <vsavu@google.com>
    Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
    Cr-Commit-Position: refs/branch-heads/8.6@{nodejs#55}
    Cr-Branched-From: a64aed2333abf49e494d2a5ce24bbd14fff19f60-refs/heads/8.6.395@{#1}
    Cr-Branched-From: a626bc036236c9bf92ac7b87dc40c9e538b087e3-refs/heads/master@{#69472}

Refs: v8/v8@482e5c7
targos added a commit to targos/node that referenced this issue Apr 27, 2021
Original commit message:

    Merged: [wasm-simd] Fix loading fp pair registers

    We were incorrectly clearing the high reg from the list of regs to load.
    The intention was to prevent double (and incorrect) loading - loading
    128 bits from the low fp and the loading 128 bits from the high fp.
    But this violates the assumption that the two regs in a pair would be
    set or unset at the same time.

    The fix here is to introduce a new enum for register loads, a nop, which
    does nothing. The high fp of the fp pair will be tied to this nop, so as
    we iterate down the reglist, we load 128 bits using the low fp, then
    don't load anything for the high fp.

    Bug: chromium:1161654
    (cherry picked from commit 8c698702ced0de085aa91370d8cb44deab3fcf54)

    (cherry picked from commit ffd6ff5a61b9343ccc62e6c03b71a33682c6084d)

    Change-Id: Ib8134574b24f74f24ca9efd34b3444173296d8f1
    No-Try: true
    No-Presubmit: true
    No-Tree-Checks: true
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2619416
    Commit-Queue: Zhi An Ng <zhin@chromium.org>
    Reviewed-by: Clemens Backes <clemensb@chromium.org>
    Cr-Original-Commit-Position: refs/branch-heads/8.8@{nodejs#28}
    Cr-Original-Branched-From: 2dbcdc105b963ee2501c82139eef7e0603977ff0-refs/heads/8.8.278@{#1}
    Cr-Original-Branched-From: 366d30c99049b3f1c673f8a93deb9f879d0fa9f0-refs/heads/master@{#71094}
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2649176
    Reviewed-by: Victor-Gabriel Savu <vsavu@google.com>
    Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
    Cr-Commit-Position: refs/branch-heads/8.6@{nodejs#55}
    Cr-Branched-From: a64aed2333abf49e494d2a5ce24bbd14fff19f60-refs/heads/8.6.395@{#1}
    Cr-Branched-From: a626bc036236c9bf92ac7b87dc40c9e538b087e3-refs/heads/master@{#69472}

Refs: v8/v8@482e5c7
targos added a commit that referenced this issue Apr 30, 2021
Original commit message:

    Merged: [wasm-simd] Fix loading fp pair registers

    We were incorrectly clearing the high reg from the list of regs to load.
    The intention was to prevent double (and incorrect) loading - loading
    128 bits from the low fp and the loading 128 bits from the high fp.
    But this violates the assumption that the two regs in a pair would be
    set or unset at the same time.

    The fix here is to introduce a new enum for register loads, a nop, which
    does nothing. The high fp of the fp pair will be tied to this nop, so as
    we iterate down the reglist, we load 128 bits using the low fp, then
    don't load anything for the high fp.

    Bug: chromium:1161654
    (cherry picked from commit 8c698702ced0de085aa91370d8cb44deab3fcf54)

    (cherry picked from commit ffd6ff5a61b9343ccc62e6c03b71a33682c6084d)

    Change-Id: Ib8134574b24f74f24ca9efd34b3444173296d8f1
    No-Try: true
    No-Presubmit: true
    No-Tree-Checks: true
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2619416
    Commit-Queue: Zhi An Ng <zhin@chromium.org>
    Reviewed-by: Clemens Backes <clemensb@chromium.org>
    Cr-Original-Commit-Position: refs/branch-heads/8.8@{#28}
    Cr-Original-Branched-From: 2dbcdc105b963ee2501c82139eef7e0603977ff0-refs/heads/8.8.278@{#1}
    Cr-Original-Branched-From: 366d30c99049b3f1c673f8a93deb9f879d0fa9f0-refs/heads/master@{#71094}
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/2649176
    Reviewed-by: Victor-Gabriel Savu <vsavu@google.com>
    Commit-Queue: Achuith Bhandarkar <achuith@chromium.org>
    Cr-Commit-Position: refs/branch-heads/8.6@{#55}
    Cr-Branched-From: a64aed2333abf49e494d2a5ce24bbd14fff19f60-refs/heads/8.6.395@{#1}
    Cr-Branched-From: a626bc036236c9bf92ac7b87dc40c9e538b087e3-refs/heads/master@{#69472}

Refs: v8/v8@482e5c7

PR-URL: #38275
Reviewed-By: Matteo Collina <matteo.collina@gmail.com>
Reviewed-By: Jiawen Geng <technicalcute@gmail.com>
Reviewed-By: Shelley Vohr <codebytere@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests