Prior difficulty persuading TC39 about F# pipes and PFA

[js-choi]

The explainer does not currently explain why the pipe champion group believes Hack pipes to be the best chance of TC39 agreeing to any pipe operator.
(See HISTORY.md for a lot of detailed background.)

During 2017, 2018, and 2021, whenever they were presented to TC39, F# pipes and partial function application (PFA) have run into pushback from multiple other TC39 representatives due to memory performance concerns (e.g., from Google V8’s team), syntax concerns about await, concerns about encouraging ecosystem bifurcation/forking, etc. This pushback has occurred from outside the pipe champion group. (See HISTORY.md for specific examples.)

It is the pipe champion group’s belief that any pipe operator is better than none (in order to easily linearize deeply nested expressions without resorting to named variables). Many members of the champion group believe that Hack pipes are slightly better than F# pipes, and some members of the champion group believe that F# pipes are slightly better than Hack pipes. But everyone in the champion group agrees that F# pipes have met with far too much resistance to be able to pass TC39 in the foreseeable future (or even ever—though I hope not).

To emphasize, it is likely than an attempt to switch from Hack pipes to F# pipes will result in TC39 never agreeing to any pipes at all; PFA syntax is similarly facing an uphill battle in TC39 (see HISTORY.md). I personally think this is unfortunate, and I am willing to fight again for F# pipes and PFA syntax, later—see #202 (comment). But there are quite a few representatives (including browser-engine implementers; see HISTORY.md about this again) outside of the Pipe Champion Group who are against encouraging tacit programming (and PFA syntax) in general, regardless of Hack pipes.

In other words, the explainer currently does not adequately explain the situation regarding the strong pushback that F# pipes and PFA syntax, since 2017, have run into from various TC39 representatives. We need to fix this sometime.

This issue tracks the fixing of this deficiency in the explainer (lack of discussion regarding the pipe champion group’s decision making, after the pushback that F# pipes and PFA syntax encountered in TC39 from outside the champion group). Please try to keep the issue on topic (e.g., comments about the importance of tacit programming would be off topic), and please try to follow the code of conduct (and report violations of others’ conduct that violates it to tc39-conduct-reports@googlegroups.com). Please also try to read CONTRIBUTING.md and How to Give Helpful Feedback. Thank you!

[aadamsx]

It is the pipe champion group’s belief that any pipe operator is better than none (in order to easily linearize deeply nested expressions without resorting to named variables). Many members of the champion group believe that Hack pipes are slightly better than F# pipes, and some members of the champion group believe that F# pipes are slightly better than Hack pipes.

[But everyone in the champion group agrees that F# pipes have met with far too much resistance to be able to pass TC39 in the foreseeable future (or even ever—though I hope not).]

So minimal/F# style will ultimately never to get through TC39 based on what you have said above. I have some thoughts after this realization...

Currently, the people on TC39 board get to decide what goes into JS because the compaines they work for control the browsers. Therefore we in the JS community have no real representation or voice, and can be marginalized with just a click.

Maybe the process becomes more balanced if the corporations cede some control ( I know, good luck with that right? 😉 ).

For example, add real JS community members voted in by us on the TC39 board. And have two representatives per proposal, one community and one corporation derived respectively. This way we actually have representation and some sense of accountability -- closer participation.

Right now, this doesn't feel like participation -- it feels like subjugation of the JS community by the corporations that control TC39.

[js-choi]

So minimal/F# style will ultimately never to get through TC39 based on what you have said above.

@aadamsx: I would not yet say “definitely never”. But I would say “definitely not for a long time” and that “I plan to keep fighting for F# pipes after this but I expect continued strong difficulty”.

I hear your frustration at not feeling represented. (See also #215.) Although any individual human who can pay can join TC39 by becoming a member of Ecma, it is true that, functionally, most members of TC39 are companies.

What I would say (and which the explainer should also mention) is that, no matter how much representation a community gets in the Committee, in the end, the engine implementers will always have final say on the language.

It’s impossible to have the engine implementers, because they are the ones implementing the language. The worst case scenario would be for one browser member to refuse to implement something that got standardized and fork the language; then the standard is no longer a standard. So when an implementer says that they have deep concerns, e.g., about F# pipes’ or PFA syntax’s memory problems, then we have to really pay attention to that. There’s no point in pushing something that they will block or not implement.

Like @ljharb said a few years ago about a different proposal (tc39/proposal-cancelable-promises#70 (comment)):

One of the important priorities in TC39 is to achieve [Committee] consensus - which means that any employee of any member company can always block any proposal. It's far far better for anyone to be able to block, than to arrive at a spec that "not everyone” [not every engine] is willing to implement - this is, by a large large margin, the lesser evil, I assure you.

Having said that, I agree that it is frustrating. I can only say that I am sorry, that we the pipe champion group should acknowledge it in our explainer, that we hope you would still sometimes find Hack pipes useful, and that we’ll try our best to do right by everyone within the constraints of Committee consensus.

Anyways, this all belongs in the explainer, and we’ll add it later so we can point to it.

[mAAdhaTTah]

I also strongly disagree with the characterization of the members of the committee as separate from the community. They're all involved in designing or developing core elements of JavaScript or the web that I'm sure you've used before. Many of them have worked in functional languages, including both JS + Tab. They're not outsiders imposing outside opinions on JavaScript. They're part of making JavaScript what it is throughout the ecosystem.

That certainly doesn't mean it's a perfect system, or even necessarily a good one (I worry about Google's influence on the web, especially now w/ Edge on Chromium), but I do think language design needs to be done by experts, with community input, but not by democracy. Those outside of the committee have perspectives & experience to share but we the community are no longer exploring new territory. Despite all the back and forth, I didn't see (m)any novel arguments in favor of F#, and only a few in favor of Hack, throughout that thread. This is not to minimize the community involvement generally but to say that in fact your points were deeply considered and your input was taken into account, and Hack faced fewer obstacles and objections from the committee and would be more likely to get to consensus.

[js-choi]

Note for readers: There’s a lot of related talk about browser implementers’ concerns with F# pipes’ memory performance, starting from #215 (comment). I’m fine with that, since it feeds into frustration about TC39’s process and the feeling of F# pipes being promised and being taken away by TC39 representatives’ concerns. (Maybe I should have made an issue devoted to performance concerns, haha…)

Pasting from #217 (comment): Since 2018, TC39 representatives outside of the champion group have brought up several strong concerns, several times, about F# pipes and partial-function-application syntax (both of which I plan to fight for in the future, after Hack pipes).

Memory performance is indeed one of those deep concerns, although it is not the only one.

With regards to performance, though browser implementers have said repeatedly to the group that people generally overestimate how much optimization they can do, and they feel that it probably applies to this case (e.g., whenever a curried function is declared separately, how error stack traces make inlining observable to the developer, etc.).

To be more specific, engine implementers have been concerned about F# pipes and PFA syntax encouraging the rapid allocation of new function objects in hot paths—function objects that might often not be able to be inlined. For example, the callback in [1, 2, 3].map(x => x + 1) may be inlined. But, given a curried add function, [1, 2, 3].map(add(1)) may not be inlined—especially if add can throw an error. Error stack traces are observable by the developer, and any function that may throw an error must be preserved for the error stack. It’s these sorts of concerns that have driven engine implementers’ resistance to F# pipes and PFA syntax. (@mAAdhaTTah’s comment in #215 (comment) is also a good read.)

I personally support F# pipes, and I support PFA syntax, in addition to supporting Hack pipes. But the buck stops with the engine implementers, because the engine implementers can simply refuse to implement a feature, while another engine does deploy the feature—and then the language is forked in real-world code. This would be the worst-case scenario for any standard.

Anyways, there’s more talk about engine implementers’ memory concerns, going on in the frustration issue (#215 (comment)). I’m fine with the discussion continuing there, since it’s…kind of on topic there, even though it’d be also on topic here.

All of this stuff should be documented in the explainer…which we’ll get to sometime.

[JustinChristensen]

@js-choi

I don't have the full context on the history of this, but there's been a lot of reference made recently to a first-blush impression of one of the JS engine maintainers on the performance implications of PFA and composition for a language like JS.

Given that neither of these are new ideas I would think it would be worthwhile to get some opinions from those that maintain optimizing compilers for mainstream languages that have implemented these constructs so that the JS community can better understand what the actual runtime implications will be, and where we'd be able to optimize if we went down the PFA/composition route.

Perhaps the JS standards community could request a short document from the F# compiler toolchain maintainers that explains how they've chosen to represent these constructs and where they've been able to optimize within their toolchain?

[JustinChristensen]

Really, given the size and importance of a language like JS, and the scope of the proposed changes, it shouldn't be out of bounds to get a general idea of the proposed JS engine implementation for either approach before making a decision to extend the language.

[mAAdhaTTah]

Engine implementers are not going to spend engineering time implementing or even designing for a Stage 1 proposal with an uncertain future, so that sort of things isn't typical for language design (related to #216).

[JustinChristensen]

@mAAdhaTTah

They don't need to. I suggested reaching out to the those that have already done this for other languages, and not JS engine implementers.

[JustinChristensen]

@mAAdhaTTah

Also, the precise stage of a proposal shouldn't matter in the slightest if it's being weighed as a real option against another proposal that is on the same track where one of the two is very likely to succeed. It's pretty clear that the JS community wants some form of function composition. It's just a question of which design to go with, and peeking behind the curtain at possible implementations should be done.

[mAAdhaTTah]

I explained here why that comparison wouldn't be valid.

[JustinChristensen]

@mAAdhaTTah

Ah. Proper function composition, rather than this pipe operator, would address that concern:

// no extra closures
c = a(1) >> b(2);

But I see now that what's being proposed is not actual function composition, but rather the pipe operator, which does introduce the problem you're describing. My fault for skimming and prematurely posting.

[JustinChristensen]

Maybe what should be being discussed is function composition, and not pipe. Some of the posts I've seen make reference to the fact that composition can be implemented in terms of pipe and an arrow function, but as your example shows that can then be used as a justification for why it would be undesirable if used with PFA.

[micahscopes]

It'd be beneficial to document in a clear and convincing way exactly what the memory performance concerns are that Hack pipes have been chosen to avert.

I'm a big fan of at least having the option to compose pipelines of anonymous functions if I want to (higher order functions make it possible for such nice declarative APIs 😻 and I especially like the tacit syntax)

That said, the concerns raised about the potential for peoples' code to spew lots of anonymous closures really does hit home, and I really do appreciate that this has been brought up.

I've been seeing some examples in comments that I believe oversimplify the performance concerns, as though hack pipes --> no extra closures --> fast and F# --> spewing anonymous closures everywhere --> slow. But if I'm understanding the memory concerns correctly, that's just not true.

Hack pipes could be used in a way that creates a lot of extra closures on the fly and tacit syntax could be used in a way that doesn't. Seems like it all comes down to how higher order functions (curried or otherwise) are used, and that the TC39 representatives mostly want to discourage careless use of higher order functions. Is that the case? If so, I think it'd be good to clearly document that, with details.

To illustrate what I mean, I've made some examples. Could someone in the know correct me if I'm not understanding right?

Examples:

Hack pipes, spewing closures

import {hof1, hof2} from 'external-library' // bring in some useful higher order functions we need

export const process = ({ x, P, Q }) => x |> hof1(P)(^) |> hof2(Q)(^)

// use the render function
import { pt, transform, render } from 'another-library'

const P = 1
const Q = 2
for (let [i,j,k] of ...) {
  pt(i,j,k) |> transform(^) |> process({x: ^, P, Q}) |> render(^)
}

F# pipes, optimized

import {hof1, hof2} from 'external-library' // bring in those higher order functions again

export const makeProcessor = ({ P, Q }) => x => {
  const hof1P = hof1(P)
  const hof2Q = hof2(Q)
  return x => x |> hof1P |> hof2Q // hof1P and hof2Q are fixed
}

// in use
import { pt, transform, render } from 'another-library'
const process = makeProcessor({ P: 1, Q: 2})
for (let [i,j,k] of ...) {
  pt(i,j,k) |> transform |> process |> render
}

Furthermore, to cut back on wasteful allocation, shouldn't users of higher order functions be encouraged to memoize/pre-allocate in cases where that makes sense, regardless of they're using a pipeline operator or not?

Personally, I think that usage of higher order functions should be celebrated in JavaScript, and that users ought to be educated about how to use them with care, and when and how to optimize.

JavaScript is said to be a language where "functions are a first class citizens", but apparently it's a little more complicated than that when it comes to maintaining a JavaScript engine... 👀

Personal context: I've been using the F# style pipeline operator in an audio thread to do non-linear music/sfx sequencing. The audio thread in Chrome on Linux/Android is highly sensitive to memory pressure, and it sounds real bad when I'm making a mess on the heap. I've had to be very mindful about how many objects I'm creating, and especially about making sure stuff gets cleaned up regularly. Otherwise I get clicks and pops.

[js-choi]

@micahscopes: Thanks for your well-reasoned comment.

From what I know, the engine representatives’ performance concerns stem from the encouragement of widespread tacit programming, especially widespread curried functions, within hot paths, without developers bothering to cache them as your code does above.

It is true that caching functions created from HOFs would mitigate many of these concerns. Your sample code seems well, good, and wise to me.

But it is also true that F# pipes may encourage widespread and careless function-object creation in hot paths, especially in order to pipe through n-ary functions. Many people would not bother naming every single unary function they create from HOFs, as you do with transform, process, and render; instead, they would simply leave in hof(P) or arrow functions or curriedFn(blah). And they anticipate that inlining such functions in hot paths would be quite difficult (observable error stack traces, non-reusability of HOF-created closures, etc.). At least that’s what I know.

Also keep in mind that performance concerns are not the only concerns that F# and PFA syntax have encountered from TC39 representatives. (Other serious concerns they have raised included encouraging ecosystem bifurcation.)

The general gist of this issue (which I still need to turn into an explainer FAQ) is that many representatives are concerned that F# pipes would encourage widespread tacit programming (especially curried functions) and want to avoid encouraging widespread tacit programming, for various reasons that include but are not limited performance. I want F# pipes and PFA syntax in addition to Hack pipes, and I plan to fight for F# pipes and PFA syntax later—but I know that there are many barriers and much skepticism that they still would have to surmount, and while I am hopeful that they’re possible, I am also realistic about their odds.

I appreciate your well-reasoned comment, and I think your code is fine, as it caches every unary function created from HOFs. But the implementers would probably not be concerned about your code—they would be concerned about the widespread adoption of code like x |> hof(blah) |> x => x * 2 |> curriedFn(3) in hot paths.

[ghost]

I plan to fight for F# pipes and PFA syntax later

I need some clarification on this. If Hack pipes are adopted, what chances are there for F# pipes to ever be? I cannot imagine support for an alternate syntax to the same feature.

[js-choi]

I plan to fight for F# pipes and PFA syntax later

I need some clarification on this. If Hack pipes are adopted, what chances are there for F# pipes to ever be? I cannot imagine support for an alternate syntax to the same feature.

This is sort of getting off topic for this thread (it probably belongs on #202 instead), so I will hide my answer in this details element.

---

The chances that TC39 would approve F# pipes have always been small, for reasons unrelated to Hack pipes being an alternative (see HISTORY.md). It is true that TC39 approving Hack pipes would (at least slightly) lessen the likelihood of F# pipes, but F# pipes’ chances were already small.

If Hack pipes successfully advance to Stage 4, I would try to argue in a new proposal that the |> foo(%) pattern is going to be common enough that we should lubricate it with a secondary unary-function-application operator |>> foo. This separate operator would have to fight for itself on its own merits. If TC39 deems this use case not common enough or that the (%) suffix is fine in that use case, then it will reject my proposal.

This is a long-term plan; it would not happen soon. A closer-term goal would be for me to propose Function.compose (#76), Function.constant, and Function.identity built-ins, which I also hope to do sometime soonish.

[Edit: See proposal-function-helpers.]

[shuckster]

Something came to my mind on the subject of performance: Does anyone have an idea of how much greater (or less/neutral) the performance concern about F# pipes would be if V8 still implemented Tail Call Optimisation?

[mAAdhaTTah]

@shuckster I don't think it has any bearing here because only the last function call in an F# pipe could possibly be in tail position.

[shuckster]

Thanks for the reply @mAAdhaTTah , that makes sense. I was rather hoping there was some obscure optimisation opportunity lurking, but perhaps not.

[mAAdhaTTah]

FWIW, I'm not really sure eliminating the performance issue is a sufficient condition for advancement. I think there's been a lot of noise around that issue in this repo that far exceeds the significance the performance issue actually has on F#'s status.

[shuckster]

Fair enough. I have to ask though -- do you know how "in proportion" the cited performance concerns were across each style?

I can't imagine that Hack affects performance whatsoever against the baseline, but I'm just wondering if this set an unreasonable standard for the other contenders.

F# would have been handicapped purely by virtue of being fp vs. imp. to start with. Did the argument against it take the same shape as any fp vs. imp. argument? I know closure allocation is a concern when using such pipelines with lambdas, but tacit-style is not as prone to this, right?

[mAAdhaTTah]

I have to ask though -- do you know how "in proportion" the cited performance concerns were across each style?

I can say that there were concerns raised about F# pipes that weren't raised for Hack; the perhaps-obvious reason being the need for point-free style & the creation of closures. Elixir-style pipes also don't have closure problems because it's ultimately an argument insertion into the function-as-written, rather than an evaluation-then-application. The issue with Elixir (which I share) is it feels pretty un-JavaScript-y to insert an argument in the beginning and breaks the expectation that foo(a, b) evaluates foo with the arguments a & b because in a pipe, it would be like x |> foo(a, b) evaluates as foo(x, a, b).

I know closure allocation is a concern when using such pipelines with lambdas, but tacit-style is not as prone to this, right?

It's actually the opposite. The inline lambdas could be easily optimized away. In fact, the babel plugin currently does this now, extracting the body of the lambda to be run inline. Tacit-style, or more specifically, point-free style code suffers from this issue. For a small example:

const add = (a, b) => a + b;
const addCurried = a => b => a + b;

// Not an issue for an engine to inline.
// The babel plugin will inline this properly.
1 |> x => add(2, x);

// Can be inlined as:
const _ref = 1;
add(2, _ref);

// Requires addCurried to be evaluated,
// and thus a closure to be created first, then run.
// This is hard/impossible to optimize away.
1 |> addCurried(2);

// Desugars to:
const _ref = 1;
const _step = addCurried(2); // this can't be optimized away
_step(_ref); // _step is the closure that needs to be allocated

As (I believe?) you mentioned, it is a reasonable question to ask how much this matters, and if the other objections were overcome, I suspect we'd be able to push through the performance issue. That said, it's also worth noting that current FP style pipe also doesn't suffer from this issue, in that pipe actually creates a new function, rather than evaluates it:

// In the current version, the add(2) & multiply(2) closures are allocated only once.
const add2ThenDouble = pipe(add(2), multiply(2));

// In the F# pipe version, they're allocated on each invocation.
const add2ThenDouble = x => x |> add(2) |> multiply(2);

// You'd have to manually pull them out to avoid this.
const add2 = add(2);
const double = multiply(2);
const add2ThenDouble = x => x |> add2 |> double;

// At that point, the original version with pipe is definitely better.
// F# pipe actually introduces a performance footgun vs your baseline tools!

[tabatkins]

That said, it's also worth noting that current FP style pipe also doesn't suffer from this issue, in that pipe actually creates a new function, rather than evaluates it:

That depends on the exact library being used; RxJS, for example, uses a .pipe() method on observables, which is back in the single-use category, akin to F#-style pipes.

The pipe() you're referring to is just function composition, and usually goes by a different name ("compose" or "flow", right?).

[js-choi]

I know closure allocation is a concern when using such pipelines with lambdas, but tacit-style is not as prone to this, right?

It's actually the opposite. The inline lambdas could be easily optimized away. In fact, the babel plugin currently does this now, extracting the body of the lambda to be run inline. Tacit-style, or more specifically, point-free style code suffers from this issue.

For what it’s worth, optimizing away an inline lambda would have also been user observable due to error stacks. I think we would have had to specify the automatic unwrapping of inline lambdas in the F#-pipe specification itself.

The pipe() you're referring to is just function composition, and usually goes by a different name ("compose" or "flow", right?).

Lodash and fp-ts call LTR function composition flow, but Ramda and Sanctuary call it pipe. It’s a little confusing.

[mAAdhaTTah]

Yeah, and my experience has been primarily with Ramda, hence calling it pipe. I will say, I don't know as there are too many userland implementations of the pipe(input, ...fns) variant, altho RxJS is effectively that variant with input as this.

[noppa]

optimizing away an inline lambda would have also been user observable due to error stacks.

That's true for lambdas, but not for partially applied functions 1 |> add~(?, 2) if we got those, since the proposal promised shallow stacks

---

fp-ts has both pipe(input, ...fns) and flow(...fns)(input). The input-first variant tends to be nicer to use with TypeScript/Flow.

[samhh]

The input-first variant tends to be nicer to use with TypeScript/Flow.

How so? That's not been my experience as someone using it extensively for a few years in TypeScript. It's effective at achieving point-free code.

[noppa]

You can get good autocomplete and type checking in the functions that follow the input-argument without any extra explicit types, i.e.

declare function sortBy<T>(iteratee: (val: T) => number | string): (arr: readonly T[]) => T[]

pipe([1, 2, 3], sortBy(_ => _.toFixed(2)))

just works™. The .toFixed is autocompleted and checked for typos etc.

The compose variant requires explicit types either to the final created function or to the sortBy argument position. That's probably ok and even good practice when creating functions that are supposed to be reused or exported, but it makes it unnecessarily cumbersome to use with simple one-off pipelines. Both have their uses.

Flow (the type checker) works a bit better than TS with flow (the compose helper function) without explicit types, but still requires you to awkwardly jump first to the end to add (input) and then back to adding the function calls if you want to keep autocomplete working.

[samhh]

The compose variant requires explicit types either to the final created function or to the sortBy argument position. That's probably ok and even good practice when creating functions that are supposed to be reused or exported, but it makes it unnecessarily cumbersome to use with simple one-off pipelines. Both have their uses.

This matches my experience, but it's also pretty rare that it can't be inferred within a callback which for me is where most of those one-off pipelines arise. Here the flow is redundant but you could populate it however you'd like, everything still works:

pipe([[1, 2, 3]], A.map(flow(sortBy(x => x.toFixed(2)))))

[js-choi]

Starting on 2021-10-25, another formal Commitee plenary occurred. The notes have now been published. We have summarized these updates in #256, and there are some relevant points about PFA and F#-style pipes:

• PFA (partial function application) syntax was presented again to the Committee plenary for Stage 2. The Committee rejected this proposal; several representatives, including those from Mozilla SpiderMonkey and Google V8, stated that there were insufficiently specific and compelling use cases presented, with high syntax cost and novelty in comparison to arrow functions. This rejection was independent of and not related to Hack pipes—instead, they were related to PFA syntax’s own merits versus arrow functions.

• A proposal for several Function helpers including Function.pipe and flow was also presented to the Committee plenary for Stage 1. The Committee also rejected this proposal due to its being overly broad, and it requests that it be split up into multiple proposals. There is now a new Stage-0 proposal devoted to Function.pipe and flow.

[js-choi]

I will be presenting proposal-function-pipe-flow for Stage 1 at the plenary meeting later this week. Feel free to leave feedback about the proposal (and my slides) there. I will follow up here with the results.