Reviving proper tail calls?

[pygy]

Hi all,

as much as some people love to hate Safari ("the new IE", yadda, yadda), I've yet to see a complain about their support for (implicit) proper tail calls in the wild. The debugging problems mentioned in this article seem to be non-issues in practice.

Maybe it is time to revisit the situation?

[ljharb]

Since code using proper tail calls to avoid blowing the stack isn’t web-compatible, and would only work in Safari - which would imply that nobody is shipping it - how would you expect to see complaints about the implementation?

[Pauan]

@ljharb Tail call optimization applies even if the programmer didn't intend for it to apply.

So let's say somebody debugged their code in Safari, and their debugging experience was fine (despite the unintended tail call optimization). Obviously they would not complain.

On the flip side, if somebody is debugging code in Safari and they experience weird behavior (such as call stacks being missing), then they would complain!

So the lack of complaints is (a little bit of) evidence that tail call optimization does not harm the debugging experience, and thus is acceptable to implement in other browsers as well.

[pygy]

I'd argue that, as far as debugging is concerned, accidentally optimized tail calls are the only ones that matter.

People who use them deliberately would not complain, they know what they are doing.

A third case would be someone who relies on a 3rd party library that uses tail calls internally. It may make it harder to debug said lib if it were buggy. If it is a real problem in practice, people will avoid such libs and give them a bad name (I'd be surprised if it happened though.)

[pygy]

Another thing I see raised against proper tail calls is perf issues.

They can be made fast in LuaJIT where the equivalent of

function _while(condition, body) {
  if (condition() && !body()) { // return true in the body to break
    return _while(condition, body);
  }
}

let a = 0;
_while(()=> a < 1000000000, () => {a++});
console.log(a);

is as fast as a

let a = 0;
while (a < 1000000000) a++;
console.log(a);

(and that has been the case since 2010).

What does JS have that makes the code above impossible to optimize?

Edit: FWIW, Safari is almost as good as LuaJIT here.

while 100000000 394 msec
_while() 100000000 526 msec

Putting both benchmarks in the same tick has the plain while loop slower than the functional version.

while 100000000 873
_while() 100000000 503

Edit2: well LuaJIT has both versions at 150 ms, actually :-) But still, the overhead of the functional verison over the plain loop is rather small in Safari.

[getify]

@ljharb as I've pointed out multiple times, including in #4, tail calls are not web-incompatible. I have code shipped in production (that I wrote years ago) using that "adaptive" pattern, which ostensibly is working fine (as far as I know anyway) in non-PTC browsers, and in Safari has "progressively enhanced" to using tail calls.

Whether that pattern is common or not is a separate discussion. It is in fact web compatible and should therefore not be continually dismissed on such grounds.

[getify]

Also, @Pauan is correct here:

Tail call optimization applies even if the programmer didn't intend for it to apply.

There must be tons of code (especially non-recursive) out in the wild that is unintentionally being tail-call'd by Safari. One would have to assume that if tail calls were such a problem (either for performance or debugging), then surely by now enough of a complaint would have been raised in the community as backlash against Safari, we'd certainly know it.

I'm not aware of any such backlash against Safari, and they clearly haven't changed their minds to back it out. More than a year in production strongly suggests the resistance to PTC was at least partially FUD and not grounded in fact.

[pygy]

@getify you phrased it better than I did, and I too think that the resistance is misguided, but I'd avoid throwing arounds terms like "FUD" (which I associate with wilful disinformation), it will not help our cause.

[ljharb]

@getify your code is not "web compatible PTC code". It's specifically web-incompatible PTC code, combined with a fallback mechanism to the true web-compatible code.

By saying PTC is not web compatible, I'm not saying that you can't write code that utilizes PTC. I'm saying that PTC code without a fallback is web incompatible.

The resistance to PTC is not solely based on debugging concerns; there's also the education aspect of implicit tail calls having magic behavior, but also, there's implementation concerns, where at least one of the major browsers is unable to implement PTC at all.

[getify]

@ljharb seems like nitpicking on words. you used "not web compatible" as an excuse to dismiss the possibility that anyone may have deployed PTC code (or something demonstrably like it) and thus that was the reason for "silence" in terms of complaints.

Not only is my code pattern a counter-example that excepts that reasoning, but you also don't account for web code that's affirmatively using PTC (for recursion or otherwise) but is deployed in a safari-only way, such as in web views in iOS apps.

I still maintain your reasoning is faulty, no matter what label we bikeshed on to label it.

[pygy]

@ljharb I'm surprised that the architecture of one of the VMs is incompatible with PTC.

I'm going to suppose it is Edge we're talking about, since Brendand Eich had started working on the Firefox implementation (I assume he had a general idea of the path forward when he wrote the parser patch), and V8 had it done before it was unshipped.

[elibarzilay]

TL;DR: +1, but with some more details:

1. First of all: I completely agree with the point of this issue. It's even better than splitting the fine hair splittage of what's proper web code or not. The bottom line is that JS code cannot for the most part rely on tail calls for "random web stuff", so the safari move would get developers no real benefits due to portability, but it would get them into the diminished debugging experience that many people are afraid of. The fact that people did not complain is a good indicator that these fears are unfounded.

2. I have said in the past that of all languages, JS should be extremely ready for having tail call elimination (even under a misguided "PTC" name...) -- a quick way to see this is the fact that JS programmers are notoriously using callback-rich code, as well as being comfortable with tricks like deferring code to a timeout which essentially destroys the stack context it originally has. Yet even JS developers are worried about the bad effects on debugging.

3. I think that "FUD" is a very peroper term to use here. There are no hard facts about debugging problems so people are definitely left with uncertainty, there is definitely a fear of hurting productivity as a result, and the very existence of the idea in this repo or the pulling-out of tail call elimination from v8 are making it clear that there is no shortage of doubts. It's not FUD that is actively encouraged by some company, but I think that it is FUD.

4. To make this a bit more concrete: see this reply in that v8 thread as a perfect example of the FUD effect, and later on the same person goes on to conclude that tail calls "kill debugging and profiling". And this is coming from a person who self-identified as someone who's working on DevTools.

   These are exactly the claims that would anticipate some developer backlash against an implementation. Apparently in the safari case such backlash didn't happen, so it looks like "kills debugging and profiling" is much exaggerated.

5. As a side note, I seriously hope that this proposal for an explicit syntax will die. The thing about tail calls is that they are something that affects control flow, and as such there could be issues that are a result of using some 3rd-party library with higher-order functions (ie, callbacks). The main point of making tail calls implicitly is that in most cases such libraries are fine as they are, without me asking the developers of such libraries to change their code (which, if the explicit syntax is used, will undoubtedly lead to redundant arguments with people who would refuse to make such changes worrying about hurting debugging).

[PinkaminaDianePie]

Can it be implemented vice versa? Use proper TCO by default, and in case of greater need use some explicit keyword/expression/directive what else like debugger, which will deoptimize code and generate low-performance version, but with proper call stack? It would be much better to optimize by default and deoptimize on demand for debugging purposes only. Is it possible or such way also break some compatibility?

"education aspect" just makes me laugh, in that case, we can never add anything to standard because people will need to learn something, we should never create any new tools, libraries, frameworks etc, it will take an effort to learn them! In reality, anyone who will face with "broken" call stack will spend 30 seconds to read first entry from google search and will find solved question on StackOverflow with lots of links, and people who don't like to learn anything will still make mistakes like they do right now with any other language features. "some people won't like to learn something new" - it's totally not a valid reason to take it into consideration.

Only reason which can be valid, is the complexity of implementation in browser engines. But if it's not an issue - all other issues can be solved without introducing explicit keyword "optimize me please".

[getify]

Can it be implemented vice versa?

I've thought about a similar path, too. For example, for any libs that want to collect (error).stack in the background, we could have a static boolean property like Error.stack to opt-in to full stack traces. This isn't really a deopt like stopping tail calls, but just indicating to the engine that the shadow stackframes should be collected. If it's not set, it's up to the engine if they want to collect, but if it's set the engine knows the page's code wants that data if possible so it should collect if it can.

For the open devtools use case, it could automatically set that same flag to trigger the same behavior.

[Othreumaru]

TCO is just an optimization, stacktraces which are smaller in size than the limit of a stack in browsers which is usually few thousand may not be optimized at all, the optimization should imho trigger when the stack reaches this limit. At this point stack would be overflown anyway with exception. Usecases where TCO is useful are when code is running recursion over huge number of function calls something like billions or possibly infinity.

[getify]

@Odrinwhite disagree. PTC (not necessarily TCO) is important even without recursion. Recursion is an obvious and acute use case but certainly not the only one.

I want (in FP) to make multiple wrapping layers of functions on top of functions, preserving parameters at each level, via closure, and I don't want there to be a deeper call stack as a result. In that perspective, function wrapping calls being exposed in the call stack is only extra noise for debugging and is thus a leaky implementation.

[Pauan]

@Odrinwhite The strategy of "wait until the stack overflows and then activate TCO" is used in Chicken Scheme (and other compilers as well).

It is a valid way of implementing TCO, so browsers can use that implementation strategy if they wish.

But that has nothing to do with the spec: the spec does not specify how browsers achieve TCO, it merely mandates that they use some method of achieving TCO.

@getify That can be easily handled by having the browser record which frames are tail-call frames, so it can hide them from the debugger. So there's nothing stopping browsers from implementing the Chicken Scheme style of TCO. I doubt that browsers will actually do that, since there are downsides to it, but they could if they wanted to.

[getify]

@Pauan of course, I don't only care about the debugger output... that was just one convenient illustration of a non-recursion use case.

What I actually want is all those wrapped FP functions to not actually cost function calls. And BTW, "cost" is not as much wanting to save CPU cycles in those calls, but wanting to minimize all the memory/GC churn of all those extra stack frame allocations/deallocations.

IMO the entire theory of FP rests on being able to assume there's no "penalty" (memory or otherwise) for an extra wrapping layer of function. In JS we just whistle along, merrily wrapping all these calls, and just winking and nodding to pretend it's OK... that what is conceptually one function call is actually 9 calls... because the cost of the extra 8 calls is generally still small so who cares?

But as soon as you wire up such a wrapped abstraction as a transducer, and hook that up to a continual stream of data events, each one firing every few hundred milliseconds, and run that on a user's mobile device for hours on end, you start wondering if FP is actually worth all that. memory burn.

I'm just trying ensure it's on record, because it seems all too easily lost/ignored/overlooked in these discussions, that the need for PTC goes well beyond recursion.

You can "solve" the lack of tail call recursion with a trampoline and even a code transform or macro... but you can't prevent all my extra function wrappings in FP from churning memory... unless you give me PTC.

Sorry I'm being so persistent/insistent in this thread. It's frankly quite frustrating to have spent basically 5 years campaigning for this vital feature and yet people still casually discount it as niche or suggest narrow work-arounds that aren't relevant to the broader perspective.

[Othreumaru]

@getify Don't get me wrong I would like to see your version implemented but I think the argument that it breaks the web may be valid here for some people. And not breaking web was a big deal in designing ES6 so my proposition is to get the foot into the doors and allow people write TCO in some code which may not need to be very efficient but still fun to do. And with more people trying and using this style of programming a better or more aggressive solution can be established.

[PinkaminaDianePie]

@Odrinwhite proper TCO doesn't break the web itself. Only some ways of implementation break it, and only in some cases. So instead of rejecting standard, it's better to find other ways of implementation, which will be acceptable. One proposal was to give developer possibility to deoptimize code on demand and see a proper stack trace. We will have proper TCO, developers will be able to check stack trace. Why not to go this way?

[Pauan]

@getify What I actually want is all those wrapped FP functions to not actually cost function calls. And BTW, "cost" is not as much wanting to save CPU cycles in those calls, but wanting to minimize all the memory/GC churn of all those extra stack frame allocations/deallocations.

That is not guaranteed by the spec, the spec only guarantees amortized behavior. In general the spec doesn't say much about performance.

Also, stack allocations/deallocations are not the same as heap allocations: stack allocations are not managed by the garbage collector, and they are generally free (zero-cost). There is no performance penalty to using the stack.

Things get more complicated with JS, because browsers might have shadow stacks, or put additional information onto the stack/heap, etc. But the general principles of stack vs heap still applies.

Also, trying to guarantee no-extra-stack-frames-ever can actually make the performance worse, not better. Performance is complicated.

that what is conceptually one function call is actually 9 calls... because the cost of the extra 8 calls is generally still small so who cares?

That's not how Chicken Scheme works. It doesn't actually pay any performance penalty for the function calls.

It doesn't say "oh we'll create extra stack frames because the cost is low". Instead it says "we do normal function calls without any performance penalty, but then once in a while (when the stack overflows) we have to do garbage collection of the stack (which is slow, but not as slow as heap garbage collection)".

It's similar to how vectors/arrays work: when you insert elements into an array, most of the time it is extremely fast. But sometimes it has to resize the array, which is very slow. The idea is that because the resizing happens very rarely (usually exponentially rarely), it "averages out" to be very fast in practice. The same principle applies to hash tables. Amortized performance is extremely common.

I'm just trying ensure it's on record, because it seems all too easily lost/ignored/overlooked in these discussions, that the need for PTC goes well beyond recursion.

I agree with that: PTC is not merely an optimization, it is a feature of the language which enables new kinds of programs to be written (including, but not limited to: state machines, CPS transforming compilers, new flow control constructs, call-with-continuation, async code, etc.)

---

@Odrinwhite Don't get me wrong I would like to see your version implemented but I think the argument that it breaks the web may be valid here for some people.

As far as I can tell, TCO/PTC does not break the web, as indicated by Safari. The resistance to PTC is not about breaking the web, it is because PTC makes the JS engines more complicated, and it possibly has some performance penalties as well.

[elibarzilay]

It seems to me that it will be helpful to clarify the purpose of tail call elimination: the main goal is not to reduce runtime, but to reduce memory use --- and the actual stack frames are not the main concern there. When I write this code:

function foo(x) {
  let y = ...;
  return bar();
}

then having PTC means that when bar() is called I want to know that the x and y bindings no longer exists, and can be GCed.

[This is the main thing that Guido missed (or more likely chose to miss) in his shutting down of tail calls --- the example he uses makes it a requirement to keep x and y for debugging. So if the "imperative viewpoint" is to keep this information for the sake of debugging, then the "functional view" would raise an eyebrow and ask why is the old value of x after x = f(x) is not preserved by the same coin. (Or more practically, view saving these values as a debugger de-optimization.)

It's also interesting to see Guido's initial bottom line, "After all TRE only addresses recursion that can easily be replaced by a loop", is something that is clearly questionable to modern JS programmers (for example, trying to convert a tail-recursive promise loop into a while), and that's besides the added fragility of an imperative loop over using tail calls.]

[glathoud]

Here is one possibility for developers to use fast tail calls today, without needing extra keywords in JavaScript: https://github.com/glathoud/fext

[kaizhu256]

+@douglascrockford

recent jslint versions have 2 tail-call "bugs" where it raises RangeError in certain cases in v8/nodejs/chrome (but not safari) [1], [2]. these bugs were closed by the author as "wont fix" (under assumption bugs will go away when engines eventually fall inline with ptc-spec).

will that assumption happen in the forseeable future? yes/no/maybe? an authoritative answer would provide clarity/guidance to development-roadmap of jslint.

[1] jslint-issue - RangeError when linting excessive leading-newlines
jslint-org/jslint#244

[2] jslint-issue - RangeError when linting json with large string-values
jslint-org/jslint#239

[kaizhu256]

oh also, firefox doesn't seem to have jslint tail-call issues under normal use-cases - because it has a ridiculously large callstack-depth limit (~100,000 vs. chrome's measly ~12,000). you can get these numbers by running the script below in the respective browser's console.

maybe a happy-medium/short-term solution is for v8/chakra to raise their callstack depth-limit to the same level as firefox?

function computeMaxCallStackSize() {
    try {
        return 1 + computeMaxCallStackSize();
    } catch (e) {
        // Call stack overflow
        return 1;
    }
}
computeMaxCallStackSize()

[slikts]

Why would you use jslint, though.

[kaizhu256]

@slikts because eslint is too slow on the ultra-portable laptop i use while backpacking. javascript product-development should not follow java's model where the only reason you need powerful laptops (with poor battery-life) is because of bloated-tooling.

also i don't have strong feelings about PTC either way. i'm just unhappy that jslint is currently in a state thats not ideal for production-use, and wish its author and nodejs/tc39 can come to some compromise to resolve that. raising v8's stackcall-limit to firefox's seems like something reasonable to me.

[ljharb]

This repo isn’t the place to complain about, or seek change in, any of those things. jslint’s author by their own admission doesn’t care about web reality, only what’s in the spec; file a bug on Firefox if you want them to implement something; use something other than eslint if you like, just be aware that eslint is the de facto standard for “product-development” in the entire JS community.

While PTC is in the spec, STC (this proposal) is untenable, and there’s nothing of use to discuss here.

[kaizhu256]

@ljharb, my understanding is this thread is about PTC (not STC), and i'm bringing up a PTC issue in the wild.

[ljharb]

The thread is also not appropriate for this repo, since this entire repo is about removing PTC and replacing it with STC.

You’ll have to to take it up with each engine that has chosen not to ship PTC if you want PTC “revived”, or with the one engine that has if you want it removed.

[getify]

PTC says all such call frame must be eliminated.

That is not actually what's required. Here's the relevant part of the spec:

A tail position call must either release any transient internal resources associated with the currently executing function execution context before invoking the target function or reuse those resources in support of the target function.

First, this doesn't require elimination of a frame. It only talks about freeing up resources. That's a subtle but important difference, which implementations like Safari seem to have taken to allow things like the Shadow Stack.

But further, I maintain that rather than being concerned about the letter of the law, the spirit of the law is the most important thing here. What the spec is getting at is less that there must strictly not be any extra resource allocation, but rather that the allocation growth must be O(1) instead of O(n). I would like TC39 to reword this section in this respect, as it would clear the way for lots of other creative ways of accomplishing the bigger goal of tail-calls, which is that I could make any depth of call stack in my program without fear of running my device out of memory.

That's why people want tail calls, not that they explicitly want every single non-essential stack frame thrown away.

[kaizhu256]

Lastly, for my information, what is what is "Guido-style rejection" ?

python's creator [in]famously rejected TCO in a 2006 blog-post [1]:

"This is also the reason why Python will never have continuations, and even why I'm uninterested in optimizing tail recursion."

it caused some brouhaha back in the day (similar to what's going on in this thread) [2].

[1] original-quote from python-creator's 2006 blog-post "Language Design Is Not Just Solving Puzzles"
https://www.artima.com/weblogs/viewpost.jsp?thread=147358

[2] backlash to [1] on [Python-Dev] mailing-list
https://mail.python.org/pipermail/python-dev/2006-May/064817.html

[3] python-creator's response to backlash (with further community "feedback" in the comments-section)
http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html

[elibarzilay]

@kaizhu256: it might be that some promises are forced in async-ly in a different tick and therefore there is no problem with tail calls, but that might not always be the case. For example, this code (or some similar variant):

const loop = n =>
  new Promise((res,rej) => res((n > 0) ? loop(n-1) : Promise.resolve(n)));

should be a loop even though there's no such deferring. But more generally, in the simple case of

const loop = n => foo(() => loop(n-1));

I'm relying on foo to call my callback in a tail position, and if syntactic tail calls are a thing, that means that I now need to ask foos author nicely to fix the code.

@concavelenz: (1) yes, that was the point I made later: losing bindings from tail-calles is the feature of reducing resource use; (2) if there are "ways to record context", then wonderful -- tail calls could use the same tools.

For (3), quite the opposite. The statistical sampling fits perfectly with tail calls, if f does almost nothing before tail calling g it will not be shown -- at this point the developer is concerned about runtime and g (the main focus) is still just as visible. If, OTOH, f does a non-tail-call to g (or of no tail call elimination is done), then you get what you have today: you would have an f block and on top of that a g block that is the same width, and you need to infer from that that f is noise and the real runtime hit is from g.

Finally, in those let a = f(); return foo(); cases the runtime does need keep a, because it might be used in a stack trace. A tail call means that it can be GCed.

@concavelenz / @kaizhu256: yes, it's that mess that I'm referring to, but IIRC the relevant post had some "final" in its title and that's when Guido dropped a concrete lid on it.

[concavelenz]

@getify I fully understand why folks want tail recursion. The text requires that everything be reused or released, nothing remains of the stack frame. The implementation of the shadow stack has nothing to do with that text.

@elibarzilay RE: (2) recording the context for every function that has a call in the tail position is impractical, as it involves recording the stack trace, which is expensive. Doing this for timer, which are far rarer, is sometimes reasonable.

RE: (3) You imply that a you only care about the runtime of a function, but in my experience how you reach the function (why it was invoked at all) is usually as important that the cost of the function proper.

[elibarzilay]

Well, you brought it up... It can be expensive, but you have mentioned the optional shadow stack thing...

And in that context I obviously cared only about runtime, since I was talking about one common concern about the usability of flame graphs. (And further, in a statistical profiler context you generally don't care about precise information which is why the random stack polling works fine.) The "why it was invoked" can be addressed using tools that were mentioned many times in this thread; tools that can be used for the sake of skeptics who don't believe functional programmers who say that in practice this is not a problem. (Discussing why it's not a problem in practice will further derail this thread so I'll avoid doing that.)

[hax]

This thread is very unfortunate that nothing we (programmers) can do except waiting. But I still want to share my thought about PTC vs STC.

1. I agree that any action is better than no action.
2. I think tail call is important for FP, without it, JS can never have real support FP even we add many FP features like pipeline operator.
3. I prefer STC. The problem of PTC is, it's too implicit. If someone write a code rely on PTC, it could be easily ruined by others because we don't know whether a tail call is intentionally.

Example:

function f() {
  if (x > max) return
  ++x
  return f()
}

This is a valid PTC code. Consider if we introduce consistent-return rule, ESLint will complain. How to fix? Someone will just change return f() to f(), and break PTC, and may introduce a potential bug which normal test cases can never find --- even you have a test case for it, you need a correct max to trigger stack overflow, it should be big enough, but not too big (for speed of test running, especially if you run tests on low-end mobile devices). You may need config different max for different user agent, and new versions of them could just increase their stack and fail you...

[sbuller]

'consistent-return' seems a little misguided here. Proper type checking reveals that the types remain consistent in your example.

I regularly notice when my code would benefit from TCO, and it's disappointing to know that there's this misguided resistance to it. There's a long history to TCO, and its absence in JS is a serious deficiency.

[pygy]

What more, the technical reason for not adding them (the ChakraCore engine having the wrong calling conventions hardwired) is soon going away since MS is co-opting Chromium.

The other technical problem isn't really one (cross-realm calls can't be eliminated in Firefox) since that can be spec'ed around.

[jswalden]

Last I heard, ChakraCore will continue to exist as a JS engine for use in any number of projects not including a browser, so that point doesn't quite work out that way.

[sarimarton]

Once PTC starts to spread in practice due to its actual availability, and it eventually becomes a problem to identify TCs in code, IDEs will very quickly kick in with syntax highlight help - just as they do with dead code by fading it.

[hax]

@sarimarton Not everyone in a team use same IDE/editor. And lightweight editors, tools may not identify TCs easily without full AST parsed.

[kaizhu256]

i write code exclusively in vim (and use a customized version of jslint that allows ignoring foreign code-blocks)

[littledan]

I have a feeling that we might be able to make more progress on explicit tail call syntax in JavaScript. If everyone able to think through and agree on a syntax, I think it'd be worth bringing back to TC39 to see if we can get consensus on pursuing this proposal.

[elibarzilay]

@sarimarton, yes -- identifying tail calls is in most cases extremely simple, almost to the point of regexp-ing it. With a parser it becomes straightforward.

@littledan, the problem is not in agreeing on a syntax -- the real problem is the fact that "opt-in" syntax is going to nullify much of the point of tail calls. See the last bullet in my original post above.

[littledan]

Well, if we can't agree on a syntax, I imagine the current situation will remain as is.

[elibarzilay]

@littledan, please see that comment that I pointed to: the problem is not the syntax, it's the fact that you'll need to modify code to "opt into" a tail call, which would lead to situations like what I describe there. (IOW, this is an objection to any explicit syntax for tail calls.)

[ljharb]

I'm still not clear on why that's an obstacle - libraries needing to change their code is a temporary problem, as new code would theoretically be written with the explicit syntax where desired.

[elibarzilay]

I'll try to explain it with an actual (but very simple) example.

• I implement a useful text-lines library, which has:

  // calls f on a list of lines from the file
  function withLines(file, f) {
    let lines = file2lines(file);
    return f(lines);
  }
  

• You use this libray in:

  function waitUntilItsOK() {
    withLines("some-file", lines => isOK(lines) || continue waitUntilItsOK());
  }
  

• Your code is perfectly fine, except that it's broken because my code doesn't use continue. You'll ask me nicely, but I don't want to complicate my code with strange keywords, or I need the code to work on older JS engines, or my company won't allow me to invest time to change code that already works, or I don't like this new "PTC" feature, or I heard that it code run slower, or whatever. You're now stuck.

In a theoretical world where none of these reasons hold, everyone would eventually add continue at all places that matter — but then what was the point of adding an extra keyword in the first place?

The only "real" answer to that is to make it easier to verify that what you think is a tail call, is ineed one. IMO, this makes exactly the same sense that the "capture clause" in C++ lambdas does. And for all I know, you might like that feature (maybe because you're doing C++ at google). But like I said previously, in the current non-theoretical world, the much likelier result is that tail calls will not be widely used (if only because of the natural bias against longer code).

[ljharb]

You could also imagine the syntax enabling proper tail calls for the entire stack it generates - avoiding the need to add the keyword all the way down.

[elibarzilay]

That sounds interesting, but it's not what's suggested in this proposal...

(Sidenote: I imagine that you're talking about something that happens at the beginning of the stack since going the other way is impossible. Something that I thought could work is some "use strict and tail" declaration where all code under it creates a dynamic stack context that makes all later calls be tail calls. If that's what you're talking about then that sounds like great idea. Again, regardless of actual syntax: I don't really care if it's a top "use strict and tail" or some (()=> continue { ... }) wrapper around my whole file.)

[PinkaminaDianePie]

Enabling the entire stack it generates? Why not go other way and have PTC but provide unfolded stack trace when required? So my app will have PTC, but if i'll struggle with something, I will be able to enforce unfolded stack trace, debug my app, and disable it back again? But having ETC keyword which enables PTC for stack trace is fine enough if I will be able to do it once at the root of my app (e.g. in index.js) and work with PTC everywhere. Anyway, explicit TC, which I will be able to use only once at the top of app will be good thing, but have it explicit everywhere will be much much worse

[getify]

I prefer "implicit PTC" (current spec) but with revised wording that would widen the possibilities for various engine implementations, as discussed earlier in this thread.

However, it seems almost impossible for that to ever happen from the current state. The years-long stalemate is bad for JS and for engines.

As for whether TC39 could remove PTC without consensus, I have changed my mind from earlier positions: I now strongly feel that TC39 can, and should, remove PTC from the spec. This needs to happen first before any possible progress on tail calls can happen.

The webkit folks don't have standing to object to the removal, as this doesn't betray or invalidate their implementation at all. Had PTC never been added, I believe webkit could have added this "feature" as an optimization without being in spec violation. So even if the spec part is removed, that doesn't harm webkit.

I believe the editor of the spec should make an executive decision that any spec'd feature (or typo or bug or wording or ..) which is later openly defied by implementation(s) -- irreconcilably so -- is a post facto veto, and therefore can and must be removed, retroactively stricken from the standard.

I imagine there may be some "legal" objection (ECMA, etc) to that, so in effect the next best thing is just to inline the change in the next spec revision without need for consensus.

In any case, we need to do a reset on this topic so we can then have productive discussion about any possible path forward.

[ljharb]

That is not within the power of an editor - every delegate has standing to block anything for any reason, including removing PTC from the spec. Consensus is required for normative changes, which is why we’re in the stalemate we’re now in.

I agree that no progress can be made until PTC is removed.

[fselcukcan]

I just do not understand why we cannot have an important programming feature like TCO (not the syntactical proposal, but natural as it supposed to be) due to some claims that it has negative effects on somethings which does not have much ground as we have seen in different threads so far.

• debugging
• education?. Gee, Are you serious!?
  Ok sorry let me rephrase that. Gee, are you serious!?
• what is next, health?

Shortly, for debugging I can say;
We debug code when we debug code, not when we run code.
Is it not possible to have a switch like Disable TCO (when Developer Tools are open, that is when we debug)? We have this for disabling caches Disable Cache (when Developer Tools are open, that is when we debug).
Disable Tail Call Optimizations (When Developer Tools are open) #25

[mlhaufe]

Gilad Bracha has shared some advice on how to have your cake and eat it too with PTC. Stack Compression, Heap Allocation, Ring Buffers, etc.

https://gbracha.blogspot.com/2009/12/chased-by-ones-own-tail.html

[douglascrockford]

TC39 made a mistake in naming the feature 'Proper Tail Calls'. Lots of coders have no interest in propriety. It should have been called 'Tail Call Optimization'. Coders will jump through all sorts of hoops to optimize, even when that work has no observable benefit.

The only change TC39 should make is to correct the name of the feature. Meanwhile, all JavaScript engines should implement it in order to be called Standards Compliant, and debuggers should be enhanced to mitigate the debugging experience.

[ljharb]

Standards document reality, they don’t dictate it, and the proper thing to do would be to remove PTC from the spec, and attempt to reintroduce it through the modern proposal process.

[getify]

the proper thing to do would be to remove PTC from the spec

I asserted years ago that TC39 could/should adopt the position that any feature in the spec which is willfully violated by the majority of implementations is de facto already voted down, and should thus be removed/demoted (back to stage 2 or 3) as an editorial change, requiring no further consensus vote.

Even if webkit still wants to keep its PTC implementation, they should be allowed to do, because there's no current assertion being made that the spec would include a prohibition on PTC. It could/should just be an experimental feature that webkit implemented that distinguishes them from the other engines. Therefore, it doesn't require a consensus vote to remove the specification of the feature.

If PTC/TCO was demoted to stage 2 (or even 1), it would (as @ljharb suggests) be given the time to go through a more rigorous exploration and specification cycle. If it ever reached stage 4 again, it would certainly have done so after convincing all of its detractors, and we'd be much more likely to get actual implementations of it (beyond webkit). Of course, webkit could still veto any substantial design changes (such as STC) while it was in those earlier stages. But that should be a healthy debate process that's allowed to happen, instead of the stalemate we currently have.

---

Side note 1: I would feel differently if two or more implementations had a current shipping implementation... in that case, I would argue that TC39 must at a minimum move the feature specification into appendix B, in the spirit of documenting web reality even if the specification doesn't necessarily endorse some feature or behavior as a first-class citizen.

Side note 2: I still absolutely strongly feel JS should have PTC (or TCO -- yes, I'm agreeing with Doug). My assertion above should not be construed as being against the feature, but rather against the (IMO) unacceptable indefinite standoff and willful violation by most engines.

[elibarzilay]

1. There is no point talking about "majority of implementations." It's very clearly a term that could freely be substituted with "Google" -- and seen in this light, leaving anything for the majority is just serving at some idealistic "community forces that shape the language" when in fact it's a single-company dictatorship, and what you suggest should be an obviously bad idea to most sane people.

   (That "indefinite standoff" was almost resolved when this exact dictator was going forward with an implementation, and then decided to drop it for reasons that are not visible to outsiders. So yeah, there was no discussion on the merits of the feature, just a bunch of the usual misguided FUD as a weak explanation --- and no way to properly comment about or discuss that conclusion. This further demonstrates why it's nonsensical to leave it for "the majority" when that majority is one company.)

2. There is also no point in trying to make it into some kind of a symmetrical "both sides" argument WRT enforcing no tail calls. How would such an enforcement look like? Any way of defining this would require going down to implementation levels which doesn't make sense for a language spec. For example, "must maintain a stack", "must throw a stack overflow error after N calls" are both nonsensical for a spec.

3. Re the name, "tail call optimization" is a bad term that makes pretty much anyone confused about the motivation and the expected results. Example: way too many people hear "optimization" and immediately conclude "runs faster", and when they learn that it doesn't, they see no point in it. As usual, an "optimization" is not a reliable language feature but an opportunistic way of running your code in some better-than-expected-way, and if it's beyond the scope of the compiler then you can't really complain. "Proper tail calls" (or "tail call elimination") are a language feature that allows writing code in a way that could not be done without it: IOW, you write code that depends on this feature, and therefore it is not "an optimization".

4. Finally, re "it would be given the time to go through a more rigorous exploration and specification cycle" -- that's bogus in this context too. The feature itself is very well-researched, well-used, and overall well known. It had time to "mature" since the 70s. Everything that could be explored around it had been explored, many times in many contexts. As productive as TC39 could ever dream of being, it pales in comparison to decades of actual research and actual use, so it's only logical for them to defer to existing work instead of "exploration". That holds also for specification: the actual spec of tail calls is relatively small, and not too complicated.

   What keeps astonishing me is the one thing to take from the fact that there is one "relatively known" browser that has had tail calls for a while now, and that hasn't lead to the total disaster of destroying debuggability which is at the core of the anti-PTC crowd -- and in fact was the very same reason given by that company when they dropped the work that they already did toward an implementation (though the droppage predates the webkit implementation).

   Javascript has had a major surge in functional-style work in the last decade, and it's about time that people will be more vocal about getting the last bit that makes it possible to use functional style over being forced into while loops.

   (And perhaps a slightly more anecdotal point: one of the main problems with syntactic tail calls is that they abolish a substantial part of the benefits. One of the major points of PTC is that it "just works" even through a HO function from an independent library. Suggesting a syntax for tail calls (especially one as obnoxiously verbose as return continue is guaranteed to keep people away from the feature, which means that the benefits would be only the ones that are limited to your own code. And as long as you care about just your own code, then switching your healthy thought process into the sick world of stateful loops and such is something that is doable -- and effectively enforced by the dictatorship (since nobody sane would require a non-v8 runtime). And so the wheel spins year after year with no light on the horizon.

   The bottom line here is that it took decades to get to this point: dropping the requirement to some "in the future" appendix is going to take it further back.)