repl: allow await in REPL

[benjamingr]

I've seen this requested a few times in StackOverflow now:

People have code that is asynchronous and would like to use values from it in the REPL:

// how do I access in the REPL? returns promise
Transaction.where('reference', '1').fetch().then((res) => { return res }); 

A common workaround is to store a global reference in the REPL, which is a little hacky and relies on timing, but can work for simple cases:

> Transaction.where('reference', '1').fetch().then((res) => out = res)
[Object Promise]
> out
   /* response available here*/

This works, but has timing issues and generally isn't great. We could however run the REPL in the context of an async function potentially - which would allow awaiting values in the REPL:

let res = await Transaction.where('reference', '1').fetch(); 

I think it could be an interesting enhancement but would like more feedback on the idea. @nodejs/collaborators

[not-an-aardvark]

👍

I looked into implementing this when Node 7.6.0 came out and async functions were originally added. At first glance it seemed like it would be a bit difficult because the REPL currently doesn't run in a function context at all. However, there might be a better solution that I'm not aware of.

[refack]

I'm +1.
But there are userland solutions, quick googling found https://github.com/skyrising/await-repl, https://github.com/StreetStrider/repl.js, and I'm sure there are more

[not-an-aardvark]

await-repl seems to only await a promise if the line starts with "await ", which is probably insufficient (e.g. it doesn't allow foo = await bar().

repl.js seems to auto-await all Promises that are returned from an eval, but that also probably isn't what we want (and it doesn't allow for nested await either).

[refack]

I'm not arguing that this is a beneficial feature. But IMHO the REPL is an exploratory tool, so those tools at least enable easier exploration 🤷‍♂️
On the other hand the cli -e option, again IMHO, is a more "important" feature to enable easy await usage for.
node -e "await new Promise.resolve(4)"

[vkurchatkin]

The question is, is it really possible? It seems like it's not

[not-an-aardvark]

It would definitely be possible in theory by reimplementing await semantics and parsing, but that seems like a lot of work. Ideally there would be some --allow-top-level-await flag in V8 that we can use.

[joyeecheung]

There was a feature request issue for this and there are some solutions proposed there: #8382 although it seems most of them are somewhat hacky

[benjamingr]

@vkurchatkin

The question is, is it really possible? It seems like it's not

Why can't we just wrap the content with an async function?

[ChALkeR]

@benjamingr, that will change the semantics of repl. We need to detect it somehow, or use a flag, or force this as a semver-major change for everyone.

[benjamingr]

@ChALkeR what semantics would it change? I figured that since the REPL is not synchronous anyway (being user input and all) the microtick of wrapping it in an async function - unwrapping it and giving the user back the result wouldn't be too bad.

Am I missing anything?

[targos]

@benjamingr do you have ideas about how to make this work:

> const a = Promise.resolve(42);
undefined
> const b = await a;
undefined
> b
42

If we wrapped the code in an async function, b would be local to this function. How to unwrap it?

[benjamingr]

@targos you're absolutely right! I totally missed the scoping issue.

Parsing the line and figuring out if it's a variable declaration seems easier than detecting an async function, it would still be problematic for things like f(); const a = 5; which are legit.

I'm wondering if we could have an escape hatch at the V8 level perhaps? Optimally something that lets V8 know it should evaluate the code passed as an async function?

Essentially, we'd need a way for Script to take an async function (well, the Script::Compile method).

I'm wondering if that's possible - @nodejs/v8 @fhinkel ?

[bergus]

This also needs some kind of cancellation option - like Ctrl+C - to stop waiting. Something like

await new Promise(() => {}) // forever pending

should not break the REPL

[Fishrock123]

This seems like an odd idea to me, what does the repl do during an await? Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Either way seems confusing and poor design to me, maybe I'm not considering something?

I suppose it's not much different that running a long sync operation although that is probably less common.

[Qard]

Yeah, I think stalling is the desired behaviour. The idea is to be able to inspect the return values of a promise API and use them for subsequent operations in a more friendly way. Currently you have to attach then() handlers to write values into global s and check repeatedly to see when they become available. It's very unfriendly to promise users.

[refack]

Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Maybe show a spinner?

[addaleax]

Does it just stall and not respond to further user input? How do you know when the await is completed otherwise?

Maybe show a spinner?

Sounds good. As long as you can see that something is happening and you can abort it with Ctrl+C, we should be good. :)

[vkurchatkin]

Maybe show a spinner?

Sounds inappropriate for REPL. Aborting is a must, though. That said, I think it's a waste of time to discuss such details until we figure out a way to do this.

[bmeck]

at some point people might want to think of foreground and backgrounding await if you need to test how 2 async things interact... like Jobs in shells

[refack]

foreground and backgrounding await

Isn't there a key for that in bash? CTRL-Z? than also jobs and bg and fg? (not a bash expert)
We already have .* REPL commands, so .jobs / .fg / .bg doesn't sound absurd...

[RReverser]

It would definitely be possible in theory by reimplementing await semantics and parsing, but that seems like a lot of work.

FWIW that's what regenerator already did (as used by Babel for transpiling generators and async-await to ES5).

So for a problematic example above, wrapped into an async function:

(async function() { 
 const b = await a;
})();

it would generate

(function _callee() {
  var b;
  return regeneratorRuntime.async(function _callee$(_context) {
    while (1) {
      switch (_context.prev = _context.next) {
        case 0:
          _context.next = 2;
          return regeneratorRuntime.awrap(a);

        case 2:
          b = _context.sent;

        case 3:
        case "end":
          return _context.stop();
      }
    }
  }, null, this);
})();

From here, we could easily unwrap the IIFE and execute body, which would solve the scope propagation issue as all the modified variables would be visible after the body is executed.

However I'm not sure if we would want to integrate such, even minimal, third-party transpiler into Node.js REPL.

But if we do, I could try to make this happen :)

[chicoxyzzy]

IMO REPL should be predictable. await is valid identifier outside of async function. So it should work as identifier.

[addaleax]

However I'm not sure if we would want to integrate such, even minimal, third-party transpiler into Node.js REPL.

If you can make this happen and the code doesn’t make the node binary turn huge, I would be in favour of doing that.

[ebraminio]

I think you should consult some of v8 guys for help and taking best strategy here as the same feature would be nice for Chrome console as well so a similar approach could be used for node.js I guess.

(BTW I am very happy that this is proposed here as the idea of top-level await itself seems is rejected but all I personally liked to have was a console one)

[ebraminio]

Filed as http://crbug.com/727924 Edited: already available on http://crbug.com/658558

[ebraminio]

According to this comment and my test, this is already supported on Safari so I guess Chrome team should/will soonish pick up the task to fill the gap up with the other browser.

[esteban-uo]

thanks for supporting it!. I don't know if it is just me, but in Node 10.1, it just hangs out while using REPL await... anyone else experiencing the same?

[vsemozhetbyt]

@esteban-uo It seems you need --experimental-repl-await for this now.

[esteban-uo]

@vsemozhetbyt yes and actually it works just fine 👍 Seems to be it happens when an exception occurs, which is very confusing sometimes, since you could think that the promise can't be resolved at all or a timeout is happening

[domainoverflow]

Forgive me if I am misreading but according to the above I would be able to CLI node app.js where app.js could contain awaitS placed outside functions ( ex: copy and paste from puppeteersandbox.com examples without wrapping and cli running it ) ?

[Krinkle]

@domainoverflow Thank you for asking!

This issue is only about using "await" in the special "REPL" mode of Node.js. If you run node as CLI command without JS file, that is the REPL mode. This mode behaves similarly to the DevTools console in a web browser.

REPL is a different from how programs run normally. And for JS programs, it is not possible to use await outside async functions because the JavaScript programming language is specified in a way that does not allow this. There is a proposal from TC39 to allow "await" outside functions in normal programs. More about that at https://github.com/tc39/proposal-top-level-await. This was relatively easy to do in REPL for debugging and testing, but is less easy for normal programs. If the proposal is approved, then the JavaScript engine for Node.js (Google V8) may implement the change, after which it may eventually become available in Node.js.

[techsin]

+1

[jiayuzhang]

@esteban-uo It seems you need --experimental-repl-await for this now.

Any way to specify the experimental flag programmatically (instead of command line)? since I'm using repl.start api

[vsemozhetbyt]

You can try NODE_OPTIONS=options...

[mariapryimak]

@jiayuzhang in linux repl module will inherit the flag --experimental-repl-await, and replServer.start() will have top level await, but it seems like it doesn't inherit it in other OSes and it doesn't work. Looks like a bug.

[filips123]

Is there any specific reason why is await under an experimental flag? Why is it not enabled by default?

[devsnek]

@filips123 when await is enabled we run input through a separate parser which doesn't necessarily match the way node normally parses code, so there could be incompatibilities. In the long run, a combination of top-level-await and a standardized repl parse goal should fix this.

https://github.com/tc39/proposal-top-level-await
https://github.com/bmeck/js-repl-goal

[FossPrime]

Summary, it works with an experimental flag. use something like the following to load some js and drop you into a repl after the variables are set, kind of like bashrc.

#!/bin/bash
node  --experimental-repl-await -i -e "$(< scripts/repl.js)"`

With enough run commands, it might be possible to replace bash with node lol
https://stackoverflow.com/questions/33850903/use-node-js-as-shell

[nahtnam]

Has anyone figured out how to get the experimental flag to work in a REPL that is spawned with .start()? The following snippet doesn't work.

process.env.NODE_OPTIONS = '--experimental-repl-await';
const local = repl.start({ prompt: '> ' });

[aherlihy]

Having the same issue as @nahtnam with trying to get await mode turned on from repl.start. Any suggestions?

[bapti]

I did this and it worked

"scripts": {
    "repl": "node --experimental-repl-await ./repl.js"
}

And in repl.js

const { readFileToArray } = require("./utils");
const repl = require("repl");

repl.start("> ").context.readFileToArray = readFileToArray;

When I run it it works fine and I can await on my function readFileToArray

[tomasgvivo]

Top level await is now supported in V8 (https://v8.dev/features/top-level-await)
Is it possible to add something like vm.runAsyncInContext to avoid using a different parser for code containing async in repl?

Also, I could really use vm.runAsyncInContext for a jupyter-notebook-like node project.

[devsnek]

@tomasgvivo TLA is part of modules. Once we land support for it (#30370), you can use it in a vm.SourceTextModule instance.

There is another thing V8 is working on called "repl mode" which might help with await in the repl though: https://docs.google.com/document/d/1bIKMRrLVObsAfmMAQ-tmsWkQZu1cFg87RRrPpiwTL2M/edit

[cyrus-and]

When the adding the command line --experimental-repl-await is not an option, one could use this hack to always await promises on a custom REPL:

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
    defaultEval(cmd, context, filename, async (err, result) => {
        if (err) {
            // propagate errors from the eval
            callback(err);
        } else {
            // awaits the promise and either return result or error
            try {
                callback(null, await Promise.resolve(result));
            } catch (err) {
                callback(err);
            }
        }
    });
};

It is not a general solution but worked well for my use case.

[YoraiLevi]

When the adding the command line --experimental-repl-await is not an option, one could use this hack to always await promises on a custom REPL:

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
    defaultEval(cmd, context, filename, async (err, result) => {
        if (err) {
            // propagate errors from the eval
            callback(err);
        } else {
            // awaits the promise and either return result or error
            try {
                callback(null, await Promise.resolve(result));
            } catch (err) {
                callback(err);
            }
        }
    });
};

It is not a general solution but worked well for my use case.

// index.js
const repl = require("repl");
const myRepl = repl.start();

const defaultEval = myRepl.eval;
myRepl.eval = (cmd, context, filename, callback) => {
  defaultEval(cmd, context, filename, async (err, result) => {
    if (err) {
      // propagate errors from the eval
      callback(err);
    } else {
      // awaits the promise and either return result or error
      try {
        callback(null, await Promise.resolve(result));
      } catch (err) {
        callback(err);
      }
    }
  });
};

usage node index.js
type:

function delayPromise(delay){
    //async delay function
    return new Promise(resolve => setTimeout(() => resolve(), delay))
}
delayPromise(3000)
undefined
//wait 3 seconds until can proceed typing

[andyvanee]

I'm not sure why, but using --experimental-repl-await affects how util.inspect.defaultOptions are interpreted. In normal code you could adjust the output of console.log using the following:

import util from "util"
util.inspect.defaultOptions.getters = true
util.inspect.defaultOptions.depth = 4
util.inspect.defaultOptions.maxStringLength = 80

But when using --experimental-repl-await it needs to be changed to this:

import util from "util"
util.inspect.replDefaults.getters = true
util.inspect.replDefaults.depth = 4
util.inspect.replDefaults.maxStringLength = 80

The docs seem to suggest that the replDefaults should mirror defaultOptions, but this doesn't seem to be the case when using the async version of the REPL.