Make lumo repl "upgradable"

[cgrand]

Add a public asynchronous input-stream (of characters/strings) that would make upgradable REPLs possible. Ideally it should work in the non-socket repl too.

[cgrand]

The thing is that the repl has to be synchronous all the way down:

• read because IO is asynchronous
• eval because code in eval might do async stuff (like calling read or doing some IO)
• print at least when dealing with sockets

[hlolli]

@cgrand is this what is needed for unrepl to be possible for lumo? Also your comments of synchronicity, as far as I see now, the repl seems to append it's commands at the beginning of the event queue, as via process.nextTick(), this I think is true due to the behaviour I experience here, https://github.com/hlolli/lumo-repl-noqueue-benchmark where the repl IO's comming from the repl is always of higher precedence than running loops.

But I want to rethink the repl a bit, as it is now it's causing me problems, but if async/await operations on top of setImmediate() would set repl evaluations at the back of the event queue but bring a promise to the return value, that could be a solution.

I'd like to see unrepl implemented but at the same time not wanting to see the repl distracting already running tasks.

[cgrand]

@hlolli yes it is what is needed for making possible to "upgrade" a repl. But my comments was a mix of brain dump and thinking out loud -- sorry for the spam.

Let's try to put things in an intelligible order:

1. To upgrade a repl you need to take control of the repl event loop. In sync systems, it's easy: you just don't return.
2. A rough draft of an async REPL should be:

(.once in "data" (fn [chars] (.write clj-reader-pipe chars)))
(.on clj-reader-pipe "data" (fn [form] (eval form ctx ... completion)))
(defn completion [v ex]
  (prn (or ex v))
  (.once in "data" (fn [chars] (.write clj-reader-pipe chars))))

So the trick is to have once-like semantics (the loop has to complete to re-read) and to be sure that evaluated async code (e.g. another loop) can trigger the completion. So the completion callback has to be passed around or has to be global.

Passing around is awkward, tedious and error-prone. Having it global but you have to think that "upgrade loops" may be nested. So the completion callback must have stack semantics.

Given that we get a second draft:

(def completion-cbs #js [])
(defn completion [v ex] ((.pop completion-cbs) v ex))
(defn start-loop []
  (.push completion
    (fn [v ex]
      (prn (or ex v))
      (start-loop)))                          
  (.once in "data" (fn [chars] (.write clj-reader-pipe chars)))
(.on clj-reader-pipe "data" (fn [form] (eval form ctx ... completion)))
(start-loop)

This completions stack is a kind of continuation (or stack of).

However all the above is slightly wrong because you may need several chunks of chars to have a whole form.

Third draft

(def completion-cbs #js [])
(defn completion [v ex] ((.pop completion-cbs) v ex))
(defn start-loop []
  (.push completion
    (fn [v ex]
      (prn (or ex v))
      (start-loop)))                          
  (.once in "data" (fn [chars] (.write clj-reader-pipe chars)))
(.on clj-reader-pipe "data" (fn [form] (eval form ctx ... completion)))
(.on clj-reader-pipe "need-more" #(.once in "data" (fn [chars] (.write clj-reader-pipe chars)))
(start-loop)

I would tend to merge in and completions together.

(Thanks for pushing me to explain, it really helps!)

[arichiardi]

This really looks cool and scary at the same type @cgrand 😄

Do you think it could be turned into an event-based state machine? Because it would probably be easier to read and reason about IMHO.

[cgrand]

At last, I've been able to give a second try at it and I have a concrete proposal which differs from what I proposed earlier.

This file is the public interface -- could be spawned as a lib so that all cljs.js impls can include it.

The public interface is made of an AsyncReader protocol (not a lisp reader, an asynchronous stream), a suspend-host function (which returns a suspension request), suspension? and yield-control a function that allows hosts to yield control to the emitter of a suspension request.

So there's no public *in* because it was awkward having it global but requiring some exclusive ownership. Now the input stream (an AsyncReader) is passed by the host to the requester via yield-control. This makes input ownership a bit more linear (from a syntax point of view) and thus easier to track.

How does it impact lumo? Well in repl if the result of evaluation is a suspension the repl must yield control to the suspension and resume when the suspension completes (by calling the resume-host callback).

So including this file and making lumo.repl to handle suspensions is all it's required for making lumo upgradable.

[cgrand]

plain-repl buries the lede (there's the api, a sample REPL using it, an implementation of the api for lumo etc.) so I'm going to fork lumo to show how it could be done.

[cgrand]

See this branch of my fork https://github.com/cgrand/lumo/tree/suspension for a working implementation. Feedback welcome.

Example of an upgrade:

(lumo.repl/suspension-request
  (fn [r resume]
    (let [print-fn *print-fn*]
      (lumo.repl/read-chars r
        (fn loop [s]
          (if (re-find #"bye" s)
            (resume {:value :ok})
            (binding [*print-fn* print-fn]
              (println "you said: " s)
              (lumo.repl/read-chars r loop))))))))

Clojure equivalent:

(loop []
  (let [s (.readLine *in*)]
    (if (re-find #"bye" s)
      :ok
      (do
        (println "you said: " s)
        (recur)))))