Implement bquote macro in Bel itself

[masak]

Note: Builds off of #173, which I expect will merge first; please rebase this one on master after that. Thanks. Rebased.

Work in progress — so far, managed to late-bind the bquote call, but it still calls into a Perl version of the expander. Next step: define it among the globals.

Closes #174.

[masak]

Making this work has to count as the most mysterious experience in the development of Bel so far.

1. After re-generating the globals in ed1b391, a second generation of the globals yields a different result. This instability was always possible but hasn't happened yet. It's as if the generated globals are "coming down" after their dependency on early bquote expansion. The two impacted globals seem to be cor and simple. All this would be fine, except (a) I don't really understand why this happens in the first place, and (b) it seems that, in the second generation, they don't work anymore. At least cor doesn't. It just hangs.

2. After I put in all the definitions of bquote, bqex, bqthru, bqexpair, spa, and spd — that is, all the ones whose boxes say they are waiting for backquotes — and regenerate, now (bquote x) hangs. I do not understand why.

Prior to installing all those globals (and regenerating), (bquote x) would be expanded via the Perl backquote expander. (It's directed to this handler via the awful hack using a fake primitive.) The result would be (quote x), which would evaluate to the symbol x.

After installing the globals (and regenerating), here's what I think ought to happen, in sequence:

• bquote is recognized as a macro, and called
• (let (sub change) (bqex e nil) ...) is evaluated
• ...oh.

I see now.

Let me lay it out clearly. Here's the definition of let:

(mac let (parms val . body)
  `((fn (,parms) ,@body) ,val))

...yeah.

The first thing the let macro does, after binding its parameters, is to call the bquote macro. The first thing the bquote macro does, after binding its parameter, is to call the let macro. I imagine eventually we'll run out of stack, but thanks to the design of the Bel interpreter, it will run for a long time before that happens.

This is very interesting! I have no idea how to fix this. I also have no sense how serious this problem is.

[masak]

Here's one way to solve the bootstrapping problem: make sure that bquote does not transitively call into itself.

• For one thing, it involves removing the explicit uses of quasiquotes in the definitions in the bquote group. No problem.
• But also, all the (function and macro) calls out of this group (such as the let call) must be guaranteed not to invoke bquote.
  • We could do this by changing a bunch of definitions, such as let, to let them not use bquotes. This transformation is always possible, since that's exactly what the Perl backquote expander already does. (In fact, we could use it to generate the code instead of guessing it ourselves.) In fact, this is what we did before the backquote expander, to get some definitions early. I would need to check exactly how many global definitions would be affected by this.
  • Or, maybe somewhat better, we could produce a set of "safe" globals to use inside of the bquote group; basically bquote-expanded versions of the regular ones. Maybe prefix them all in some way, such as bq-let. I think I like this approach better, because it leaves the definition of the original let alone.

With my current understanding, I definitely think that this is a "bug" in the spec. I'm pretty sure pg did not intend for the unmodified execution of bquote to lead to an infinite recursion. But that's what it does.

In light of this, we have to implement something slightly different from what the spec says.

[masak]

I would need to check exactly how many global definitions would be affected by this.

At a rough count, it's these:

• let
• fn
• case
• with
• or
• pcase

[masak]

Just a quick status report: it was possible to add the definitions of bqex and bqexpair, and to run simple cases of those on the bel REPL, but adding bquote and testing it in an identical way (which ought to have delegated to bqex and bqexpair) hangs. It is not at all clear to me why.

Oh! It's possible I pasted in bquote with let instead of bq-let. D'oh. Yes, that would explain it.

Going to try again tonight.

[masak]

Something is still a bit off, and I don't know why. I'm guessing I'm still causing an infinite recursion somewhere.

I just tried running the tests after rebasing on top of #194, under the hypothesis that if the tests are just running a bit slow, they would at least run faster on top of that branch. But no, they definitely stall as soon as we're doing nontrivial backquoting.

Interestingly, t/bquote.t passes! But t/fn-array.t stalls because (array '(3) 0) stalls. (And that, in turn, is because a call to nof stalls.)

[masak]

Superseded by (the already-merged) #277. Closing.