Make it possible to switch on optional parameters/parameter defaults/rest parameters

[masak]

An exercise in Qtree extensibility.

Something like this:

use optional_parameters;
sub ex1(x, y?) { say(y) }
ex1(1, 2);    # 2
ex1(1);       # None

use parameter_defaults;
sub ex2(x = 7, y = x) { say(x); say(y) }
ex2(1, 2);    # 1 2
ex2(3);       # 3 3
ex2();        # 7 7

use rest_parameters;
sub ex3(x, y, *z) { say(z) }
ex3(1, 2, 3, 4, 5);    # [3, 4, 5]

The addition could happen in two steps:

1. Provide these as "internal" modules, basically faked in by the parser and runtime, which have some insider knowledge about these extensions to the grammar, to the Qtree API, and to semantics in general.
2. Do it properly, as pure 007 modules, using userland language extension APIs.

The second step is far tougher, and will likely require exposing some of the function calling mechanism as a public interface or a MOP. But it's also bound to shake out interesting answers to how to safely evolve 007 from userland.

Note that all three of these pragmas ought to be able to work independently, with or without any of the other two.

[masak]

How would these work together? Well, with three different pragmas, there are +combinations(3, 2) == 3 ways to pair them up:

• optional_parameters with parameter_defaults
  • Fine. Just means that a ? is redundant if there's also a =.
  • Perl 6 allows these to be combined.
  • TypeScript doesn't: Parameter cannot have question mark and initializer
• optional_parameters with rest_parameters
  • Would be fine, since rest parameters kind are "zero or more", so making them optional doesn't add any information.
  • However, Perl 6 doesn't allow this: Malformed parameter
  • Neither does TypeScript: A rest parameter cannot be optional
• parameter_defaults with rest_parameters
  • Seems like it could be fine, as long as the default is an array.
  • But Perl 6 doesn't like it: Cannot put default on slurpy parameter @y
  • Neither does TypeScript: A rest parameter cannot have an initializer

Note that we're not talking about combining the pragmas themselves (which is fine, all three at the same time), but using two syntactical extensions together on the same parameter.

[masak]

An interesting bit of parsing manifests itself when we start thinking about a script with the preamble

use optional_parameters;
use parameter_defaults;

versus

use parameter_defaults;
use optional_paramters;

...both of which should obviously work and produce "the same" parser in the current compunit.

To be precise, one might naively think that both these pragmas extend rule parameter in 007's grammar by wrapping it with some extra "parsing stuff", in this case both at the end. Also, we can safely assume that there is no black magic that re-shuffles the order in which the two pragmas load. And yet, we would be disappointed if the second loading order resulted in a parser which allowed

sub fn(x = "hi"?) {}    # wrong

but dissallowed

sub fn(x? = "hi") {}

In other words, there's more to it than just adding stuff to the end of a rule, and possibly to the beginning.

This is problematic for conventional Perl 6 grammars, because grammar rules are methods, which are routines, which are opaque and don't allow modification "in the middle".

The standard route for extensibility in Perl 6 grammars is subclassing, along with protoregexes. But protoregexes require forethought — the base grammar author had to be aware that a certain point was a desirable point of extension for pragma/slang/DSL authors.

What 007 (and ultimately Perl 6) needs is something more like CSS :before and :after pseudo-elements. The thing corresponding to CSS elements in this analogy would be a part of a grammar rule. Something like this:

loadingCompunit.parser.attach("parameter :end", ...);  # default value rule fragment

loadingCompunit.parser.attach("parameter.identifier :after", ...);  # optional parameter rule fragment

AOP and pointcuts also come to mind.

Two different modules attaching a rule to the exact same point — also interesting to consider. To take a concrete example, consider a x! modifier on parameters marking them as required, not optional. The ! would want to inject itself at parameter.identifier :after, just like the ?. Let's assume for the moment that there were good reasons to have both. Then we'd want to be able to parse five different things: empty suffix, ?, !, ?! and !?. But not (say) ?? or !?!.

In other cases they might choose to impose an internal order even when they go on the exact same point. (Perhaps via some kind of voting process.)

In yet other cases (and the case of ? and ! likely is one such), they'd want to be mutually exclusive.

These are just a few comments about the parsing. There are also considerations regarding extending the action methods (which basically need a way to "decorate" Q nodes) and runtime behavior, in this case the signature binder.

[masak]

As OP stated, a non-passed optional parameter binds to None. This seems reasonable, since that's why we bother with None in the first place.

use optional_parameters;
sub ex1(x, y?) { say(y) }
ex1(1, 2);    # 2
ex1(1);       # None

But this is going to rub up against #33, in a way I hadn't thought of before:

use optional_parameters;
sub typed_ex1(x: Int, y?: Int) { say(y) }
typed_ex1(1);    # `y` wants to bind to `None`, but None !~~ Int

Here are possible solutions.

• Make None a type habitable by everything, like undefined in TypeScript. We're not going to do that.
• Disallow both ? and type annotation on parameters, at least in the short term until we decide on a solution. (Tempting.)
• Allow both ? and type annotation, but only if it looks something like Int | None, that is, a union type with None included.
• Allow y?: Int, but interpret it as y?: Int | None.

[vendethiel]

Last one for me.

[masak]

Aye, that one does make some sense, and I like it for its intuitiveness. I don't have the certainty to declare that it's a completely consistent, trap-free solution, though.

[masak]

Also interesting to go check what TypeScript does in this space, after nullable types landed.

[masak]

All parsing challenges aside, the trouble with making signature parsing truly extensible is that it also has consequences for the signature binder, a part of the runtime. Two options present themselves:

• Cheat and stuff the signature binder with logic that knows how to deal with the new forms of parameter. Definitely works, but leads to something that isn't very extensible — the next person who wants to toy around with signature parsing might very well have to add more special cases to the 007 internals.
• Expose the signature binder in 007 user space. Totally doable, and possibly quite interesting. Introduce a hooking system where people can register their own custom Q::Parameter subtypes that then participate in the signature binding through an exposed API.

You're probably thinking what I'm thinking — the second option sounds like a lot more fun. Even doing it just for kicks to see how such a binder API would look in the language might be worth it.

But there are still lingering doubts and problems. For one, extending the signature binder will have to be a global effect (as opposed to locally modifying a parser for a particular scope). The difference is that runtime values are not constrained to scopes or even compilation units. Higher-order functions can arrange to have an "enhanced" function be called in a region of code that was never aware of such enhancements in the first place. This troubles me.

Or maybe a routine's signature binder should actually hang off the routine itself? Each routine carries around (a reference to) the signature binder, which has been assigned by the parser so it has all the information needed. Very, uh, object-oriented — the receiver knows how to receive the message. But this possibility also fills me with dread and vertigo.

[vendethiel]

Or maybe a routine's signature binder should actually hang off the routine itself? Each routine carries around (a reference to) the signature binder, which has been assigned by the parser so it has all the information needed. Very, uh, object-oriented — the receiver knows how to receive the message. But this possibility also fills me with dread and vertigo.

Oh, that sounds really fun! Next up is implementing samewith and some MOP with that. :P

[masak]

It bears stressing that parameter defaults do change the way we do signature binding... or at the very least brings out a feature of it that wasn't necessary before.

Here's the example I used in OP:

sub ex2(x = 7, y = x) { say(x); say(y) }

Note that x is in scope as the default of y. In a world without parameter defaults, we can bind all the parameters "at once" (conceptually). In a world with them, we have to do it one by one, left-to-right, with onion layers of scoping.

[masak]

Oh, and I would expect

sub ex2b(x = x) {}

to be a compile-time error, because initializing a variable with itself is always a thinko.

[vendethiel]

to be a compile-time error, because initializing a variable with itself is always a thinko.

[snark]Python developers want to have a word with you :P[/snark]

[masak]

I got that joke. 😄 I taught a beginner's Python course two weeks ago where I explained this language design mistake (because what else can you call it?) as it incarnates both in function parameter defaults and in class property defaults.

Funnily enough, Python is quite principled when it comes to local variables, and won't take any prisoners:

$ python3 -c 'x = x'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
NameError: name 'x' is not defined

$ python3 -c 'x = 2
> def foo():
>   x = x
> foo()'
Traceback (most recent call last):
  File "<string>", line 4, in <module>
  File "<string>", line 3, in foo
UnboundLocalError: local variable 'x' referenced before assignment

"Referenced before assignment" is a good description of the thinko, actually.

Unfortunately, it throws away the same principle when it comes time to talk about parameter defaults...

$ python3 -c 'L = []
def bar(L = L):
  L.append(1)
  return L
print(bar())
print(bar())'
[1]
[1, 1]

...or class property defaults...

$ python3 -c 'L = []
> class D:
>   L = L
> D().L.append(2)
> print(D().L)'
[2]

Needless to say, we will take a leaf from Perl 6's book here, not Python's.

(Perl 6 is happy-go-lucky about many things, but working on 007 has made me realize that scoping is not one of those things.)

[masak]

Also, while it's always hilarious to tease Python about scoping snafus, it would be hypocritical to let Perl 5 completely off the hook:

$ perl -Mstrict -wE'my $x = 7; { my $x = "00$x"; say $x }'
007

You're reading that right — we can still access the outer $x in the assignment in the initializer of the inner $x.

Perl 6 is wise to such tricks, and politely informs you that you have unreasonable expectations:

$ perl6 -e'my $x = 7; { my $x = "00$x"; say $x }'
===SORRY!=== Error while compiling -e
Cannot use variable $x in declaration to initialize itself
[...]