SIP-53 - Quote Pattern Explicit Type Variable Syntax

[nicolasstucki]

No description provided.

[prolativ]

After thinking about this proposal more deeply I have some concerns:

1. Even though matching several identical types at once like in

case '[ (t, t, t) ] =>

might seem handy, it would introduce inconsistency with regard to how this works with non-quote-based match expressions. E.g. neither of the snippets below compiles

def mergePair(pair: (Int, Int)) = pair match
  case (x, x) => x

type MergePair[P <: (Int, Int)] = P match
  case (x, x) => x

To keep things consistent I think we would have to make them work too. As that would extend the scope of the SIP, I would suggest extracting the possibility to reuse a pattern variable (also when matching on a non-quote expression or on a type) to a separate SIP.

2. Auxiliary type declarations in quoted expression pattern don't seem to create any problem from a syntactic point of view as it's also possible to declare a type in quoted expressions and braced blocks of code in general (with the possibility to replace braces with indentation). However in the world of terms one can also use parentheses, which use slightly different rules of parsing for what is inside them and don't allow indroducing auxiliary definitions. Until this point square brackets for types have behaved very much like parentheses for terms, not like braces. So allowing leading auxiliary type declarations (followed by semicolons - not allowed in parentheses - or new lines) would introduce a syntactic inconsistency

3. In the given examples

case '[ type t; List[`t`] ] => f[t]
case '[ type tail <: Tuple; *:[Int, `tail`] ] => g[tail]

the auxiliary type declarations are redundant. Could you instead give some examples where this new functionality would be actually useful? From what I understood such syntax would only make sense to specify some additional type bounds which cannot be inferred, though in most cases that would be just syntactic sugar over type extractors. The only thing that I didn't manage to express with type extractors seems to be <: AnyVal bound but if that's the case then maybe our real problem is that type extractors themselves are not handled properly?

[nicolasstucki]

1. Even though matching several identical types at once like in
   ...
   To keep things consistent I think we would have to make them work too. As that would extend the scope of the SIP, I would suggest extracting the possibility to reuse a pattern variable (also when matching on a non-quote expression or on a type) to a separate SIP.

Both cases case (x, x) => do not have type variables. These are term bindings. Those are not really comparable.

An equivalent usage with a normal Scala pattern could be case x: (t, t) =>. I have no idea if duplicate variables like this would work on normal patterns. In the case of quote patterns we get the extra constraints from the expression at runtime and make sure that the type is sound.

[nicolasstucki]

In #59 (comment) there is some misleading nomenclature. These type declarations type t are not auxiliary, they are the canonical form of a quote pattern with type variables. The issue we have is that we do not have the syntax to write the pattern in its canonical form. The inline t declarations are auxiliary and get transformed into canonical ones.

[prolativ]

Sorry for the confusion in the terminology. These expressions were not mentioned in the document so I would suggest introducing them there for clarity.

My general concern is that in the context of match and case keywords the meaning of different syntactic constructs might change depending on the particular context, making the overall syntax somehow irregular, difficult to understand/explain and surprising. That's already a problem IMO and I'm afraid introducing new syntax might make this even worse, especially if we add more exceptional rules rather than reuse the more general ones, which already exist.

For instance: I am aware we cannot treat term and types in exactly the same way but being able to manipulate types analogically to terms seems to make operations on types more intuitive for users and easier to reason about, as show above in the example with mergePair/MergePair. In scala 2 achieving the same semantics on types required a lot of boilerplate and indirectness. And while case (x, x) => works for both terms and types, for some reason case y => doesn't work for types. Also if guards are allowed only when matching on terms.

Also, one of the features of pattern matching that makes it so convenient and intuitive to use is the duality of construction and deconstruction. So, e.g.

Foo(1, 2) match
  case Foo(x, y) => Foo(x, y)

would return Foo(1, 2) back. This analogy seems to also work for match types.
However, even though one can use quoted type patterns to match on Types, '[...] syntax cannot be used to construct them. Also matches like

  '{ type t; 1 } match
    case '{ type t; 1 } =>

don't seem to work.

I suppose some of these peculiarities are there for a reason and they might make some more sense if analyzed from a different perspective and explained/documented in a better way. However, in that case, we should first at least document these things properly. Secondly, if there's anything that could be regularized from the syntactic point of view, we should do that too before extending the syntax.

[lrytz]

There are possibly larger changes that could be proposed to improve the consistency / usability of quoted patterns. However this SIP is a relatively small iteration on the current syntax, it proposes two changes:

• explicit type variable definitions in quoted type patterns; this was missing because of an oversight, causing limitations in expressiveness
• a simplification when using explicit type variable definitions

Both makes sens to me, so I can recommend accepting this SIP.

[julienrf]

@Kordyjan and @chrisandrews-ms could you please review this proposal and comment here?

[chrisandrews-ms]

I also recommend accepting this SIP

[julienrf]

@Kordyjan what is your recommendation regarding this proposal?

[Kordyjan]

Sorry for not giving my recommendation earlier.
I'm slightly confused by the wording of the proposal. Will the patterns case '[ (t, t, t) ] => f[t] or case '[ head *: tail ] => h[tail] be accepted as they are after the implementation of this SIP or will they require explicitly specifying type declaration with (in the second case with bounds)?
If the answer is "no" and this proposal only adds a possibility to declare types with bounds in patterns, then I'm for accepting it.

[nicolasstucki]

This SIP only adds the ability to explicitly define the type variables. It will not remove any functionality.

[soronpo]

IMO, this SIP needs a (Pre-SIP) discussion thread on contributors before we can vote on it.

[nicolasstucki]

What would be discussed in a Pre-SIP? The syntax of type variable definition was set before 3.0 as it is the same as in expression patterns (same holds for the semantics). The only addition is the simplification to be able to use refer-to type variables without backticks (`u` -> u).

I could open a pre-SIP about removing backticks and maybe split this SIP into two.

[smarter]

If I understand this proposal correctly, explicit type variables in type patterns '[ ... ] are only needed to specify bounds, correct? I think the proposal is fine as-is, but we might want to consider whether we can avoid introducing this new syntactic concept: right now '[ ... ] is a context where one can only write a reference to a type, but if we introduce type definitions and semi-colons, we've now blurred the distinction between type contexts and definition contexts in an unprecedented way in the language (it would be a bit like allowing val x: List[type t = Int; t]).

The example usage for this feature in the document is:

case '[ type tail <: Tuple; *:[Int, tail] ] => g[tail]

But the definition of *: already require the tail to be of type Tuple, so intuitively I would expect the type match to be able to infer that bound and allow me to write:

case '[ *:[Int, tail] ] => g[tail]

This would mirror how type patterns in regular term-level pattern matching works:

case x: *:[Int, tail] => ... // In this expression, we have tail <: Tuple

This does make it a bit harder to restrict bounds further than what can be inferred, although it could be achieved with something like:

type ListOfInt[T <: Int] = List[T]
// ...
case '[ ListOfInt[t] ] =>

Again this has the advantage of not mixing type definition syntax and type reference syntax in an unprecedented way, and of working in regular term-level pattern matching.

WDYT?

[nicolasstucki]

Quote type variable inference is one of the problematic features that made us realize we absolutely needed to be able to write bounds explicitly. We made the mistake of adding type inference of type variable bounds to some simple cases, such as is case '[ *:[Int, tail] ] to overcome the limitation of not having a way to write explicit bounds. Now we know that quote-type variable bound inference cannot be implemented correctly. Because of this, we need to provide a way to write the type variable bounds without using inference.

Removing quoted type variable bound inference is the subject of a future SIP.

[smarter]

What are the issues with bound inference?

[smarter]

It would be good to specify whether semicolon inference works here too:

case '[
  type tail <: Tuple
  *:[Int, `tail`] 
] => g[tail]

[prolativ]

If we really need a way to define type bounds inline in quoted type patterns, why don't we consider a slightly different syntax to avoid the precedent of type ... declarations and semicolons inside square brackets?
How about something like
a)

case '{ type tail <: Tuple }[ *:[Int, `tail`] ] => g[tail]

or
b)

case { type tail <: Tuple }'[ *:[Int, `tail`] ] => g[tail]

instead of

case '[ type tail <: Tuple; *:[Int, `tail`] ] => g[tail]

?
Type definitions inside braces is something that is already possible in the language. In such position the rules for using semicolons are also specified. IMO that would solve the syntactic problem of this SIP.

Even though what I'll write next would be out of scope of this SIP, the potential syntaxes shown above seem to open some more new possibilities towards making the general syntax of pattern matching more expressive and uniform, e.g.

1 )
The existing syntax for matching quoted expressions with explicit type variables like

case '{ type t <: Tuple; $x: `t` } => f[t](x: Expr[t])

could be treated as syntactic sugar for

case '{ type t  <: Tuple }{ $x: `t` } => f[t](x: Expr[t])

or

case { type t <: Tuple }'{ $x: `t` } => f[t](x: Expr[t])

with the actual matched expression $x: `t` explicitly separated from the declaration(s) of type variable(s)

2 )
We could use braces before a pattern to express general preconditions for the pattern, e.g.

case { val t }(t, t, t) =>

could be a shortcut notation for

case (t1, t2, t3) if t2 == t1 && t3 == t1  =>

[Kordyjan]

After a debate, the SIP committee voted and decided to accept the proposal.
However, we believe there is a severe lack of documentation about how quote patterns work. The Committee strongly recommends that the pull request to the compiler implementing the new syntax should also contain documentation specifying what exactly is allowed in quote type and term matching.

[nicolasstucki]

The Committee strongly recommends that the pull request to the compiler implementing the new syntax should also contain documentation specifying what exactly is allowed in quote type and term matching.

@Kordyjan could you elaborate on which kind of specification should be enhanced. Currently we have the high level documentation in macros.md#quote-pattern-matching. Is there anything specific that was found missing from there? Or is it the documentation of syntax.md? Or should some other kind of specification be added?

[Kordyjan]

During the discussion, we were unaware of this documentation, as it is different and more thorough than what can be found in the current reference. For me this documentation is satisfying.

@odersky, @sjrd & @bjornregnell: You raised concerns about the lack of reference. What do you think about the documentation linked above by @nicolasstucki?

[sjrd]

I was happy with what was there already. It must have been someone else.

[smarter]

• Support type variable with bounds in quoted pattern scala3#16910 doesn't currently update https://github.com/lampepfl/dotty/blob/main/docs/_docs/reference/metaprogramming/macros.md#quote-pattern-matching, but I assume this will come eventually?
• Does anyone know how often https://docs.scala-lang.org/scala3/reference is updated to match http://dotty.epfl.ch/docs/reference ?

[julienrf]

• Does anyone know how often https://docs.scala-lang.org/scala3/reference is updated to match http://dotty.epfl.ch/docs/reference ?

It should be done after each stable release (see e.g. scala/scala3#16662, but I couldn’t find how this issue was generated)

[nicolasstucki]

scala/scala3#16910 doesn't currently update https://github.com/lampepfl/dotty/blob/main/docs/_docs/reference/metaprogramming/macros.md#quote-pattern-matching, but I assume this will come eventually?

I have not updated the docs in that PR yet. That is the last thing I need to do to finish that PR.

I have already updated the docs of the backticks PR scala/scala3#16935.

[nicolasstucki]

I combined the two PRs of the proof of concept into scala/scala3#17362.

[julienrf]

Hey @nicolasstucki,
Since your proposal changes the syntax of the language, it has impacts on tooling support. Would you mind going over the list of impacted tools, checking their status, and if needed creating new issues to track the support of your proposal?

You can find a list of relevant tools here: https://github.com/scala/improvement-proposals/blob/main/tooling-ecosystem.md