Declaration expressions and declaration promotion

[lrhn]

Edit 2021-11-24 - Restructure, move "maybe we can also" parts into separate section.

---

This is inspired by #1091, #1201 and #1210, but is slightly different in approach/scope.

We have an issue with promotion of instance members (or any non-local variable in general).
The #1091 approach is to do a binding at the promotion point. The #1201 approach is to introduce new variables in tests (but do it implicitly in some cases), and #1210 introduces new variables with a new syntax (and also potentially implicitly).

This is a proposal for two features which takes some of #1201 and #1210, but do not introduce any names implicitly, and where the binding can be used independently of the need to promote the variable.

Feature: Assignment Promotion

(Note: discussed in #1844)

First, allow a local variable assignment of the form id assignmentOp expression (potentially parenthesized) to act like the variable itself, when used in a test. We currently allow if (x != null) ... to promote x. This change would also allow if ((x = something) != null) ... to promote x.

It only affects the left-most variable of an assignment, so if ((x = y = something) != null) ... will only promote x, not both x and y (although that's also an option if we really want it - treat both x and the assigned expression as being tested)

If you do if ((x += 1) != null) ... that still works, but there aren't that many operators which return a nullable result, so the usefulness is limited.

This is a very small feature, but it allows you to do “field promotion” as:

int? c;
if ((c = this.capacity) != null) { ... use(c)... }

Feature: Variable declaration expression

Allow var x = e and final x = e as expressions. No types, only var and final. Must have an “initializer expression”.

The expression introduces a new variable named x in the current block scope, and it’s a compile-time error to refer to that variable prior to its declaration.

For statement-level declarations, the “prior to declaration” is anything prior to the source location of the declaration. For a declaration with an initializer expression, that’s equivalent to hoisting the declaration to the top of the current scope block, but keeping the assignment at the original declaration point, and saying you must not refer to the variable where it’s not definitely assigned.

These expression variables, which must have initializer expressions, has the same behavior:

• The scope of the variable is the current scope, and the variable is initially unassigned (even if it’s nullable). Even if that scope is not one that normally allows local variable declarations (like an initializer list scope or the scope of a => function body).
• That variable is assigned only where the declaration expression occurs.
• It’s a compile-time error to refer to the variable where it’s not definitely assigned. Even for assigning to the variable.

For type inference, the context type of the declaration becomes the context type of the RHS, the static type of the RHS becomes the declared type of the variable.

The construct is an <expression>:

<expression> ::= (`var'|`final') <identifier> `=' <expression> | …

An <expressionsStatement> expression also cannot start with final or var, just like it currently cannot start with {.

The new constructs, being <expression>s, need to be parenthesized in most places, including before an is check, but also that it can contain any expression as a RHS, including a cascade.

Usage

This feature can introduce a variable at the first point where you need its value, rather than having to go back and declare it further up, even though that’s effectively what it does.

If you have an expression where a sub-expression is repeated twice, you can name it the first time, and then refer to that name the second time:

foo(v.property.name, v.property.value);

Can become:

foo((var p = v.property).name, p.value);

The variable declaration works anywhere there is a surrounding scope (which is any expression), even if you can’t normally have statement-level variable declarations there.

As opposed to let x = e1 in e2-like constructs, where the scope is only e2, the variable uses the same scoping as other local variables, which is “until the end of the current scope”. Because of that, it can also be used where the expressions do not share a common (or at least not close) parent expression, like lists (potentially deeply nested inside another expression):

var list = [1, 2, var x = compute(), 3, x];
fooWithArgumentList(1, 2, var x = compute(), 3, x);

or where we can’t technically have variable declarations, like initializer lists:

class C<T> {
  final StreamController<T> controller;
  final Stream<T> stream;
  C() : controller = (var c = StreamController()), stream = c.stream;
}

(which would currently be implemented using an extra helper constructor), or => function bodies:

BTree<T> buildDag<T>(int depth, T leafValue) => 
   depth == 0 
      ? BTree.leaf(leafValue) 
      : BTree.node(var dag = buildDag(depth - 1, leafValue), dag);

(where we would currently use a {} body to be able to declare the variable up-front.)

Promotion

The var id = e or final id = e counts as assignments for assignment promotion (above), so if ((var c = this.capacity) != null) { ... } would both read capacity into a local variable, then promote that local variable to non-null if possible.

This is the proposed solution to promoting non-local-variable expressions: Introduce a new local variable with the same value, then promote that, and do it in a single expression. Only, unlike #1191, the binding feature is generally useful and not restricted to checks or promotion, and it doesn't clash with a potential pattern syntax for is checks. The binding is not linked to the test, the features are orthogonal, and should therefore also work if we introduce more tests (like pattern matching) in the future.

Alternatives and similar features

There is no implicit assignment of a name to an expression, unlike #1201 and #1210. I personally found those hard to read. We can introduce those as well, so, for example, var foo.bar.fieldName as an expression would be equivalent to (var fieldName = foo.bar.fieldName). Basically: var selector.chain.last, not followed by = and in an expression position, is equivalent to var last = selector.chain.last (and similar for final).

Since this is based on the same logics as the current definite-assignment analysis, it can make variables available in only some continuation of the declaration.

For example:

if ((var y = this.y) != null && (var x = this.x) != null) {
  // `y` definitely assigned here, and not null.
  // `x` definitely assigned here, and not null
} else {
  // `y` definitely assigned here, may be null.
  // `x` not definitely assigned here.
}

Control flow statement scopes

Currently there is no special scope for the conditions of an if or while statement. The condition expression belongs to the surrounding scope. (Or rather, there is no way to tell since you cannot introduce variables inside the condition expression

For a for (;;) statement, there is a new scope introduced for the for statement itself, separate from the block scope of the body (it’s the parent scope of the body scope). The variables declared in the initializer part of the for (;;) loop are declared in that scope.

We could introduce a condition-scope for if and while statements, so that variable declarations in the test belongs only to that scope, and not the surrounding block scope.

Example:

if ((var x = this.x) != null) {
  doSomethingWith(x);  
}
// Should x be available here, unpromoted?

It would be consistent to introduce a wrapper scope for the control flow statement itself, like for for (;;). It’s not necessary, but it means that the variable belongs to the outer scope, and may conflict with other variables in that scope.

On the other hand, it also prevents constructs like:

if ((var x = this.x) == null) return;
// Use x as non-null.

which is a logical extension of the same pattern that we support for promoting local variables.

I’d recommend not introducing that scope.

That means that a variable unconditionally declared in test of while (test) { ... } is available in the body.

Feature: Shorter syntax for local declaration

Alternatively, maybe preferably, introduce x := e as an in-line final declaration, equivalent to the above final x = e, and do not introduce any way to declare non-final local variables.

That means that locally declared variables will all be final. I think that's a good thing. It prevents some use case, like:

 var list = [f(x := 0), f(++x), f(++x), f(++x)];

but I'm not sure such uses are really that essential.

If the local variable declaration expressions can only introduce final variables, it means closures over them can just capture the value, and not worry about getting the correct variable. (Also x := e; can still be used as an expression statement as a short declaration of a final local variable).

Examples (using := only).

class C {
  final int? nullable;
  C(this.nullable);
  String doSomerhing() {
    if ((n := nullable) != null) {  // Use to promote non-local variables.
      return n.toRadixString(16);
    }
    return "0"
  }
}

class Streamer<T> {
  final StreamController<T> _controller;
  final Stream<T> stream;
  // Introduce local variables in initializer list, avoids the "two-constructor-hack".
  Streamer([StreamController<T>? controller>])
      : _controller = c := controller ?? StreamController<T>(),
        _stream = c.stream;  

  // Two-constructor hack for introducing shared computed value.
  StreamController.hack([StreamController<T>? controller>]) 
      : this._hack(controller ?? StreamController<T>()); // Create and forward.
  StreamController._hack(this._controller) : stream = _controller.stream; // Use twice.
}

// Binary tree structure of nodes with two subtrees, and leaves with a value.
abstract class BTree<T> {
  Btree();
  factory Btree.leaf(T value) = BtreeLeaf<T>;
  factory Btree.node(Btree<T> left, BTree<T> right) = BtreeNode<T>;
  BTreeLeaf<T>? asLeaf() => null;
  BTreeNode<T>? asNode() => null;
}
class BTreeLeaf<T> extends Btree<T> {
  final T value;
  BTreeLeaf(this.value);
  BTreeLeaf<T>? asLeaf() => this;
}
class BTreeNode<T> extends Btree<T> {
  final BTree<T> left, right;
  BTreeNode(this.left, this.right);
  BTreeNode<T>? asNode() => this;
}

/// Builds single-width DAG of [depth] nodes and one leaf value.
BTree<T> buildDag<T>(int depth, T leafValue) => 
    // Is useful inside `=>` functions. 
    // Would otherwise require a block body and local variable, or a helper function.
    depth == 0 
        ? BTree.leaf(leafValue) 
        : BTree.node(dag := buildDag(depth - 1, leafValue), dag);

void main() {
  // Just use it as a plain final declaration inside a method body, 
  // but not outside of it (it's an expression).
  // No conflict with local `var`/`final` declarations, unlike `final x = 42`.
  x := 42;  
  
  // Works just like the variable itself when tested for promotion.
  if ((y := intOrNull()) != null) { ... y promoted to int ...}

  LinkedListNode<T> current = ...;
  // Can be used in loop conditions and seen into the body.
  while ((next := current.next) != null) {
    current = next;
  }
  // `current` is last element of in linked list.

  // Useful in collection comprehensions:
  var list = [
      for (var v in someElements) if ((p := v.some.property) != null) p  // Good!
  ];
  // Instead of having to repeat it:
  var list = [
      for (var v in someElements) if (v.some.property != null) v.some.property // Bad!
  ];
  // Or make a hack to bind a variable using `for`:
  var list = [
      for (var v in someElements) for (var p in [v.some.property]) if (p != null) p // Ugly!
  ];
}

Summary

This is really three features:

• Assignment promotion: Allow (x = e) is T and (x = e) == null to promote x.
  • Nice feature by itself.
• One of:
  • Local variable declarations: var x = e and final x = e are expressions. They introduce variables into the current block (just like the similar statement-level declarations), but must not be used where the variable is potentially unassigned. Both are also subject to assignment promotion.
  • Local final variable declaration: x := e is an expression. It introduces a final variable into the current block (like final x = e above, but nothing similar to var). Also subject to assignment promotion.
• Or both, they’re compatible, but that’s probably overkill.

[eernstg]

I like this! I think it would be helpful to include the rule that every composite statement introduces a scope which is enclosing the entire composite statement and nothing else. For example:

void main() {
  if (b) {...} else {...}
  // works like:
  {
    if (b) {
      ...
    } else {
      ...
    }
  }
}

such that any variables introduced by b would be available in the entire if-statement (including the else part), but nowhere in the enclosing statement list. I think this amounts to a much more comprehensible scoping for such variables.

Compared to #1210, binding expressions, the main difference to a variable declaration expression is that the former uses : in order to disambiguate parsing, and the latter uses an existing declaration syntax, but presumably forces the use of parentheses in almost all contexts. The parentheses could be quite helpful from a readability point of view, so the added verbosity might not be a problem.

The implicit variant (where the name of the new variable is derived from the initializing expression in some way) is orthogonal: Both forms can easily omit that feature or include it, and it would be a non-breaking change to add it later.

[lrhn]

Thanks for the corrections.

Using with isn't really necessary, you can just do ...use(x:=xx, y:=yy)....

Generally the idea here is to not change the structure of expressions, but allow you to name existing sub-expressions, and then reuse them later.
If you currently have foo(e1, e1), with a repeated expression, you can instead do foo(tmp := e1, tmp), which names the value the first time it occurs, then uses the name later to avoid computing the value.

With this, we can desugar the e1.x += e2 operation as ($tmp := e1).x = $tmp.x + e2 (where $tmp is fresh).

About only allowing :=, I mean that we should not allow ...(var x = 42)... or ...(final x = 42)... as expressions anyway, they'll stay declarations only. Only the declaration form ...(x := 42)... can be used as an expression, and that ensures that the variables introduced in the middle of an expression are always final. I think that makes some things easier.
You can still write x := 42; as an expression statement, and use it as a shorthand for final x = 42;, because you can do expr; for (almost) any expression.

[Cat-sushi]

Can we hide the original field/ variable with new variable like use(xx := xx, yy := yy)?

[lrhn]

@Cat-sushi The scope would not support that. Local variables are added to the surrounding scope, even before their declaration.
As such, the variable is in scope for its own initializer expression, you're just not allowed to use it yet.
You'd have to write use(xx := this.xx, yy := this.yy) to make the distinction.

We could change the way scopes work in Dart, and introduce a new scope for every var x = y/x:=y declaration, covering everything after the initializer expression. That would allow xx := xx to work, because the new xx variable won't be in scope until after the xx expression has been evaluated. That's a quite significant change to how scopes work now, and it's a change that can be made independently of what we do here, so I don't want to include it in this feature.
I don't think it's an implementation problem to do this. We currently have to detect whether a variable reference is valid and give an error if it's not. If we just continued lexical lookup when finding a not-yet-initialized value, instead of giving an error, it would simulate the variable not being in scope until it becomes valid. It's much harder to specify (but I guess it could be hacked the same way as the implementation, by keeping the current scope + when variable references are invalid, and then skipping past currently invalid variables during lookup.

[lrhn]

@eernstg About wrapping all composite statements/control flow structures in extra scopes, I'm not sure it's precisely what we want.

There is me wanting to be able to do something about the variable set in a failed test, so while ((next := current.next) != null && test(next.property)) { ... } would allow me to check what next was after the loop. That can perhaps be solved by giving loops an else branch which is inside the statement (#171).

Apart from that, it should be one scope per iteration in order to allow variable declarations in the test to be a new variable on each iteration. So a while loop would introduce a new scope, the "test scope", per iteration, in which the test is evaluated. Then the body is evaluated in the new "body scope", which has the "test scope" as parent scope.
We could perhaps drop the "body scope" - if the body is a block statement, it gets a new scope anyway, if not, it can perhaps live in the "test scope", which would then be the "iteration scope".

For for (var i = 0; i < something; i++) body we need to keep the var i = 0 outside of the iteration scope since it's evaluated once, but i < something is inside the iteration scope. It's somewhat equivalent to:

{ // Outer scope
  var i = 0;
  loop: { // Iteration scope
    if (!(i < something)) break loop;
    $body: { // body scope (if necessary?)
      body[continue -> break $body]
    }
    i++;
    restart loop; // pseudocode.
  }
}

The for (var x in e) ... needs to keep e outside the iteration loop, but var x inside it.

So, I think it's possible to do something with scopes here, but it's more complicated than just wrapping every composite statement in an outer scope (although that too might be worth it, to contain variables ... as long as I get my else on loops).

[Cat-sushi]

@lrhn Thank you. I probably understand what you said.
My motivation of the question was, thinking out new names is bothersome.
Then, writing xx := this.xx is also bothersome.
Do you have any good idea in this thread or another post?

[lrhn]

Fixed the "dag" to use := too.
I admit that the var dag = ... does read well. I think the advantage of := is more that it forces the variable to be final, because otherwise I'd use var every time to save on writing (and reading, for that matter).

For try, you're already at the statement level, so putting a variable declaration before the try is just a normal variable declaration.

As described, there is no int x := 0 syntax. The grammar for the declaration expression is identifier `:=` expression. That's an expression (likely with the same precedence as other assignments, but not just a plain assignmentExpression because it only allows a single identifier as LHS), and there is no use of := anywhere else in the grammar. You can't write int x := 1; or var x:= 1; any more than you can write int x += 1; or var x += 1;.

And yes, restricting to only allowing := as inline expression declaration is an attempt to not have too many ways to do the same thing, and to force all such inline declarations to be final.

[lrhn]

Good. What about void foo([x:=0]) {...}? Allowed?

That's not an expression, so no.

As for try... the symmetry is there, you can declare variables inside expressions, and try doesn't have a leading expression.
(Just like while { ... } do (test); has the test last, so declaring variables there won't help much).

[eernstg]

@lrhn wrote:

There is me wanting to be able to do something about the variable set in a failed
test, so while ((next := current.next) != null && test(next.property)) { ... } would
allow me to check what next was after the loop.

The reason why I'd recommend limiting the scope of the variable next in this example to the while statement (so you can't access it after the final }) is readability: I think it's a source of confusion if next is in scope for the next 30-or-so lines of code, but it is introduced by a construct which is visually difficult to find if you're looking at a usage of next 20 lines after the end of the loop.

If you want the variable to be available after the loop then you'd simply use an old-fashioned local variable declaration outside the while statement.

[Cat-sushi]

@tatumizer
Using same name should have special meaning, so lastName := this.lastName is OK for me.
I'm looking for even easier way to do so.
And you can access this.lastNmae even after lastName := this.lastName.

[Cat-sushi]

It is for (var i = 0; i < 10; i++) print(i); that the exception is, which is already confusing.
for (var i = 0; i < 10; i++) print(i); print(i); // error

[lrhn]

The biggest argument (to me) against prolonging the lifetime of variables declared inside a construct is what it would do to existing variables in the surrounding scope.

int i = 0;
while (i := iteration.next()) {
  .. something(i);
}
use(i); // <--- Which `i`.

Here the "Which i?" question should be resoundingly answered by the int i;. Anything else is crazy-talk.

So, if the while does not introduce a new scope, then the i := iteration.next() would be an error because i is already declared in the same scope.
If the while introduces a new scope, then the variable should not leak from that scope.
Since for already does introduce a new scope for the loop variables, it would be consistent to do the same for the rest of the loops, and then the rest of the composite statements too, for more consistency.

All in all, I think I agree with Erik that all constructs should introduce a new scope.
If I want to use the variable after the loop, I'll just have to introduce an else branch as part of the loop:

while(test(i := next()) {
  use(i);
} else {
  discard(i);
}

(So #171, please!)