Async Iteration and down-level Generators

[rbuckton]

This PR adds full support for Async Generators and Async Iteration (for..await..of) as per the current TC39 Proposal for all script targets. It also contains full support for Generators in ES5/3, including for..of, spread, and iterator destructuring.

Iterators

ES6 introduced Iterator, which is an object that exposes three methods, next, return, and throw, as per the following interface:

interface Iterator<T> {
  next(value?: any): IteratorResult<T>;
  return?(value?: any): IteratorResult<T>;
  throw?(e?: any): IteratorResult<T>;
}

This kind of iterator is useful for iterating over synchronously available values, such as the elements of an Array or the keys of a Map. An object that supports iteration is said to be "iterable" if it has a Symbol.iterator method that returns an Iterator object.

Async Iterators

The Async Iteration proposal introduces an AsyncIterator, which is similar to Iterator. The difference lies in the fact that the next, return, and throw methods of an AsyncIterator return a Promise for the iteration result, rather than the result itself. This allows the caller to enlist in an asynchronous notification for the time at which the AsyncIterator has advanced to the point of yielding a value. An AsyncIterator has the following shape:

interface AsyncIterator<T> {
  next(value?: any): Promise<IteratorResult<T>>;
  return?(value?: any): Promise<IteratorResult<T>>;
  throw?(e?: any): Promise<IteratorResult<T>>;
}

An object that supports async iteration is said to be "iterable" if it has a Symbol.asyncIterator method that returns an AsyncIterator object.

Generators

ES6 also introduced "Generators", which are functions that can be used to yield partial computation results via the Iterator interface and the yield keyword. Generators can also internally delegate calls to another iterable through yield *. For example:

function* f() {
  yield 1;
  yield* [2, 3];
}

Async Generators

The Async Iteration proposal introduces "Async Generators", which are async functions that also can be used to yield partial computation results. Async Generators can also delegate calls via yield* to either an iterable or async iterable:

async function* f(p) {
  yield 1;
  await p;
  yield* [2, 3];
  yield* (async function *() { 
    await p; 
    yield 4; 
  })();
}

As with Generators, Async Generators can only be function declarations, function expressions, or methods of classes or object literals. Arrow functions cannot be Async Generators. Async Generators require a valid, global Promise implementation (either native or an ES6-compatible polyfill), in addition to a valid Symbol.asyncIterator reference (either a native symbol or a shim).

The for-await-of Statement

Finally, ES6 introduced the for..of statement as a means of iterating over an iterable. Similarly, the Async Iteration proposal introduces the for..await..of statement to iterate over an async iterable:

async function* g() { ... }
async function f() {
  for await (const x of g()) {
  }
}

The for..await..of statement is only legal within an Async Function or Async Generator.

Generators and Iteration for ES5/3

In addition, the following changes are included to bring full support for Generators to ES5/ES3:

• Added a -downlevelIteration compiler option, used when targeting ES5 or ES3.
  • When set, the compiler uses new type check and emit behavior that attempts to call a [Symbol.iterator]() method on the iterated object if it is found, and creates a synthetic array iterator over the object if it is not. This requires a native Symbol.iterator or Symbol.iterator shim at runtime for any non-array values.
  • When not set, array destructuring, spread, and for..of are only supported for arrays as is the current behavior in TypeScript 2.1 and earlier.
• Array Destructuring now uses Symbol.iterator in ES5/3 if available when using -downlevelIteration, but can be used on an Array even if it does not define Symbol.iterator at run time or design time.
• Spread elements in Array, Call, and New expressions use Symbol.iterator in ES5/3 if available when using -downlevelIteration, but can be used on Array even if it does not define Symbol.iterator at run time or design time.
• for..of statements support Symbol.iterator in ES5/3 if available when using -downlevelIteration, but can be used on an Array even if it does not define Symbol.iterator at run time or design time.
• Async Functions do not require a Symbol.iterator polyfill at run time to operate.

Related issues

• Fixes Generator with Array Spread #11970 - Generator with Array Spread
• Fixes Asynchronous Iterations #11326 - Asynchronous Iterations
• Fixes for-of with map.keys() gives TS2495 #11209 - for-of with map.keys() gives TS2495
• Fixes iterators with ES5 target #4031 - iterators with ES5 target
• Fixes ES6 Generator support #1564 - ES6 Generator support

[TonyPythoneer]

This PR is planed on TypeScript 2.3 release. 😭
I hope this can be released soon.

[rbuckton]

@mhegazy, Is there anyone else you want to review this or should I merge?

[mhegazy]

we need to update VS setup to include this file as well.

[mhegazy]

nit: the parens are not needed

[mhegazy]

same here

[mhegazy]

why do you need this here, should not the new helpers only be visible in expressions?

[mhegazy]

if it is needed, then it should be isInAmbientContext

[jocull]

Just curious, why is the downlevelIteration still undocumented under tsc --help? It seems to work, but there's no mention of it, and the errors leading up to needing it are unhelpful.

[mhegazy]

Just curious, why is the downlevelIteration still undocumented under tsc --help?

it is, just use tsc --help --all