Annotate immediately-invoked functions for inlining flow control analysis

[RyanCavanaugh]

Per #9998 and its many offshoots, we have problems analyzing code like this

let x: string | number = "OK";
mystery(() => {
  x = 10;
});
if (x === 10) { // Error, we believe this to be impossible  
}

The problem is that we don't know if mystery invokes its argument "now" or "later" (or possibly even never!).

The converse problem appears during reads:

let x: string | number = ...;
if (typeof x === 'string') {
  // Error, toLowerCase doesn't exist on string | number
  let arr = [1, 2, 3].map(n => x.toLowerCase());
}

Proposal is to allow some form of indicating that the function is invoked immediately, e.g.

declare function mystery(arg: immediate () => void): void;

This would "inline" the function body for the purposes of flow control analysis

[tinganho]

I suggested something similar in #7770 (comment).

Though after thinking about it, I thought it was better to annotate callbacks as non-immediate #10180 (comment):

Immediate case:

function doSomething(callback: () => any) {
     callback();
}

function fn(x: number|string) {
  if (typeof x === 'number') {
    doSomething(() => x.toFixed()); // No error.
  }
}

Non-immediate case:

function doSomething(callback: async () => any) {
     setTimeout(callback, 0);
}

function fn(x: number|string) {
  if (typeof x === 'number') {
    doSomething(() => x.toFixed()); // Error x is  'number|string'
  }
}

Sync callbacks is only assignable to sync callbacks and vice versa for Async callbacks:

function doSomething(callback: () => any) {
     setTimeout(callback, 0); // Error a sync callback is not assignable to an async one.
}

function doSomething1(callback: () => any) {
     callback();
}
function doSomething2(callback: () => any) {
     doSomething1(callback); // No error
}

Though re-using async keyword as a type annotation might conflict with the meaning of async declarations in async/await which makes a function returning a promise. An alternative is to use nonimmediate, though I think that keyword is too long.
.

[yortus]

For reference, there was also some discussion of deferred and immediate modifiers in #9757.

[yortus]

@RyanCavanaugh you commented here that implementing this kind of type modifier would require a massive architectural change. Is that still the case?

[kitsonk]

For completeness #11463 proposed an alternative solution for this problem. Since that was just an "escape hatch" and this is an instruction to CFA to better understand the code, I would prefer this.

The use case where we have run into this is converting a Promise into a deferral and having to generate a noop function to get around strict null checks:

const noop = () => { };

function createDeferral() {
    let complete = noop;
    let cancel = noop;
    const promise = new Promise<void>((resolve, reject) => {
        complete = resolve;
        cancel = reject;
    });

    return { complete, cancel, promise };
}

const deferral = createDeferral();

Though after thinking about it, I thought it was better to annotate callbacks as non-immediate

While it is more "burden" on the developer, in the sense of "strictness" I still think it is safer to assume the worst (that it might not be immediate). Like with strictNullChecks overall, so I would be only in favour of an annotation that indicates that it is immediate.

[kitsonk]

Also 🚲 🏠 while it might be a tad bit confusing, why not sync to be a counter to async:

interface PromiseConstructor {
    new <T>(executor: sync (resolve: (value?: T | PromiseLike<T>) => void, reject: (reason?: any) => void) => void): Promise<T>;
}

[tinganho]

I forgot to mention my points on why I think annotating it as async rather sync.

1. Annotating it as immediate or sync means nearly all callbacks needs to be annotated, assuming most callbacks is sync. Async is the exception, sync is the rule.
2. We already annotate async functions(as in async/await) as async. Sync functions has no annotations. So to keep consistent with the current design we should not annotate with immediate.

Maybe to minimize confusion that async only accept a Promise returning callback, a deferred keyword might be more desirable.

[yortus]

I think to cover all cases of "used before assigned" checks you would need both a deferred and an immediate modifier.

The deferred modifier would be needed in cases like @kitsonk reported here where a non-immediately invoked callback references a variable that is initialized later in the calling scope (i.e. lexically later but temporally earlier). immediate wouldn't help with that (unless the compiler assumes that an un-annotated callback must be deferred).

[yortus]

What about the interplay between immediate vs deferred invocation and synchronous vs asynchronous functions, which can occur in any combination?

function invokeImmediately<R>(cb: () => R) {
    return cb();
}

let p = invokeImmediately(async () => {
    console.log(1);
    await null;
    console.log(2);
});

console.log(3);

p.then(() => {
    console.log(4);
});

// Output:
// 1
// 3
// 2
// 4

Here we have immediate invocation of an asynchronous function (fairly common in real code). As shown by the output, the async function executes synchronously until it either awaits or returns. Then the rest of the function is deferred to another tick.

It would be pretty hard to do accurate CFA in this case with types and assignments involved, since some of the callback happens 'in line' with the surrounding code and some execution is deferred until later.

[novemberborn]

@tinganho,

Annotating it as immediate or sync means nearly all callbacks needs to be annotated, assuming most callbacks is sync. Async is the exception, sync is the rule.

In the new Promise() example, the executor function implements the Revealing Constructor Pattern. I wouldn't classify it as a callback — the constructor guarantees it's invoked immediately. Callbacks are often invoked in a future turn.

@yortus,

Here we have immediate invocation of an asynchronous function (fairly common in real code). As shown by the output, the async function executes synchronously until it either awaits or returns. Then the rest of the function is deferred to another tick.

It would be pretty hard to do accurate CFA in this case with types and assignments involved, since some of the callback happens 'in line' with the surrounding code and some execution is deferred until later.

Not knowing anything about how CFA is actually implemented, presumably it could still make a prediction up until the first await statement?

For both your points it would seem that sync doesn't quite cover the nuance of immediate invocation, so immediate may be a better keyword.

[tinganho]

@novemberborn I'm assuming though no annotation is immediate?

[novemberborn]

@tinganho,

I'm assuming though no annotation is immediate?

How do you mean?

[kitsonk]

Still think it would be dangerous to have implicit immediate. There are many callbacks where it doesn't matter and if it does matter, the developer should be explicit in order for CFA not to make assumptions.

[tinganho]

@novemberborn

In the new Promise() example, the executor function implements the Revealing Constructor Pattern. I wouldn't classify it as a callback — the constructor guarantees it's invoked immediately. Callbacks are often invoked in a future turn

In below, my arguments so far have been that no annotation means sync:

interface PromiseConstructor {
    new <T>(executor: (resolve: (value?: T | PromiseLike<T>) => void /* is sync*/, reject: (reason?: any) => void) => void): Promise<T>;
}

[tinganho]

Still think it would be dangerous to have implicit immediate. There are many callbacks where it doesn't matter and if it does matter, the developer should be explicit in order for CFA not to make assumptions.

@kitsonk the CFA need to assume either implicit deferred or implicit immediate in case of no annotation.

Or are you arguing for that either immediate or deferred must be annotated in all callbacks? Or just implicit deferred is more preferable? In case of the latter, CFA still makes assumption.

[novemberborn]

@tinganho I don't think this discussion is about how the callback behaves, it's about whether it's called before or after the function it's passed to has returned. PromiseConstructor guarantees it always invokes the `executor callback before returning.

[yortus]

the CFA need to assume either implicit deferred or implicit immediate in case of no annotation

It could be conservative and make neither assumption, consistent with present compiler behaviour. Otherwise all the existing code out there with no annotation would suddenly compile differently (i.e. assuming immediate invocation). Being explicit with immediate (and/or deferred) opts in to extra CFA assumptions that the compiler would not otherwise make.

[kitsonk]

the CFA need to assume either implicit deferred or implicit immediate in case of no annotation.

No, it needs to assume what it assumes today, which is it could be either, therefore values could have changed, so reset the types.

And picking an example of a Promise resolve is exactly the sort of situation where you can run into problems assuming too much, since promises are always resolved at the end of the turn:

let foo = { foo: 'bar' };

new Promise((resolve) => {
        console.log('executor -->', foo.foo);
        resolve(foo);
    })
    .then((result) => {
        console.log('result -->', result.foo);
    });

foo.foo = 'qat';
console.log('post -->', foo.foo);

/* logs
executor --> bar
post --> qat
result --> qat
*/

[tinganho]

What's the difference between neither, both and async? Isn't neither or both assuming worst case which is the async case?

[yortus]

@tinganho currently this example has a "used before assigned" error under conservative (i.e. 'neither') assumptions. But CFA could compile this successfully if it knew that the callback would be invoked later (e.g. with a deferred annotation). So there is a difference between no annotation and a deferred annotation (and an immediate annotation).

[tinganho]

@yortus that was filed as a bug and fixed, the neither assumption is still in my opinion an async assumption no?

If you thought the bug should be there, then you are assuming an implicit immediate callback and not neither?

I mean let say a callback can be both. Wouldn't it in all cases assume the async case since it is the worst case?

Just for clarity I think when you say neither, you actually mean either? You must choose between async or sync or both. I don't think a callback can be classified as none of them.

[yortus]

@tinganho yes you are right, I suppose the 'fix' there means the compiler now does assume deferred execution on the basis of pragmatism. But to be clear, it assumes deferred invocation, not whether or not the callback itself is asynchronous.

[aluanhaddad]

I'm sure you have considered this at length but given the original example

declare function mystery(f: () => void);
let x: string | number = "OK";
mystery(() => {
    x = 10;
});
if (x === 10) { // Error, we believe this to be impossible  
}

I think the most intuitive behavior would be for a closure which rebinds x to cause the type of x to revert to string | number.

As @kitsonk remarked

While it is more "burden" on the developer, in the sense of "strictness" I still think it is safer to assume the worst (that it might not be immediate). Like with strictNullChecks overall, so I would be only in favour of an annotation that indicates that it is immediate.

While I am in favor of what CFA brings especially in catching logic errors and redundant (superstitious) checks, I think the example above should not be an error because the caller is justified in checking x against 10 simply because mystery may have invoked the closure but it may not have invoked the closure.
Consider a case where mystery invokes the closure immediately, but conditionally based on some other arbitrary state.
consider:

function conditionalBuilder() {
  return {
    when: (predicate: () => boolean) => ({
      then: (action: () => void) => {
        if (predicate()) {
          action();
        }
      }
    });
  };
}

conditionalBuilder()
  .when(() => Math.random() * 100) % 2 > 1)
  .then(() => {
    x = 10;
  });

EDIT fixed typo, formatting

[ghost]

The useful distinction isn't between sync vs async.
There are many synchronous higher-order functions (let's not call them callbacks, because that sounds async) which are not always immediately invoked.

// Invoke N times when N may be 0
function f() {
    let x: number
    ([] as number[]).map(n => {
        x = n
    })
    console.log(x)
}

function provideDefault<T>(x: T | undefined, getDefault: () => T) {
    return x === undefined ? getDefault() : x
}

// Invoke 0 or 1 times
function g(n: number | undefined) {
    let x: number
    const y = provideDefault(n, () => {
        x = 0
        return 0
    })
    console.log(x)
}

function createIterator<T>(next: () => IteratorResult<T>) {
    return { next }
}

// Invoke later, but *not* asynchronously
function h() {
    let x: number
    const iter = createIterator(() => {
        x = 0
        return { value: undefined, done: true }
    })
    console.log(x)
}

We correctly flag all of these currently.

[LRagji]

My bad updated the code its inside a Promise. I assume then its just like all the above people talking...

Hi @LRagji I'm not seeing an error on that line:

Playground link

[rChaoz]

I find it pretty weird that this issue still hasn't been tackled 6 years later. IMO, this fix of this issue has nothing to do with annotating lambdas as immediate/sync or async. I understand that it might work as a fix, but it feels more like a workaround. I don't think it touches the root of the problem: the compiler "assumes the best". I think it makes more sense for the compiler to "assume the worst", instead. ("assuming the best" means here that the compiler makes completely arbitrary assumptions in order to narrow the type as much as possible).

let x: string | number | null = null;

// Why would the compiler assume arbitrarily that the function is not called immediately?
// It could or it could not be instantly invoked - so should assume the worst (i.e. both)
instantInvoke(() => {x = "hello"});
// at this point, the variable could either stay null, or it could be "hello", the type should be string | null
if (x != null) console.log(x.length);

The compiler is able to do this with an if statement with an unknown condition:

Why is the earlier case being treated differently, from a design point of view? I believe this is the real issue here.

At the same time, I don't believe the second issue of widening the variable inferred type is actually an issue:

let x: string | number =  ["hello", 5][0];
if (typeof x === 'string') {
    let arr = instantInvoke(() => x.length);
}

Here, the compiler is assuming the worst, as it should: either the lambda is invoked immediately (it's a string, due to the if), or it's invoked later (variable could've been changed to a different value - any string | number). So, the inferred type should be string | (string | number), which simplifies to just string | number.

[SergeyPoznyakAkvelon]

Workaround that worked for me:

const _hack: { myMutableVariable: string | null } = {myMutableVariable: null}

immediateInvoke(() => { _hack.myMytableVariable = 'hello' })

_hack.myMutableVariable // type: string | null

[egucciar]

+1 to @rChaoz recommendation to "assume the worst" , narrowing the type such that it won't let you write proper code seems like a weird bug in TS

[egucciar]

@MartinJohns can you help me understand why this is a desirable experience?

function testA() {
    let a: string | null = null;
    const list = [1];
    list.forEach((item) => {
        a = "hello";
    });
    if (a !== null) {
        a.trim(); // TS2339: Property 'trim' does not exist on type 'never'.
    }
}

Wouldn't it make more sense to preserve the type string | null since TS cannot assume the callback within the forEach did not run in between? Why should typescript be enforcing things that are not actually issues?

[egucciar]

I read through some of the related comment threads and it appears as though this is not something TypeScript will fix due to it widening the types / removing narrowing hence breaking existing workflows. I saw some suggestions to add compilerOptions but this was rejected. I honestly rarely run into these issues with "real" code regardless and I would rather use a .reduce (or other array functions) than do a callback such as this, though its very surprising this is the existing behavior with no way to opt out, I'd rarely want to write my code this way to begin with.

[DmSor]

Workaround that worked for me:

let x = "OK" as string | number;

[fatcerberus]

@rChaoz You're basically asking that TS be pessimistic in all cases - #9998 talks about this, but in summary: The compiler doesn't analyze the contents of called functions or passed callbacks for performance reasons (and sometimes doesn't even know what a called function does), and cancelling all narrowings every time any function is called would be impractical (people would just start putting type assertions everywhere) - hence the request for an immediate annotation to opt in to the more expensive analysis on a function-by-function basis.

Anyway... #56407 involves an array.map() and this made me realize, being able to annotate as immediately-invoked isn't actually enough--we'd also need a way to annotate callbacks that should be analyzed as loop bodies.

[malthe]

For mystery and Promise the problem right now is that the compiler assumes deferred, refusing to widen the type. With an immediate typing, it could widen the type but as @aluanhaddad mentions, at the cost of a redundant check.

Perhaps immediate always is an acceptable solution?

• defer – this is the default behavior which gives 'item.data' is possibly 'undefined' in Ternary operator type daemon failed #56407 and incorrectly narrows the type in the mystery example.
• immediate – this would fix 'item.data' is possibly 'undefined' and widen the type in the mystery example.
• immediate always – this would fix the mystery example.

Since defer represents the default semantics, there's no reason to have the keyword of course.

@RyanCavanaugh I assume that it's still the case that "reachability and control flow analysis is done before typechecking" as you mentioned in #9757 (comment). That probably means this is all pretty unrealistic, but even so it would be useful to agree on the ambition.

Here's another example that illustrates why the current narrowing is bad:

class Foo {
    private n?: number;

    async test() {
        this.n = 123;
        await new Promise(resolve => {
            this.n = undefined;
            setTimeout(resolve, 1000);
        });
        const n: number = this.n;
        console.log(n);
    }
}

The narrowing of this.n to just number after await is incorrect:

1. The arrow function is called immediately.
2. If the object is used concurrently, this.n could have been reassigned this way too.

[Igorbek]

Regarding the annotation using defer, immediate, immediate always, etc, I think that number of possible permutations to describe the possible CFA states would be enormous and even if these seem to solve some of the common case, it still would be limited in what you can possibly describe.

As a discussion, I would suggest considering some form of richer assertions about the function intent as part of their signature. My idea is to have some asserts block in the return type position which can contain some limited code that would make some sense to CFA to infer possible states at the call site.

declare function mystery1(cb: () => void): void & asserts {
  cb() // same as 'immediate always', but also says it is invoked exactly once
}

declare function mystery2(cb: () => void): void & asserts {
  if (_) {
    cb() // invoked optionally, but immediately (sync)
  }
}

declare function mystery3(cond: boolean, cb: () => void): void & asserts {
  if (cond) { // the condition can be used in CFA too
    cb() // invoked optionally based on provided condition, but immediately (sync) and once only
  }
}

mystery3(true, () => { ... }) // CFA knows it is called
mystery3(false, () => { ... }) // CFA knows it is not called

declare function mystery4(cb: () => void): Promise<void> & asserts {
  await _
  cb() // is called asynchronously, same as 'defer'
}

The allowed code in the asserts { ... } block should be very limited subset of TS with clear patterns (the subset of what CFA can make sense of) what it would mean:

• invoking of the callbacks, based on known or unknown (_ as an example of an unknown sigil) conditions
• using await to indicate that some are deferred
• number of calls
  • cb(); cb(); would say it is invoked twice
  • while (_) { cb() } would say it is possibly invoked multiple times (0+)
• order of invoking: a(); b(); would imply b is called after a

[dead-claudia]

@Igorbek Callback timing doesn't suffer from that much combinatorial explosion - there's only four possible combinations (the comment you're replying to misses one of them). And everything else (is, asserts, etc.) is orthogonal.

Think of it this way:

• A function can only either 1. be potentially called after return or 2. not potentially be called after return. The presence of immediate indicates 2, and the absence of it indicates 1.
• A function can only either 1. be always called some point before return or 2. potentially not be called before return. The presence of always indicates 1, and the absence of it indicates 2.

These two aren't mutually exclusive on the surface:

• func in new Promise(func) should have both immediate and always. It's always called synchronously.
• func in array.filter(func) should have immediate but not always. The array may be empty, but is never called async.
• func in queueMicrotask(func) should (in browsers) have always but not immediate in its type. It's always scheduled to run async, assuming the event loop isn't shut down. (If/when it becomes cancellable, that may change.)
• func in setTimeout(func, ms) should have neither always nor immediate in its type. The timer could get cancelled.

For control flow checking, the last two have the same effect. Async doesn't imply timing at all, and so there's no use in allowing always without immediate. immediate without always is not unlike initializing/assigning inside an if, so it could still impact flow-sensitive typing.

[nicosampler]

Workaround that worked for me:

let x = "OK" as string | number;

I would give you a hug if I could