Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report unmeasurable variance when measurements are circular #45578

Closed
wants to merge 3 commits into from

Conversation

andrewbranch
Copy link
Member

Fixes #44572 - see my comment on the bug for a detailed explanation of what was happening.

@typescript-bot typescript-bot added Author: Team For Milestone Bug PRs that fix a bug with a specific milestone labels Aug 25, 2021
src/compiler/checker.ts Outdated Show resolved Hide resolved
src/compiler/checker.ts Outdated Show resolved Hide resolved
@ahejlsberg
Copy link
Member

Instead of the refactoring to return Ternary instead of boolean, couldn't you just call the outofbandVarianceMarkerHandler and rely on the existing mechanism for reporting unmeasurable variance? It seems like we otherwise end up with two ways of doing it.

Type 'string' is not assignable to type '"a"'.
Types of property 'x' are incompatible.
Type 'string' is not assignable to type '"a"'.
tests/cases/compiler/varianceMeasurement.ts(11,7): error TS2322: Type 'Foo1<string>' is not assignable to type 'Foo1<unknown>'.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new error comes after a comment that says

// The type below should be invariant in T but is measured as covariant because
// we don't analyze recursive references.

interface Foo1<T> {
  x: T;
  y: Foo1<(arg: T) => void>;
}

declare const f10: Foo1<string>;
const f11: Foo1<'a'> = f10;
const f12: Foo1<unknown> = f10;

👀

@andrewbranch andrewbranch changed the title Report unmeasurable variance when covariant and contravariant comparisons are circular Report unmeasurable variance when measurements are circular Aug 26, 2021
@andrewbranch
Copy link
Member Author

andrewbranch commented Aug 27, 2021

It’s OOMing? It worked on my machine...

// effectively means we measure variance only from type parameter occurrences that aren't nested in
// recursive instantiations of the generic type.
if (variances === emptyArray) {
outofbandVarianceMarkerHandler!(/*onlyUnreliable*/ false);
Copy link
Member

@DanielRosenwasser DanielRosenwasser Aug 27, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we know that outofbandVarianceMarkerHandler will always be defined when variances is emptyArray? Kinda sketchy, but I guess so is the current mechanism.

@DanielRosenwasser
Copy link
Member

?

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 27, 2021

Heya @DanielRosenwasser, I've started to run the perf test suite on this PR at 8194a58. You can monitor the build here.

Update: The results are in!

@ahejlsberg
Copy link
Member

I was experimenting with effectively the same change as you now have in the PR, and did indeed see the tests stall out towards the end of the test run. I didn't let it run for long enough to get an OOM, but I'm sure that would have eventually happened. I'm guessing the issue is that being more aggressive about marking variances as unmeasurable causes us to resort to structural comparisons in more cases, and apparently that goes badly. Would be interesting to see which test is to blame and whether there's something specific we can do about it.

@andrewbranch
Copy link
Member Author

@ahejlsberg one thing I’ve been having trouble conceptualizing is what it means if only one of the two the assignability checks with marker types (i.e., either the covariant or contravariant check) is circular, while the other comes back as definitely assignable or definitely unassignable. In my first iteration of this PR, I only set the variance to unmeasurable if both checks were circular, and tests passed, but it didn’t fix the result in varianceMeasurement.ts.

@ahejlsberg
Copy link
Member

@andrewbranch Variance measurement requires us to successfully compare two type instantiations in both directions, so if we fail in either direction, we have to consider the their variance to be unmeasurable.

@ahejlsberg
Copy link
Member

@andrewbranch One way to think of it is that variance is a measurement of whether two types A and B substituted for a type parameter are compared in one direction, in the other direction, in both directions, or not at all. The only way we can know this is when we succeed for the marker types in both directions.

@andrewbranch
Copy link
Member Author

The OOMing test is tests/cases/compiler/complexRecursiveCollections.ts

@ahejlsberg
Copy link
Member

Hmm, yeah, I can see how those types are a disaster in a structural comparison. For example, to structurally compare two List<T> instantiations, we fan out to an insane number of comparisons between instantiations of the same generic types for different type parameters, which again fans out more, etc. The cycles are very long and apparently don't get to a depth of five before we run out of memory. I stopped it in the debugger and observed a relation stack 70 levels deep that still wasn't considered deeply nested. Tough one!

@andrewbranch
Copy link
Member Author

Variance measurement requires us to successfully compare two type instantiations in both directions, so if we fail in either direction, we have to consider the their variance to be unmeasurable.

But for practical purposes, if we can determine that Foo<MarkerSubType> is assignable to Foo<MarkerSuperType>, while we cannot determine the reverse, we could still use that info to avoid a structural comparison for Foo<T>’s assignability to Foo<U> where T is assignable to U, right? But we’d have to resort to a structural comparison for the reverse.

@DanielRosenwasser
Copy link
Member

DanielRosenwasser commented Aug 27, 2021

we could still use that info to avoid a structural comparison for Foo<T>’s assignability to Foo<U> where T is assignable to U, right? But we’d have to resort to a structural comparison for the reverse.

Just to check, you're saying that in the case where Foo's type parameter is reliably covariant, but not reliably contravariant, then when checking if Foo<T> is assignable to Foo<U>, you'd first check if T is assignable to U. If it is, you can immediately stop; but if not, you have to do a full structural comparison. Right? That does makes sense to me.

@andrewbranch
Copy link
Member Author

That’s almost, but not quite, what I’m saying (and what I’m saying is a question).

Just to check, you're saying that in the case where Foo's type parameter is reliably covariant, but not reliably contravariant, then when checking if Foo<T> is assignable to Foo<U>, you'd first check if T is assignable to U. If it is, you can immediately stop; but if not, you have to do a full structural comparison. Right? That does makes sense to me.

In the case where Foo’s type parameter is measurably covariant, but not measurably contravariant, then when checking if Foo<T> is assignable to Foo<U> you’d check if T is assignable to U, and that’s your whole result. But if you were checking if Foo<U> is assignable to Foo<T>, you’d have to do a structural comparison from the start.

@andrewbranch
Copy link
Member Author

This question is apparently relevant to the OOMing test, because apparently variance probing in one direction was working, because there were no baseline changes and no OOMing when I restricted my change to both directions being unmeasurable. But, it feels like that must mean our analysis of the assignability in that test is either wrong, or only coincidentally right. Because what the current state of my PR is showing us is that at least one direction of variance probing on these types is inconclusive, but we’ve previously believed we know what the variance is, and were using that instead of structural comparisons to decide the assignability of List<T> to List<U>. That must mean we have a bug, right? (Well, we obviously do because it’s mentioned in varianceMeasurements.errors.txt—I assume that’s the bug I’m describing).

@typescript-bot
Copy link
Collaborator

@DanielRosenwasser
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..45578

Metric main 45578 Delta Best Worst
Angular - node (v10.16.3, x64)
Memory used 351,106k (± 0.02%) 351,228k (± 0.02%) +122k (+ 0.03%) 351,067k 351,364k
Parse Time 1.92s (± 1.88%) 1.90s (± 0.62%) -0.02s (- 0.83%) 1.88s 1.94s
Bind Time 0.87s (± 0.89%) 0.85s (± 0.87%) -0.02s (- 2.07%) 0.83s 0.87s
Check Time 5.40s (± 0.48%) 5.39s (± 0.57%) -0.01s (- 0.17%) 5.30s 5.43s
Emit Time 5.86s (± 0.50%) 5.82s (± 0.45%) -0.04s (- 0.68%) 5.77s 5.88s
Total Time 14.04s (± 0.40%) 13.96s (± 0.23%) -0.08s (- 0.61%) 13.91s 14.06s
Compiler-Unions - node (v10.16.3, x64)
Memory used 203,794k (± 0.39%) 203,354k (± 0.02%) -440k (- 0.22%) 203,262k 203,476k
Parse Time 0.78s (± 1.04%) 0.78s (± 0.97%) +0.00s (+ 0.13%) 0.77s 0.80s
Bind Time 0.53s (± 1.14%) 0.53s (± 1.30%) -0.00s (- 0.00%) 0.51s 0.54s
Check Time 7.84s (± 0.57%) 7.77s (± 0.32%) -0.07s (- 0.91%) 7.71s 7.82s
Emit Time 2.44s (± 0.39%) 2.42s (± 0.60%) -0.02s (- 0.70%) 2.38s 2.44s
Total Time 11.59s (± 0.41%) 11.50s (± 0.22%) -0.09s (- 0.79%) 11.43s 11.56s
Monaco - node (v10.16.3, x64)
Memory used 340,673k (± 0.02%) 340,591k (± 0.02%) -82k (- 0.02%) 340,391k 340,791k
Parse Time 1.46s (± 1.20%) 1.45s (± 0.83%) -0.01s (- 0.55%) 1.41s 1.47s
Bind Time 0.74s (± 0.95%) 0.75s (± 0.74%) +0.01s (+ 0.94%) 0.74s 0.76s
Check Time 5.42s (± 0.46%) 5.42s (± 0.86%) -0.00s (- 0.02%) 5.31s 5.51s
Emit Time 3.13s (± 0.66%) 3.19s (± 1.27%) +0.06s (+ 2.01%) 3.13s 3.31s
Total Time 10.75s (± 0.42%) 10.81s (± 0.45%) +0.06s (+ 0.57%) 10.71s 10.93s
TFS - node (v10.16.3, x64)
Memory used 304,064k (± 0.03%) 304,026k (± 0.02%) -38k (- 0.01%) 303,877k 304,175k
Parse Time 1.19s (± 0.97%) 1.18s (± 0.29%) -0.00s (- 0.34%) 1.18s 1.19s
Bind Time 0.71s (± 0.56%) 0.71s (± 1.13%) -0.00s (- 0.42%) 0.69s 0.73s
Check Time 4.89s (± 0.45%) 4.92s (± 0.50%) +0.02s (+ 0.45%) 4.85s 4.97s
Emit Time 3.32s (± 1.40%) 3.31s (± 0.82%) -0.00s (- 0.09%) 3.24s 3.36s
Total Time 10.11s (± 0.46%) 10.12s (± 0.31%) +0.01s (+ 0.11%) 10.06s 10.20s
material-ui - node (v10.16.3, x64)
Memory used 469,701k (± 0.01%) 469,860k (± 0.02%) +158k (+ 0.03%) 469,705k 469,994k
Parse Time 1.74s (± 2.11%) 1.72s (± 0.48%) -0.01s (- 0.81%) 1.71s 1.75s
Bind Time 0.67s (± 0.60%) 0.67s (± 0.56%) -0.00s (- 0.60%) 0.66s 0.67s
Check Time 14.13s (± 0.35%) 14.15s (± 0.58%) +0.02s (+ 0.16%) 14.01s 14.36s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 16.54s (± 0.31%) 16.54s (± 0.51%) +0.00s (+ 0.02%) 16.39s 16.75s
Angular - node (v12.1.0, x64)
Memory used 329,162k (± 0.03%) 329,203k (± 0.07%) +41k (+ 0.01%) 328,395k 329,552k
Parse Time 1.90s (± 0.65%) 1.89s (± 0.79%) -0.01s (- 0.47%) 1.86s 1.92s
Bind Time 0.86s (± 0.43%) 0.85s (± 0.56%) -0.02s (- 2.08%) 0.84s 0.86s
Check Time 5.28s (± 0.50%) 5.26s (± 0.74%) -0.02s (- 0.44%) 5.19s 5.39s
Emit Time 6.14s (± 0.84%) 6.10s (± 0.64%) -0.04s (- 0.60%) 6.01s 6.20s
Total Time 14.18s (± 0.58%) 14.09s (± 0.47%) -0.09s (- 0.62%) 13.97s 14.27s
Compiler-Unions - node (v12.1.0, x64)
Memory used 190,778k (± 0.15%) 190,818k (± 0.09%) +40k (+ 0.02%) 190,172k 191,032k
Parse Time 0.78s (± 0.57%) 0.78s (± 0.83%) -0.00s (- 0.51%) 0.77s 0.79s
Bind Time 0.54s (± 0.68%) 0.53s (± 0.68%) -0.00s (- 0.37%) 0.53s 0.54s
Check Time 7.37s (± 0.83%) 7.32s (± 0.53%) -0.05s (- 0.69%) 7.26s 7.41s
Emit Time 2.45s (± 0.77%) 2.42s (± 0.71%) -0.03s (- 1.10%) 2.38s 2.47s
Total Time 11.14s (± 0.62%) 11.06s (± 0.38%) -0.08s (- 0.72%) 10.98s 11.15s
Monaco - node (v12.1.0, x64)
Memory used 323,773k (± 0.01%) 323,732k (± 0.01%) -41k (- 0.01%) 323,615k 323,797k
Parse Time 1.43s (± 0.66%) 1.43s (± 0.61%) +0.00s (+ 0.00%) 1.41s 1.46s
Bind Time 0.73s (± 0.91%) 0.72s (± 0.86%) -0.01s (- 1.23%) 0.71s 0.73s
Check Time 5.30s (± 0.51%) 5.28s (± 0.54%) -0.02s (- 0.30%) 5.23s 5.35s
Emit Time 3.21s (± 0.49%) 3.18s (± 0.43%) -0.03s (- 0.93%) 3.15s 3.22s
Total Time 10.67s (± 0.39%) 10.61s (± 0.36%) -0.05s (- 0.47%) 10.55s 10.72s
TFS - node (v12.1.0, x64)
Memory used 288,800k (± 0.02%) 288,704k (± 0.02%) -95k (- 0.03%) 288,601k 288,931k
Parse Time 1.21s (± 0.90%) 1.21s (± 0.78%) -0.00s (- 0.08%) 1.18s 1.22s
Bind Time 0.70s (± 1.07%) 0.69s (± 0.86%) -0.00s (- 0.43%) 0.68s 0.71s
Check Time 4.85s (± 0.45%) 4.85s (± 0.69%) -0.00s (- 0.04%) 4.78s 4.96s
Emit Time 3.37s (± 1.20%) 3.35s (± 1.26%) -0.01s (- 0.45%) 3.26s 3.46s
Total Time 10.13s (± 0.51%) 10.11s (± 0.63%) -0.02s (- 0.20%) 10.00s 10.25s
material-ui - node (v12.1.0, x64)
Memory used 448,530k (± 0.02%) 448,471k (± 0.06%) -59k (- 0.01%) 447,362k 448,854k
Parse Time 1.74s (± 0.51%) 1.73s (± 0.64%) -0.01s (- 0.58%) 1.71s 1.75s
Bind Time 0.65s (± 1.16%) 0.66s (± 1.37%) +0.00s (+ 0.15%) 0.63s 0.68s
Check Time 12.91s (± 0.47%) 12.84s (± 0.93%) -0.06s (- 0.50%) 12.69s 13.25s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.30s (± 0.36%) 15.22s (± 0.80%) -0.07s (- 0.48%) 15.05s 15.64s
Angular - node (v14.15.1, x64)
Memory used 327,782k (± 0.00%) 327,861k (± 0.01%) +80k (+ 0.02%) 327,797k 327,942k
Parse Time 1.92s (± 0.51%) 1.90s (± 0.40%) -0.02s (- 0.94%) 1.88s 1.92s
Bind Time 0.91s (± 0.77%) 0.88s (± 0.68%) 🟩-0.04s (- 3.84%) 0.86s 0.89s
Check Time 5.33s (± 0.48%) 5.28s (± 0.44%) -0.05s (- 0.94%) 5.23s 5.34s
Emit Time 6.26s (± 0.63%) 6.19s (± 0.57%) -0.07s (- 1.10%) 6.11s 6.29s
Total Time 14.42s (± 0.51%) 14.24s (± 0.35%) -0.17s (- 1.21%) 14.16s 14.42s
Compiler-Unions - node (v14.15.1, x64)
Memory used 190,432k (± 0.51%) 191,162k (± 0.62%) +731k (+ 0.38%) 189,523k 192,806k
Parse Time 0.81s (± 0.61%) 0.80s (± 0.61%) -0.00s (- 0.25%) 0.80s 0.82s
Bind Time 0.57s (± 0.60%) 0.56s (± 0.59%) -0.01s (- 1.23%) 0.55s 0.57s
Check Time 7.55s (± 0.86%) 7.49s (± 0.60%) -0.06s (- 0.76%) 7.42s 7.58s
Emit Time 2.44s (± 0.75%) 2.43s (± 0.76%) -0.01s (- 0.61%) 2.39s 2.48s
Total Time 11.36s (± 0.58%) 11.28s (± 0.38%) -0.08s (- 0.71%) 11.19s 11.38s
Monaco - node (v14.15.1, x64)
Memory used 322,558k (± 0.00%) 322,537k (± 0.00%) -21k (- 0.01%) 322,504k 322,566k
Parse Time 1.49s (± 0.45%) 1.49s (± 0.68%) -0.01s (- 0.47%) 1.46s 1.50s
Bind Time 0.75s (± 0.65%) 0.75s (± 0.30%) -0.00s (- 0.40%) 0.75s 0.76s
Check Time 5.21s (± 0.44%) 5.21s (± 0.45%) +0.01s (+ 0.13%) 5.15s 5.27s
Emit Time 3.25s (± 0.73%) 3.23s (± 0.97%) -0.02s (- 0.52%) 3.18s 3.32s
Total Time 10.71s (± 0.28%) 10.69s (± 0.46%) -0.02s (- 0.21%) 10.59s 10.79s
TFS - node (v14.15.1, x64)
Memory used 287,718k (± 0.01%) 287,690k (± 0.01%) -28k (- 0.01%) 287,658k 287,724k
Parse Time 1.25s (± 1.84%) 1.24s (± 2.84%) -0.01s (- 0.72%) 1.20s 1.37s
Bind Time 0.74s (± 4.38%) 0.74s (± 3.45%) -0.00s (- 0.54%) 0.71s 0.83s
Check Time 4.88s (± 0.60%) 4.87s (± 0.51%) -0.01s (- 0.10%) 4.80s 4.92s
Emit Time 3.47s (± 0.77%) 3.46s (± 0.77%) -0.01s (- 0.17%) 3.40s 3.51s
Total Time 10.34s (± 0.57%) 10.32s (± 0.37%) -0.02s (- 0.20%) 10.25s 10.42s
material-ui - node (v14.15.1, x64)
Memory used 446,826k (± 0.03%) 446,960k (± 0.01%) +134k (+ 0.03%) 446,890k 446,997k
Parse Time 1.76s (± 0.40%) 1.76s (± 0.73%) +0.00s (+ 0.06%) 1.74s 1.80s
Bind Time 0.69s (± 0.43%) 0.69s (± 0.68%) +0.00s (+ 0.29%) 0.68s 0.70s
Check Time 12.92s (± 0.33%) 12.90s (± 0.57%) -0.02s (- 0.13%) 12.75s 13.05s
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) 0.00s ( NaN%) 0.00s 0.00s
Total Time 15.37s (± 0.30%) 15.35s (± 0.53%) -0.01s (- 0.08%) 15.17s 15.49s
System
Machine Namets-ci-ubuntu
Platformlinux 4.4.0-210-generic
Architecturex64
Available Memory16 GB
Available Memory11 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v10.16.3, x64)
  • node (v12.1.0, x64)
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v10.16.3, x64)
  • Angular - node (v12.1.0, x64)
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v10.16.3, x64)
  • Compiler-Unions - node (v12.1.0, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v10.16.3, x64)
  • Monaco - node (v12.1.0, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v10.16.3, x64)
  • TFS - node (v12.1.0, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v10.16.3, x64)
  • material-ui - node (v12.1.0, x64)
  • material-ui - node (v14.15.1, x64)
Benchmark Name Iterations
Current 45578 10
Baseline main 10

Developer Information:

Download Benchmark

@ahejlsberg
Copy link
Member

Took a step back and dug a little deeper to understand why the repro posted by Ryan here works when the B type parameter is deleted--which is really odd.

I now understand what the real problem is.

We have logic in structuredTypeRelatedTo that skips over variance checks for type alias instantiations and type references that contain marker types in their type arguments. The intent is to force structural comparisons during variance computations. However, the logic only skips for certain instantiations involving marker types, namely those where one type argument is a marker type and the remaining type arguments are just the type parameters themselves. That means we don't skip over variance checking for a type instantiation like Child<?, unknown> (where ? is a marker type), and that's the kind of instantiation we get in the example because of the defaulted type argument for B. So, we start reporting Ternary.Unknown results where really we should be doing structural comparisons.

Changing our logic to skip variance checking for any type instantiation where at least one type argument is a marker type fixes the issue. It causes us to do a bit more structural comparing when computing variances, but obviously it was wrong not to do this extra work. I think this is the solution we want to go with. I will put up a new PR and we'll see what the performance impact is, if any.

@ahejlsberg
Copy link
Member

Alternative PR is up at #45628.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Milestone Bug PRs that fix a bug with a specific milestone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check order dependence with mutually-recursive non-unary generics
4 participants