Make tests verify inference (not any!)

[KiaraGrouwstra]

If TS infers a function to, say, return any rather than number, can we make that a failing test?

@blakeembrey:

I think it'd be easy enough to write a little script for it. Just run tsc for x.pass.ts and x.fail.ts and make sure the error codes are correct each execution.

This taught me something new: we can verify that certain commands output errors as expected.
In retrospect though, the problem we're faced with here is that tests are passing while they shouldn't (no type properly inferred, so it'll just accept anything). I've yet to come up with a solution here.

[masaeedu]

@tycho01 You might look at how TypeScript itself does its tests. They have a harness where the tests are TypeScript files, and when they run the tests the appropriate error, sourcemap and JS is generated in an output directory. You tweak the output files so they look like what you expect, and the next time you run the tests they'll fail because the expected output files don't match the transpilation artifacts from the TypeScript.

In your case you'd expect certain errors to appear in the errors file.

[KiaraGrouwstra]

Interesting, thanks! I'll try to look into this. :)

[blakeembrey]

I assume you're referring to the internal tests for the compiler API. I'm not sure that sort of thing would really be needed, but it's always worth looking at. Seems easier just to write the output to a snapshot and make sure the snapshots don't change here (e.g. the error stays in the right place, or there's no errors, etc).

[masaeedu]

@blakeembrey The tests I'm referring to are not related to the compiler API (I think those are separate). The tests I'm talking about test core TypeScript features for various use cases (e.g. do all ES6 features transpile correctly? do the appropriate type errors appear for various type checking features? etc).

Your suggestion to ensure output snapshot remains consistent is actually what they use. There's a bunch of TypeScript files representing test scenarios in the tests/cases folder, and snapshots of the corresponding expected output in the tests/baselines/reference folder. To prevent regressions, they simply ensure the output for each TypeScript file hasn't deviated from what's stored in baselines/reference in an unexpected way.

However, since this codebase is fairly new and your primary concern is verifying correctness of new code rather than preventing regressions, you'd have to examine and tweak the output files by hand to get any failing tests.

[masaeedu]

You can even assert for the type of various expressions, which should be useful when testing whether inference works correctly.

[KiaraGrouwstra]

That looks great, I love the baseline thing, seems like a great way of getting a better idea of changes in inference. Looks like I could borrow some inspiration from their associated tasks here.

[KiaraGrouwstra]

@masaeedu: had you had experience with this test harness? I've been attempting to make a similar gulp file mimicking how they're doing it, but I'm having trouble actually reproducing their way.

Specifically, it seems internally they're relying on the availability of this Harness, which appears to be in scope by magic rather than by import. The closest thing to an import I see is the /// <reference path="harness.ts"/>, but that obviously only grabs typings.
Trying to import from that harness file yields is not a module, though apparently it's not actually exporting its namespaces anyway, which lends this some credibility.

Otherwise I might wanna go ask over there...

[KiaraGrouwstra]

The TS guys pointed me to a repo that did this, still an outstanding issue but otherwise usable. Just pushed this in, with results committed to allow checking diffs.