Detect malformed UTF-8 files and refuse to engage further #53667

RyanCavanaugh · 2023-04-05T04:27:56Z

Belt-and-suspenders on #53114.

When a file is actually binary, or at least starts with invalid UTF-8 bytes, it's not going to be a good use of our time to try to make sense of it. This PR detects U+FFFD REPLACEMENT CHARACTER in the leading character position and, if encountered, skips to the end of the file.

This fully mitigates both the crashes caused by these enormous binaries effectively fuzzing our parser, as well as the performance penalty of generating e.g. 54,000 errors on a 1.8 MB "JS" file that is actually just line noise.

The "corrupted.ts" file is the verbatim first eight bytes of a repro we got for this.

jakebailey · 2023-04-05T04:32:20Z

I'm paranoid

@typescript-bot test this
@typescript-bot test top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot test tsserver top100
@typescript-bot run dt
@typescript-bot perf test this faster

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the diff-based top-repos suite on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the extended test suite on this PR at 0c78667. You can monitor the build here.

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the diff-based user code test suite on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the diff-based user code test suite (tsserver) on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the abridged perf test suite on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the parallelized Definitely Typed test suite on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:32:23Z

Heya @jakebailey, I've started to run the diff-based top-repos suite (tsserver) on this PR at 0c78667. You can monitor the build here.

Update: The results are in!

typescript-bot · 2023-04-05T04:44:35Z

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/53667/merge:

There were infrastructure failures potentially unrelated to your change:

1 instance of "Package install failed"

Otherwise...

Everything looks good!

typescript-bot · 2023-04-05T04:45:09Z

@jakebailey Here are the results of running the user test suite comparing main and refs/pull/53667/merge:

Everything looks good!

typescript-bot · 2023-04-05T04:47:36Z

@jakebailey
The results of the perf run you requested are in!

Here they are:

Comparison Report - main..53667

_Metric	_main	₅₃₆₆₇	_Delta	_Best	_Worst	_p-value
_{Angular - node (v16.17.1, x64)}
_{Memory used}	_{364,816k (± 0.00%)}	_{364,801k (± 0.01%)}	_~	_364,770k	_364,836k	_{p=0.572 n=6}
_{Parse Time}	_{3.54s (± 0.91%)}	_{3.57s (± 1.01%)}	_~	_3.52s	_3.61s	_{p=0.169 n=6}
_{Bind Time}	_{1.18s (± 1.03%)}	_{1.18s (± 0.71%)}	_~	_1.17s	_1.19s	_{p=0.227 n=6}
_{Check Time}	_{9.53s (± 0.18%)}	_{9.52s (± 0.43%)}	_~	_9.48s	_9.59s	_{p=0.334 n=6}
_{Emit Time}	_{7.97s (± 0.34%)}	_{7.97s (± 0.35%)}	_~	_7.93s	_8.00s	_{p=0.936 n=6}
_{Total Time}	_{22.22s (± 0.29%)}	_{22.24s (± 0.37%)}	_~	_22.13s	_22.38s	_{p=1.000 n=6}
_{Compiler-Unions - node (v16.17.1, x64)}
_{Memory used}	_{192,553k (± 0.03%)}	_{192,538k (± 0.03%)}	_~	_192,449k	_192,635k	_{p=0.810 n=6}
_{Parse Time}	_{1.59s (± 0.76%)}	_{1.59s (± 1.67%)}	_~	_1.54s	_1.61s	_{p=0.681 n=6}
_{Bind Time}	_{0.82s (± 0.50%)}	_{0.82s (± 0.00%)}	_~	_0.82s	_0.82s	_{p=0.405 n=6}
_{Check Time}	_{10.38s (± 0.93%)}	_{10.30s (± 0.38%)}	_{-0.08s (- 0.80%)}	_10.24s	_10.36s	_{p=0.036 n=6}
_{Emit Time}	_{3.00s (± 0.49%)}	_{2.98s (± 0.51%)}	_~	_2.97s	_3.01s	_{p=0.187 n=6}
_{Total Time}	_{15.78s (± 0.64%)}	_{15.69s (± 0.42%)}	_~	_15.57s	_15.75s	_{p=0.109 n=6}
_{Monaco - node (v16.17.1, x64)}
_{Memory used}	_{345,677k (± 0.01%)}	_{345,688k (± 0.00%)}	_~	_345,674k	_345,699k	_{p=0.378 n=6}
_{Parse Time}	_{2.71s (± 0.69%)}	_{2.72s (± 0.36%)}	_~	_2.71s	_2.73s	_{p=0.805 n=6}
_{Bind Time}	_{1.08s (± 0.48%)}	_{1.09s (± 0.37%)}	_~	_1.08s	_1.09s	_{p=0.112 n=6}
_{Check Time}	_{7.77s (± 0.33%)}	_{7.77s (± 0.15%)}	_~	_7.76s	_7.79s	_{p=0.807 n=6}
_{Emit Time}	_{4.47s (± 0.74%)}	_{4.45s (± 0.59%)}	_~	_4.41s	_4.48s	_{p=0.295 n=6}
_{Total Time}	_{16.04s (± 0.17%)}	_{16.03s (± 0.21%)}	_~	_15.97s	_16.06s	_{p=0.421 n=6}
_{TFS - node (v16.17.1, x64)}
_{Memory used}	_{300,052k (± 0.00%)}	_{300,057k (± 0.01%)}	_~	_300,031k	_300,076k	_{p=0.575 n=6}
_{Parse Time}	_{2.16s (± 0.19%)}	_{2.16s (± 0.48%)}	_~	_2.15s	_2.18s	_{p=0.390 n=6}
_{Bind Time}	_{1.23s (± 0.80%)}	_{1.24s (± 0.79%)}	_~	_1.22s	_1.25s	_{p=0.116 n=6}
_{Check Time}	_{7.22s (± 0.29%)}	_{7.25s (± 0.33%)}	_~	_7.20s	_7.27s	_{p=0.071 n=6}
_{Emit Time}	_{4.33s (± 0.43%)}	_{4.32s (± 0.41%)}	_~	_4.31s	_4.35s	_{p=1.000 n=6}
_{Total Time}	_{14.93s (± 0.15%)}	_{14.97s (± 0.22%)}	_{+0.05s (+ 0.32%)}	_14.93s	_15.03s	_{p=0.019 n=6}
_{material-ui - node (v16.17.1, x64)}
_{Memory used}	_{479,408k (± 0.01%)}	_{479,433k (± 0.01%)}	_~	_479,382k	_479,534k	_{p=0.575 n=6}
_{Parse Time}	_{3.24s (± 0.30%)}	_{3.25s (± 0.82%)}	_~	_3.20s	_3.27s	_{p=0.463 n=6}
_{Bind Time}	_{0.95s (± 0.67%)}	_{0.96s (± 1.67%)}	_~	_0.95s	_0.99s	_{p=0.340 n=6}
_{Check Time}	_{18.20s (± 0.21%)}	_{18.19s (± 0.23%)}	_~	_18.14s	_18.25s	_{p=0.748 n=6}
_{Emit Time}	_{0.00s (± 0.00%)}	_{0.00s (± 0.00%)}	_~	_0.00s	_0.00s	_{p=1.000 n=6}
_{Total Time}	_{22.39s (± 0.16%)}	_{22.40s (± 0.22%)}	_~	_22.33s	_22.46s	_{p=0.871 n=6}
_{xstate - node (v16.17.1, x64)}
_{Memory used}	_{559,726k (± 0.01%)}	_{559,823k (± 0.03%)}	_~	_559,572k	_560,068k	_{p=0.471 n=6}
_{Parse Time}	_{3.98s (± 0.34%)}	_{4.00s (± 0.37%)}	_{+0.02s (+ 0.54%)}	_3.98s	_4.02s	_{p=0.044 n=6}
_{Bind Time}	_{1.74s (± 0.57%)}	_{1.74s (± 0.30%)}	_~	_1.73s	_1.74s	_{p=0.504 n=6}
_{Check Time}	_{3.07s (± 0.34%)}	_{3.06s (± 0.27%)}	_{-0.02s (- 0.60%)}	_3.05s	_3.07s	_{p=0.011 n=6}
_{Emit Time}	_{0.09s (± 5.53%)}	_{0.10s (± 5.76%)}	_~	_0.09s	_0.10s	_{p=0.640 n=6}
_{Total Time}	_{8.88s (± 0.30%)}	_{8.89s (± 0.15%)}	_~	_8.87s	_8.91s	_{p=0.683 n=6}

System

_{Machine Name}	_ts-ci-ubuntu
_Platform	_{linux 5.4.0-135-generic}
_Architecture	_x64
_{Available Memory}	_{16 GB}
_{Available Memory}	_{15 GB}
_CPUs	_{4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz}

Hosts

node (v16.17.1, x64)

Scenarios

Angular - node (v16.17.1, x64)
Compiler-Unions - node (v16.17.1, x64)
Monaco - node (v16.17.1, x64)
TFS - node (v16.17.1, x64)
material-ui - node (v16.17.1, x64)
xstate - node (v16.17.1, x64)

_Benchmark	_Name	_Iterations
_Current	₅₃₆₆₇	₆
_Baseline	_main	₆

Developer Information:

Download Benchmark

typescript-bot · 2023-04-05T06:36:28Z

Hey @jakebailey, the results of running the DT tests are ready.
Everything looks the same!
You can check the log here.

typescript-bot · 2023-04-05T06:49:03Z

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/53667/merge:

Everything looks good!

typescript-bot · 2023-04-05T06:55:29Z

@jakebailey Here are the results of running the top-repos suite comparing main and refs/pull/53667/merge:

Everything looks good!

tests/baselines/reference/corrupted.errors.txt

src/compiler/scanner.ts

# Conflicts: # src/compiler/diagnosticMessages.json # tests/baselines/reference/api/tsserverlibrary.d.ts # tests/baselines/reference/api/typescript.d.ts

src/compiler/program.ts

src/compiler/diagnosticMessages.json

RyanCavanaugh added 3 commits April 4, 2023 21:22

Detect malformed UTF-8 files and refuse to engage further

72f20ae

Add baselines

b832d00

Fix description

0c78667

typescript-bot assigned RyanCavanaugh Apr 5, 2023

typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Apr 5, 2023

One more baseline

cc1c688

DanielRosenwasser reviewed Apr 5, 2023

View reviewed changes

tests/baselines/reference/corrupted.errors.txt Outdated Show resolved Hide resolved

sheetalkamat reviewed Apr 5, 2023

View reviewed changes

src/compiler/scanner.ts Show resolved Hide resolved

RyanCavanaugh added 3 commits April 6, 2023 14:41

Use a new error message

51d4cc4

Merge branch 'main' into detectHermes

70d6e3e

# Conflicts: # src/compiler/diagnosticMessages.json # tests/baselines/reference/api/tsserverlibrary.d.ts # tests/baselines/reference/api/typescript.d.ts

Don't print context for binary content

0d45110

jakebailey reviewed Apr 7, 2023

View reviewed changes

src/compiler/program.ts Show resolved Hide resolved

src/compiler/diagnosticMessages.json Outdated Show resolved Hide resolved

.

046940a

jakebailey approved these changes Apr 7, 2023

View reviewed changes

sheetalkamat approved these changes Apr 7, 2023

View reviewed changes

RyanCavanaugh merged commit c89f87f into microsoft:main Apr 7, 2023

RyanCavanaugh deleted the detectHermes branch April 7, 2023 19:51

jakebailey mentioned this pull request Jul 10, 2023

Code Helper attempts to index MPEG-TS files in project and eats 100% cpu #21136

Closed

jakebailey mentioned this pull request Nov 23, 2023

TSC runs out of memory when TransportStream videos are present in the path #56516

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Detect malformed UTF-8 files and refuse to engage further #53667

Detect malformed UTF-8 files and refuse to engage further #53667

RyanCavanaugh commented Apr 5, 2023

jakebailey commented Apr 5, 2023

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

Detect malformed UTF-8 files and refuse to engage further #53667

Detect malformed UTF-8 files and refuse to engage further #53667

Conversation

RyanCavanaugh commented Apr 5, 2023

jakebailey commented Apr 5, 2023

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023 • edited Loading

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading

typescript-bot commented Apr 5, 2023 •

edited

Loading