deduplicate references lists in tsbuildinfo #43079
Conversation
typescript-bot
added
the
For Uncommitted Bug
|
This PR doesn't have any linked issues. Please open an issue that references this PR. From there we can discuss and prioritise. |
src/compiler/builder.ts
Outdated
| referencedMap?: MapLike<string[] | number>; | ||
| referencedMapLists?: string[][]; | ||
| exportedModulesMap?: MapLike<string[] | number>; | ||
| exportedModulesMapLists?: string[][]; |
There was a problem hiding this comment.
I think we can simplify and gain even more if we have single list for referencedMapLists as well as exportedModulesMapLists. Also instead of sometimes deduplicating we can make referencedMap and exportedModulesMap as MapLike<number> and the list of reference set is separate which is deduplicated across the board.
There was a problem hiding this comment.
I did that, and also deduplicated the files itself.
Sadly that makes all baselines unreadable, maybe we should decompress it for the baseline output to make it readable again...
sokra
force-pushed
the
performance/tsbuildinfo-deduplication
branch
from
d29b92e to
e4c7402
Compare
|
Some statistics: For my project this reduces the tsbuildinfo size from 14.9 MB to 2.2MB. |
|
btw. is it necessary to emit a formatted json ( |
|
@sokra Created PR #43155 based off of this one with simplification and additional optimization of fileinfos, semantic diagnostic and your suggestion of getting rid of spaces in tsbuildinfo. |
* Baseline readable buildinfo * Use file names as index in file name list This is extension of the idea given by @sokra to optimize size of tsbuildinfo * Deduplicate reference map lists and use file name index to sort them Different implementation of #43079 based on idea suggested by @sokra * Minimal json.stringify for the tsbuildinfo Again implementaion of suggestion by @sokra * Update src/testRunner/unittests/tsbuild/helpers.ts Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com> * Readable version of buildinfo all the time * Some renames for readability as per feedback Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
|
Closed with #43155 |
When
referencedMaporexportedModulesMapcontain duplicates, they are replaced with an index of the deduplicated value inreferencedMapListsresp.exportedModulesMapLists.This is actually quite common as .d.ts and .ts file often share references.
This saves a bit storage, makes restoring faster and due to using references to a shared list (since immutable) this also saves a bit of memory.
Finding duplicates during storing can be expensive, so we make use of the equal identity when this was restored from tsbuildinfo. Otherwise we create a Map by list length to avoid checking too many lists.
Backlogmilestone (required)masterbranchgulp runtestslocally