GetLocatedImports uses quadratic amount of memory in KnownTargets at startup #4317
Labels
status: needs triage
type: bug
Something isn't right: doesn't work as intended, documentation is missing/outdated, etc..
Comments
mpickering
added
type: bug
mpickering
added a commit
that referenced
this issue
Jun 13, 2024
At startup `GetLocatedImports` is called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently. In `GetLocatedImports` the known targets are consulted and the targetsMap is created by mapping the known targets. This map is used for introducing sharing amongst filepaths. This operation copies a local copy of the `target` map which is local to the rule. ``` let targetsMap = HMap.mapWithKey const targets ``` So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be 10000^2 elements in total allocated in hashmaps. This used a lot of memory. Solution: Return the normalising map in the result of the `GetKnownTargets` rule so it is shared across threads. Fixes #4317
fendor
pushed a commit
that referenced
this issue
Jun 13, 2024
At startup `GetLocatedImports` is called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently. In `GetLocatedImports` the known targets are consulted and the targetsMap is created by mapping the known targets. This map is used for introducing sharing amongst filepaths. This operation copies a local copy of the `target` map which is local to the rule. ``` let targetsMap = HMap.mapWithKey const targets ``` So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be 10000^2 elements in total allocated in hashmaps. This used a lot of memory. Solution: Return the normalising map in the result of the `GetKnownTargets` rule so it is shared across threads. Fixes #4317
fendor
pushed a commit
that referenced
this issue
Jun 13, 2024
At startup `GetLocatedImports` is called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently. In `GetLocatedImports` the known targets are consulted and the targetsMap is created by mapping the known targets. This map is used for introducing sharing amongst filepaths. This operation copies a local copy of the `target` map which is local to the rule. ``` let targetsMap = HMap.mapWithKey const targets ``` So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be 10000^2 elements in total allocated in hashmaps. This used a lot of memory. Solution: Return the normalising map in the result of the `GetKnownTargets` rule so it is shared across threads. Fixes #4317
michaelpj
pushed a commit
that referenced
this issue
Jun 13, 2024
At startup `GetLocatedImports` is called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently. In `GetLocatedImports` the known targets are consulted and the targetsMap is created by mapping the known targets. This map is used for introducing sharing amongst filepaths. This operation copies a local copy of the `target` map which is local to the rule. ``` let targetsMap = HMap.mapWithKey const targets ``` So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be 10000^2 elements in total allocated in hashmaps. This used a lot of memory. Solution: Return the normalising map in the result of the `GetKnownTargets` rule so it is shared across threads. Fixes #4317
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
At startup
GetLocatedImportsis called on all known files. Say you have 10000 modules in your project then this leads to 10000 calls to GetLocatedImports running concurrently.In
GetLocatedImportsthe known targets are consulted and the targetsMap is created by mapping the known targets. This mapis used for introducing sharing amongst filepaths. This operation copies a local copy of the
targetmap which is local to the rule.So now each rule has a hashmap of size 10000 held locally to it and depending on how the threads are scheduled there will be
10000^2 elements in total allocated in hashmaps. This used a lot of memory.
Solution: Return the normalising map in the result of the
GetKnownTargetsrule so it is shared across threads.The text was updated successfully, but these errors were encountered: