Code loading parses TOML files too many times

[KristofferC]

With the new TOML parser in Base, the situation that the project / manifest file was opened a huge number of times was slightly improved, but it could still be much better. For example (with some debug info attached):

julia> using Debugger
pid: 24502, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 24502, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
[ Info: Precompiling Debugger [31a5f54b-26ea-5ae9-a837-f05ce5417438]
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25296, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25296, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25295, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26095, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26095, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26095, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26095, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 26285, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Project.toml
pid: 25294, parsing /Users/kristoffercarlsson/.julia/environments/v1.6/Manifest.toml

As a first step, we could pass the parsed TOML dict to the precompilation process.

In general, it would be better to do a full path lookup on the master process (ref #37632) and then just invalidate that whenever needed.
The path info is very dynamic though so caching is pretty hard, some things that invalidate it are:

• Changes to the Project.toml and Manifest.toml in any of the LOAD_PATH .
• Changes to the entries in LOAD_PATH.
• Even just installing new packages into .julia/packages can cause something that just to return nothing as a path (because it couldn't find the package) to returning the path (in the same way as defining a new method can invalidate a previous UndefVarError). This one can probably be fixed by delaying the error throwing for a non-installed package to the time where the package actually tries to load.