Deduplicate project remote dependencies

[paulcadman]

Goal

The goal of this PR is to deduplicate all dependencies in a Juvix project.

Two dependencies are identical when:

• For path dependencies, their paths are equal
• For git dependencies, their URL and their resolved revision (i.e the git revision hash after resolving a tag) are equal

For example in the following dependency tree, where each of the named dependencies represent identical git dependencies.

MyPkg
|
|-- Dep1-hash1
|   |
|   |-- Dep2-hash2
|   |   |
|   |   `-- Stdlib-hash3
|   |
|   `-- Stdlib-hash3
|
|-- Dep2-hash2
|   |
|   `-- Stdlib-hash3
|
`-- Stdlib-hash3

The project MyPkg should just contain the following dependencies: Dep1-hash1, Dep2-hash2, Stdlib-hash3.

Design

Storage of transitive dependencies

Currently the transitive dependencies of a project are fetched/stored in .juvix-build directories of the corresponding dependencies. After this PR all dependencies, including transitive ones are stored in the .juvix-build directory of the root project.

Again, assuming that all the transitive dependencies have the same git hash, in the file system we label the dependencies with their git hash.

MyPkg
|
`- .juvix-build
          |- Dep1-hash1
          |- Dep2-hash2
          `- Stdlib-hash3

Say we have two versions of Dep2 in the transitive dependency graph:

MyPkg
|
|-- Dep1-hash1
|   |
|   |-- Dep2-hash2
|   |   |
|   |   `-- Stdlib-hash3
|   |
|   `-- Stdlib-hash3
|
|-- Dep2-hash4
|   |
|   `-- Stdlib-hash3
|
`-- Stdlib-hash3

we would have two copies of Dep2 in the .juvix-build directory with different hashes:

MyPkg
|
`- .juvix-build
          |- Dep1-hash1
          |- Dep2-hash2
          |- Dep2-hash4
          `- Stdlib-hash3

Storage of git clones

As a consequence of this design we cannot store the git clones for each dependency in the .juvix-build directory as we do now.

We now store the git clones in a global directory ~/.config/juvix/0.6.1/git-cache.

When a dependency at a particular revision is required, the global git clone is fetched/checked out at the required revision and copied into the .juvix-build directory of the relevant project.

Naming of git clones

The requirement for the naming of the global git clones is that they can be identified by URL.

In this PR the name of a clone is formed by taking the SHA256 hash of the dependency git URL. This is to avoid issues with file-system safe escaping of characters.

Naming of dependency directories

The requirement for the naming of the dependency directories is that they can be identified by URL.
/ revision in accordance with our definition of identical dependencies.

In this PR the name of a clone is formed by taking the SHA256 hash of the concatenation of the dependency git URL and git revision. This is to avoid issues with file-system safe escaping of characters.

The downside of this approach is that it's hard to see which directories correspond to which dependencies when navigating the filesystem. However, navigating using the Juvix tooling by using go-to-definition etc. will continue to work as before.

Benchmarks

I tested using juvix-containers test Main.juvix.

The following benchmarks show timings excluding the initial clone of dependencies (which happens in the warmup run).

Before:

$ juvix clean && juvix clean -g
$ hyperfine -w 1 'juvix compile native Main.juvix'
Benchmark 1: juvix compile native Main.juvix
  Time (mean ± σ):      5.598 s ±  0.410 s    [User: 5.020 s, System: 0.586 s]
  Range (min … max):    5.106 s …  6.382 s    10 runs

After:

$ juvix clean && juvix clean -g
$ hyperfine -w 1 'juvix compile native Main.juvix'
Benchmark 1: juvix compile native Main.juvix
  Time (mean ± σ):      4.418 s ±  0.241 s    [User: 4.083 s, System: 0.343 s]
  Range (min … max):    4.237 s …  4.927 s    10 runs

The time saved is due to the fact that before the project depends on 2 copies of the stdlib and after the project depends on 1 copy of the stdlib.

Time is also saved in the initial run because the stdlib is only cloned once instead of twice. The cached stdlib clone is also shared between all project which will improve the performance of all projects that use the stdlib.

Closes

• Deduplicate identical Package dependencies #2760