fix: reduce io concurrency

[baszalmstra]

This PR tries to implement a better IO concurrency using conda/rattler#980 . We have to test whether this improves performance before merging.

[baszalmstra]

On my local machine running pixi install with the .pixi folder removed takes:

+- 44 seconds with latest pixi.
+- 34 seconds with this PR.

[wolfv]

@baszalmstra are you using DEVDRIVE?

[baszalmstra]

Yes, both cache and target folder are on my devdrive. On an nvme ssd.

[ruben-arts]

Test results for this toml file:

[project]
channels = ["https://prefix.dev/conda-forge"]
name = "hard-to-install"
platforms = ["osx-arm64", "linux-64", "win-64"]

[dependencies]
rust = ">=1.83.0,<1.84"
scipy = ">=1.14.1,<2"

And the following command, assuming pixi is the one from this pr and ~/.pixi/bin/pixi is the latest release:

hyperfine -p "pixi clean" "pixi install" "~/.pixi/bin/pixi install" -r 3

osx-arm64

Benchmark 1: pixi install
  Time (mean ± σ):     25.292 s ±  9.372 s    [User: 4.967 s, System: 134.405 s]
  Range (min … max):   14.022 s … 35.177 s    5 runs
 
Benchmark 2: ~/.pixi/bin/pixi install
  Time (mean ± σ):     40.804 s ±  2.814 s    [User: 4.449 s, System: 281.403 s]
  Range (min … max):   36.469 s … 43.466 s    5 runs
 
Summary
  pixi install ran
    1.61 ± 0.61 times faster than ~/.pixi/bin/pixi install

With export RAYON_NUM_THREADS=2

Benchmark 1: pixi install
  Time (mean ± σ):     18.613 s ±  3.446 s    [User: 3.651 s, System: 24.948 s]
  Range (min … max):   15.698 s … 24.395 s    5 runs

Benchmark 2: ~/.pixi/bin/pixi install
  Time (mean ± σ):     39.026 s ±  2.633 s    [User: 4.437 s, System: 261.304 s]
  Range (min … max):   36.828 s … 43.429 s    5 runs

Summary
  pixi install ran
    2.10 ± 0.41 times faster than ~/.pixi/bin/pixi install

win-64

Benchmark 1: C:/Users/Ruben/.pixi/bin/pixi.exe install
  Time (mean ± σ):     20.249 s ±  0.254 s    [User: 1.145 s, System: 8.557 s]
  Range (min … max):   19.957 s … 20.421 s    3 runs

Benchmark 2: pixi install
  Time (mean ± σ):      4.621 s ±  0.217 s    [User: 1.291 s, System: 6.671 s]
  Range (min … max):    4.376 s …  4.789 s    3 runs

Summary
  pixi install ran
    4.38 ± 0.21 times faster than C:/Users/Ruben/.pixi/bin/pixi.exe install

And on a Dev Drive:

Benchmark 1: C:/Users/Ruben/.pixi/bin/pixi.exe install
  Time (mean ± σ):     11.789 s ±  8.431 s    [User: 3.864 s, System: 20.723 s]
  Range (min … max):    6.893 s … 21.525 s    3 runs

Benchmark 2: pixi install
  Time (mean ± σ):      7.361 s ±  1.815 s    [User: 1.749 s, System: 7.259 s]
  Range (min … max):    5.870 s …  9.382 s    3 runs

Summary
  pixi install ran
    1.60 ± 1.21 times faster than C:/Users/Ruben/.pixi/bin/pixi.exe install

linux-64

Benchmark 1: pixi install
  Time (mean ± σ):     980.6 ms ±  70.2 ms    [User: 1962.8 ms, System: 2311.0 ms]
  Range (min … max):   928.3 ms … 1060.4 ms    3 runs
 
Benchmark 2: ~/.pixi/bin/pixi install
  Time (mean ± σ):      2.725 s ±  0.541 s    [User: 2.605 s, System: 22.600 s]
  Range (min … max):    2.215 s …  3.292 s    3 runs
 
Summary
  pixi install ran
    2.78 ± 0.59 times faster than ~/.pixi/bin/pixi install

Summary

Only improvements!!! 💯 Some up to 4.38 times speed improvements!

Merge quickly before the gods of speed notice!