[2.1.0~alpha] opam show is noticably slower, loading depexts

[kit-ty-kate]

Using opam master (3def1a6), the following command:

opam show ocamlfind

takes less than 0.3s on opam 2.0.7 vs. almost 2s with opam 2.1.0~alpha.

[rjbou]

This can come from the depext check. At each switch load, as for external file dependencies, depext checks that required system packages are still present.
This check takes ~1s on my debian, but @AltGr fix in #4165, it is now ~0.2s. It result on the full command taking ~1s with my huge opam root, and ~0.5 with an empty one.
You can see with --debug timestamps what is the action that is slower in 2.1.0.

[AltGr]

The depext check is lazy and I believe it shouldn't be forced on opam show 🤔
I don't reproduce such a delay, so your --debug output would be helpful.
It is also possible that the repository cache was scrapped (it's the case if e.g. you just updated) ; and it was not yet rebuilt (that needs a write lock, so it's only done on write-enabled commands). Do you reproduce after a simple install ?

[AltGr]

@rjbou found it: there is actually a Lazy.force used to get the latest version, only when you opam show a package that is not installed (in which case, the installed version is shown) ; this explains why we had trouble reproducing.

It would be possible to print the highest version instead, to avoid the cost, but might be annoying in some cases (like packages where versions are actually triggers, e.g. xxx+system)

Or, more involved, it would be possible to compute the availability specifically for the package without forcing the whole stuff.

[rjbou]

It is indeed possible to compute availability only with subset of targeted packages, but a package availability depends on depexts, we still need to load it. A more optimised solution is to load depexts only for targeted packages also, it's still an exec call, but given the system package manager it can be faster.

It would be possible to print the highest version instead

it is already the case, if pkg is not ① in pinned ② in installed ③ in available, the default is the highest version of all packages.

[vbmithr]

Also opam list --installed --short --safe --color=never is slow since 2.0.7, I noticed it because emacs runs that at startup and it takes seconds before I can use it…

[rjbou]

@vbmithr You can see what takes time with --debug timestamps (disabled with --safe). As it is 2.0.7, it is not a depext loading taking time, please open a new issue.

[kit-ty-kate]

With #4165 (latest master) it takes around 1s instead of 2 for me. However it is still 3x slower than with 2.0.7. Here is the output with --debug:

00:00.006  GSTATE                 LOAD-GLOBAL-STATE @ /home/kit_ty_kate/.opam
00:00.006  RSTATE                 LOAD-REPOSITORY-STATE @ /home/kit_ty_kate/.opam
00:00.050  RSTATE                 Loaded /home/kit_ty_kate/.opam/repo/state.cache in 0.043s
00:00.328  RSTATE                 Cache found
00:00.328  STATE                  LOAD-SWITCH-STATE @ 4.11
00:00.606  STATE                  Inferred invariant: from base packages { ocaml-variants.4.11.0+trunk }, (roots { ocaml-variants.4.11.0+trunk }) => ["ocaml-variants" {= "4.11.0+trunk"}]
00:00.608  STATE                  Switch state loaded in 0.279s
00:00.616  FILTER                 ERR: value_bool: "osx"
00:00.668  SYSTEM                 mkdir /tmp/opam-3392825-dd682e
00:00.721  SYSTEM                 rmdir /tmp/opam-3392825-dd682e
00:00.864  STATE                  depexts loaded in 0.197s
[...]
opam show --debug --raw ocaml-freestanding  0.82s user 0.07s system 99% cpu 0.891 total

[rjbou]

From the log, the repo cache takes 0.3s and the invariant to infer 0.3 more. Setting the invariant with opam switch set-invariant would reduce that time.

[kit-ty-kate]

wouldn't it be interesting to set this invariant automatically when opam detects an upgrade to 2.1 from previous versions? Isn't that related to #4188?

[rjbou]

It is the same mechanism involved, indeed. My previous comment is more a hint, until this is fixed/improved/next release.

[kit-ty-kate]

Just a follow-up on this: there is no change whether or not the switch has been upgraded from 2.0 or created fresh.

[kit-ty-kate]

#4998 fixes partially this issue