Fixes to reversal of environment updates

[dra27]

Fixes for #5838, #5925 and #5926. This PR has become quite an epic rabbithole, but the fixes and changes are very, very intertwined. It is possible to separate the final fix for #5926 to a subsequent PR, but the final fix itself is actually quite small.

The first commit adds a new sort operation to the reftest language. While working on this, the stability of the output of opam env and the stability of the output when using unordered made making the true effect of each commit tricky. I'm still not sure that unordered is behaving as well as it should when there's actually a change to the output (i.e. when one line changes, it's allowing some unchanged lines to be re-ordered noisily in the diff).

I then add a whole range of tests:

• Since opening the original issue for MANPATH, I noticed that CAML_LD_LIBRARY_PATH wasn't reverting correctly on Windows either, and it turns out that's because opam wasn't correctly handling updates with forward slashes (i.e. CAML_LD_LIBRARY_PATH += "%{lib}%/stublibs expands to %OPAMROOT%\switch\lib/stublibs but opam env normalises the /stublibs to \stublibs. When reverting, this normalisation is missing.
• A test for Environment revert not working correctly for Unix-like variables on Windows #5838 ensures MANPATH is actually reverted
• A test for Fix reverting additions to PATH-like variables #4861 ensures an update containing multiple directories and is then extended to include a quoted directory
• A test for Using the += opetator in build-env to prepend an empty (but set) variable adds a trailing seperator #5925 tests +=, =+, := and =: applied to a variable set to "" in the same setenv block and shows += and =+ misbehaving.
• A comprehensive test (!!) for Unexpected behaviour when prepending/appending to initially empty variables multiple times #5926 tests those four operators on:
  • a: and :a, both set in the environment, and synthesised by using := or =: on an unset variable
  • a1::a2 (a "middle" blank) entry
  • A variable set to a single :

There have been a lot of issues and PRs in the past to do with the treatment of := and =: principally where MANPATH is concerned. I'm relatively confident that the approach for this before has not been strictly correctly (and is the cause of the issue being seen in #5926). For both PATH (as defined by Posix) and MANPATH (as defined by the tool itself), an unset value, an empty value and the value : are all equivalent. For PATH, that means the current directory and all of these entries are literally equivalent to PATH=.. For MANPATH, those three entries mean "use the default search path".

Now, as far as I can see the previous discussions have been about ensuring that MANPATH has a leading or trailing :, to ensure that opam never hides the default path. Thus, if MANPATH is not set, then we have MANPATH =: "%{man}%" giving us :/home/dra/.opam/default/man. However, the focus should not be on having a leading or trailing :, the focus should be on ensuring that a "blank" entry is created if needed (this is the rule for FOO := "a" and FOO =: "a" are a: or :a respectively if FOO is unset or empty) and but then for all updates separately ensuring that the blank entry is preserved.

i.e. if FOO is :a then both FOO += "b" and FOO := "b" should set FOO to b::a

The actual fixes then proceed as:

• The CAML_LD_LIBRARY_PATH issue arises from what appears to be an incorrect comment in OpamEnv.apply_op_zip indicating that the argument is already transformed, which may suggest why OpamEnv.reverse_env_update was missing the transformation. Essentially, the transformation ensures that opam env --revert reverts the same value as it applied in opam env
• In OpamEnv.split_var, the cases for splitting using the quoted rules were the wrong way around. Fixing that regresses Fix reverting additions to PATH-like variables #4861, though
• If an environment variable is empty or unset, opam internally synthesises that as []. However, if setenv or build-env has FOO = "" then we end up with [""] as opam must ensure that this variable is overwritten. The easiest solution for Using the += opetator in build-env to prepend an empty (but set) variable adds a trailing seperator #5925 is simply to allow for that - [""] can only arise from FOO = "" as an update, so += and =+ handle that case, and remove the "" entry from the list. := and =: at this point of course work correctly, because the "" should be there.
• The quoted splitting algorithm in OpamEnv.split_var is not quite right - it doesn't cope correctly with a separator quoted inside the double-quote marks. My initial approach therefore was to replace it with the PATH-splitter in OpamStd.Env.split_path. This, however, reveals a more fundamental problem - the PATH-splitter correctly splits the PATH, but it removes all the quote marks. The user is at liberty to add quotes in a PATH even if they're not needed. For example, it's a common mistake on Windows to quote directories with spaces in them (having, for example, Path set to C:\Windows\system32;"C:\Program Files\Something\bin"). With the corrected split function, PATH += "foo" causes Path to become foo;C:\Windows\system32;C:\Program Files\Something\bin with the superfluous quotes removed. This isn't correct, given that the effect of PATH += "foo" should morally be much closer to export PATH="foo;$PATH".
• Similarly to this dequoting, with x-env-path-rewrite, if two packages disagree over the separator, then we have the slightly odd situation where the variable is split according to the first package's separator and re-joined according to the second package's. i.e. if Path is foo;bar and the first package to be evaluated has PATH += "baz" then Path internally is ["baz"; "foo"; "bar"] and will be exported as baz;foo;bar. However, if another package is then added which incorrectly has x-env-path-rewrite: PATH ":" "target" and executes PATH += "broken" then internally we have ["broken"; "baz"; "foo"; "bar"] but the variable will be reconstituted as broken:baz:foo:bar, so from an initial value of foo;bar, opam env changed the existing separator.
• Combined with the previous issue with quoting, I've changed the internal representation. Instead of storing just the directory, we instead store the directory, the raw unquoted version of the directory and the separator. This has the benefit that concatentation simply uses the given separators and raw values. The cygpath translations, if used, happen in the raw version, with the native directories manipulated internally. This change of representation naturally fixes several of the issues in Unexpected behaviour when prepending/appending to initially empty variables multiple times #5926 and restores the superfluous quotes in environment variables.
• The final part of the fix for Environment revert not working correctly for Unix-like variables on Windows #5838 is to correct the rule used in Opam.unzip_to to split the value in FOO += value. I've commented (hopefully clearly) the three cases in the commit. At this point, the Fix reverting additions to PATH-like variables #4861 test passes again and Environment revert not working correctly for Unix-like variables on Windows #5838 is fixed.
• The next commit then handled Fixes: Env var & delimiters, unversionned opam file pin, doc #3404 in a slightly different way, creating :: as required (see the changes in the reftests in that commit)
• Finally, there's a long-standing logic error OpamEnv.reverse_env_update for =+ and =: where one list was reversed which shouldn't be. Again, the effect is visible in the reftests! The point is that a reversed environment variable was unzipped, and needs to reversed, but there was one reversal too many, given how environment variable updates are stored.

TODO

• Update master_changes.md in full
• Update documentation (once semantics presented here fully agreed)

[dra27]

OK, the fix I was proposing I didn't like because it didn't preserve unnecessarily escaped parts of existing variables. I then further found it regressed #4861 and consequently broke the msvs-detect package.

I have a plan (which I'm currently implementing) - a failing test for the main issue (#5838) added and for the first bit of the fix, which I am certain is right. The change I think is required for #5838 without breaking #4861 I think will also address #5925 and #5926, so I've added tests showing the failure behaviour for them as well.

[dra27]

The change in c4167f6 requires environment files to be reset - @rjbou... does that happen naturally with a format upgrade?

That said, it's making me wonder if we should ever be "putting" rewrite_default in the environment cache files anyway?

[kit-ty-kate]

I feel like we'd want this all the time, like globally. Couldn't we add this in run.ml instead?

[dra27]

I debated this, too - but the problem then is that it's really not visible from the test that we've done that, and I anticipated that being confusing in future... especially as it would then be really unclear if it ever got changed or broken how and when it might then fail.

[rjbou]

Thanks!

[rjbou]

On the next change/addition, we should change that into a record.

[kit-ty-kate]

Good to go once rebased

[kit-ty-kate]

Thanks!