Test reftests and add some readme files

master_changes.md

@@ -184,6 +184,9 @@ users)
   * Add a test filter on N0REP0 first line [#6105 @Keryan-dev]
   * Add a makefile target `quick-test` to launch only `N0REP0` tests [#6105 @Keryan-dev]
   * Speedup `make reftest-gen` [#6155 @kit-ty-kate]
+  * Fix some json output automatic replacement (duration and path on Windows) [#6184 @rjbou]
+  * Add test for reftest syntax [#6184 @rjbou]
+  * Add some readme file [#6184 @rjbou]
 
 ## Github Actions
   * Depexts: replace centos docker with almalinux to fake a centos [#6079 @rjbou]

tests/readme.md

@@ -0,0 +1,17 @@
+# Opam test suite
+
+There is 3 different kind of tests
+* CRAM test in `reftests`. It is used to check opam behaviour. See
+  `reftests/readme.md` for more details.
+* Bench tests in `bench`, using
+  [current-bench](https://github.com/ocurrent/current-bench). It is used to
+  benchmark some specific operations.
+* OCaml tests in `lib`. For the moment it is used to check some type
+  manipulation and patch code.
+
+There is also other tests:
+* [`opam-rt`](https://github.com/ocaml-opam/opam-rt): historical test suite. it
+  is a mix of opam cli calls and library usage. It is meant to be removed once
+  all test has been ported to `reftests` and `lib`
+* [`opam-crowbar`](https://github.com/ocaml/opam/blob/master/src/crowbar/opam-crowbar.opam):
+  some fuzz testing of some opam library main types.

tests/reftests/dune.inc

@@ -1448,6 +1448,27 @@
    %{targets}
    (run ./run.exe %{exe:../../src/client/opamMain.exe.exe} %{dep:rec-pin.test} %{read-lines:testing-env}))))
 
+(rule
+ (alias reftest-reftests)
+ (enabled_if (and  (or (<> %{env:TESTALL=1} 0) (= %{env:TESTN0REP0=0} 1))))
+ (action
+  (diff reftests.test reftests.out)))
+
+(alias
+ (name reftest)
+ (enabled_if (and  (or (<> %{env:TESTALL=1} 0) (= %{env:TESTN0REP0=0} 1))))
+ (deps (alias reftest-reftests)))
+
+(rule
+ (targets reftests.out)
+ (deps root-N0REP0)
+ (enabled_if (and  (or (<> %{env:TESTALL=1} 0) (= %{env:TESTN0REP0=0} 1))))
+ (package opam)
+ (action
+  (with-stdout-to
+   %{targets}
+   (run ./run.exe %{exe:../../src/client/opamMain.exe.exe} %{dep:reftests.test} %{read-lines:testing-env}))))
+
 (rule
  (alias reftest-reinstall)
  (enabled_if (and  (or (<> %{env:TESTALL=1} 0) (= %{env:TESTN0REP0=0} 1))))

tests/reftests/readme.md

@@ -0,0 +1,72 @@
+# Simple CRAM-like test framework for opam tests.
+
+## Features and format
+
+### General view
+
+A reftest looks like
+```
+REPO_HASH
+### opam command
+output line 1
+output...
+### <filename>
+contents...
+### opam command
+output...
+### ENV_VAR=x opam command
+output...
+```
+
+- its first line is
+  * the git hash of the opam repository to use, an opamroot is already
+    initialised with that repo as "default"
+  * or N0REP0 if there are no dependencies on opam repository. In this case, an opamroot is
+    already initialised with an empty `default` repository in `./REPO`
+    directory, that you need to populate
+- 'opam' is automatically redirected to the correct binary
+- the command prefix is `### `
+- comments have the following syntax: `### : comment`
+
+### File creation
+
+- use `### <FILENAME>` followed by the content of the file, to create a file verbatim
+- use `### <pkg:NAME.VERSION>` followed by the content of an opam file, to
+  add this package to `default` repository in `./REPO`. This will also implicitly run `opam update default`
+- use `### <pkg:NAME.VERSION:FILENAME>` followed by the content of the file, to add this
+  file as a extra-file of the given package in the `default` repository, and
+  implicitely run `opam update default`
+- use `### <pin:path>` followed by the content of an opam file, to have some 
+  fields automatically filled to be able to pin it without lint errors
+
+### Commands to run
+
+- `### FOO=x BAR=y` to export variables for subsequent commands
+- shell-like command handling:
+  * **NO pattern expansion, shell pipes, sequences or redirections**
+  * **Each `$VARIABLE` expansion is one argument unlike POSIX shell. So `rm $VARIABLE` will always remove one file, not several**
+  * `FOO=x BAR=y command`
+  * Arguments can be quoted: eg `"foo\"bar"`, `'foo\bar'`, but not combined
+    (`foo'bar'` is not translated to `foobar`)
+  * Variable expansion in arguments (`$FOO` or `${FOO}`). Undefined variables
+    are left as-is
+  * There is some rewrite functions available:
+    * `| 'REGEXP' -> 'STR'` (can be repeated; set `STR` to `\c` to
+      clear the line)
+    * `| grep REGEXP`
+    * `| grep -v REGEXP`
+    * `| unordered` compares lines without considering their ordering
+    * `| sort` sorts output
+    * `| sed-cmd command` replaces full path resolved command by `command`
+  * variables from command outputs: `cmd args >$ VAR`
+- additional commands
+  * `opam-cat file`: prints a normalised opam file
+  * `json-cat file`: print a human readable opam output json file, with
+    replacement of some duration and temporary files names. meant to be used
+    on opam generated json files.
+
+- if you need more shell power, create a script using <FILENAME> then run it.
+  Or just use `sh -c`... but beware for compatibility.
+
+The opam roots are generated using dynamically generated dune rules (see
+gen.ml and dune.inc), then the tests are run using this script.