Test suite improvement

[liyishuai]

As of v1.7.0, the test suite consists of two parts:

1. https://github.com/ocaml-community/cppo/tree/master/test, where *.cppo is expected to produce *.ref.
   • Limitation: Code/feature coverage not yet measured.
2. CI that tests downstream dependants:
   

   cppo/.github/workflows/build.yml

   Lines 176 to 198 in 21f0586

   	- name: Test dependants
   	if: >
   	(matrix.ocaml-version >= '4.05') && (matrix.os != 'windows-latest')
   	run: |
   	PACKAGES=`opam list -s --color=never --installable --depends-on cppo,cppo_ocamlbuild`
   	echo "Dependants:" $PACKAGES
   	for PACKAGE in $PACKAGES
   	do
   	echo $SKIP_BUILD | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
   	echo Skip $PACKAGE && continue
   	OPAMWITHTEST=true
   	echo $SKIP_TEST | tr ' ' '\n' | grep ^$PACKAGE$ > /dev/null &&
   	OPAMWITHTEST=false
   	([ $OPAMWITHTEST == false ] &&
   	echo ::group::Build $PACKAGE) ||
   	echo ::group::Build and test $PACKAGE
   	DEPS_FAILED=false
   	(opam depext $PACKAGE &&
   	opam install --deps-only $PACKAGE) || DEPS_FAILED=true
   	[ $DEPS_FAILED == false ] && opam install $PACKAGE
   	echo ::endgroup::
   	[ $DEPS_FAILED == false ] || echo Dependencies broken
   	done

   • Limitation 1: Only checks compilation and package-specified test; might break the semantics of under-tested packages.
   • Limitation 2: Skipping build/test for packages, for various reasons: Some packages are out of maintenance; others are failing in specific OCaml*OS versions.

Possible directions:

• For users: Document each feature with a test case.
• For devs: Annotate each branch condition with a test case.
• For CI: Refine the exclusion list from (package + OCamlVersion + OSVersion) to (package * OCamlVersion * OSVersion).

[fpottier]

Perhaps we could single out a few source files, in these packages, that use cppo in a heavy way (or in an interesting way, in some sense) and add these source files directly to our test suite. This would ensure that any change in the output of cppo, for these specific files, is detected.

[fpottier]

By the way, I notice that the files *.yml seem to mention several old OCaml versions, up to 4.07, but no newer version. Is this intended? Should we update these files so as to test cppo on 4.08-4.14 and 5.x?

[fpottier]

(I could also add a Makefile entry that runs make test in several local opam switches.)

[liyishuai]

By the way, I notice that the files *.yml seem to mention several old OCaml versions, up to 4.07, but no newer version. Is this intended? Should we update these files so as to test cppo on 4.08-4.14 and 5.x?

Did you mean https://github.com/ocaml-community/cppo/blob/master/.github/workflows/build.yml ? The exclusions there are due to ocaml/setup-ocaml#828.

[fpottier]

No, I meant the files .travis.yml and appveyor.yml at the root of the cppo repository. (I must say that I don't know what these files are used for.)

[liyishuai]

Those files are obsoleted since #79. Should be safe to remove.

[fpottier]

Thanks, I have removed these files. By the way (sorry if this is a bit off topic here), I see that many CI runs currently fail with make: dune: No such file or directory, but I have no idea why that is.

[liyishuai]

Let's track it in #99