Initial end-to-end test suite

[craigfe]

This initialises an end-to-end test suite that compares example executions against expected outputs. The process is driven by auto-generated dune files, allowing a workflow of dune runtest → dune promote.

Currently this does not allow for passing options to the executable under test, which would be a nice addition. For this, we could use extra *.opt files similarly to how it's being done in OCamlformat.

Frustrations

1. The dune action DSL has no way of negating the exit code of a command, so it's being done manually with a little script -- I might request this as a Dune feature...

2. It's necessary to sanitize the output of the tests to remove environment dependencies / randomness. Again, this is being done by an extra script.

3. I initially added symlinks to the examples/ directory to verify their output, but these appear to break the Windows CI, so I removed them.

[NathanReb]

I think this is a great improvement!

I don't think the amount of boilerplate we need for this will feel like too much once we've added more tests and as we've discussed offline, I even think parts of this could be extracted into a cram-test-like framework or upstreamed to an existing one!

I added a few minor comments but generally speaking it's 👍, thanks for your time on this!

[samoht]

I don't think the amount of boilerplate we need for this will feel like too much once we've added more tests and as we've discussed offline, I even think parts of this could be extracted into a cram-test-like framework or upstreamed to an existing one!

out of curiosity, did you consider using mdx for this?

[samoht]

Looking good! Feel free to merge it whenever you think that's ready. That's a huge improvement over the current (empty) state.

[NathanReb]

out of curiosity, did you consider using mdx for this?

I hadn't but it's true we could make it work. I'm just not sure if that's a very good idea as it's a bit outside of mdx's scope and I think it's better for mdx to focus on ensuring manuals or articles are correct than trying to also turn it into a testing framework.

I agree it wouldn't take much but I still think some features wouldn't quite fit. For instance, the random/non-deterministic output bits replacement logic we have here is quite different from what we want for mdx right now. Here we just want to replace it with a placeholder because we don't care about what it actually is as long as it matches what we expect it to be. In mdx we want a significant value in the output instead because that's clearer for the reader. I'm not quite sure how we can achieve both with a reasonable API and workflow. We could probably hack something together and make it work somehow but I'm afraid that by trying to be two different things at once the code base would suffer and we might regret it in the long run.

What I had in mind was something simpler than mdx, focused on end to end testing for CLI binaries similar to what dune uses for its blackbox tests. I was actually hoping we could extract that into a tool which we could use in both dune and alcotest.

We should probably also look into existing tools as there's probably one around!

[NathanReb]

Looks good!

Thank you very much for that, makes me want to contribute to alcotest more!

I pointed a couple unnecessary %dep in the dune rules but feel free to merge once they're fixed!

[samoht]

This is clearly off-topic, but mdx testing logic was extracted from dune blackbox tests :-) And merlin is using the alternate "cram" syntax of mdx to for its own blackbox testing so this is definitely what mdx is designed for.