-
Notifications
You must be signed in to change notification settings - Fork 232
Merlin Fuzzy CI
The Fuzzy CI is a CI on Merlin that catches regressions by checking the responses of a fixed set of ocamlmerlin
queries. That set of queries is determined by a deterministic but randomly chosen set of file location samples on the Irmin code base. The CI passes automatically if the responses on all samples remain the same before and after the PR. If there's a diff in the responses, the CI fails and makes the diff available as a GH action artifact.
If the diff looks good to the maintainers, they approve it by setting the fuzzy-diff-looks-good
label on the PR, making the CI pass. With that, the CI comments on the PR with two pieces of information: the state of approval (a sha256 hash of the approved diff) and the diff download URL. If the PR is approved via that label and the state of the diff remains unchanged in future PR updates, the CI passes automatically.
The CI runs ocamlmerlin
on over 25000 samples, so reading the entire diff can be tedious. To help, the CI additionally provides the diff of a simplified/"distilled" version of the responses, which breaks each response down into the following categories:
- The kind of response, i.e. one of
exception
,error
,failure
, areturn
containing a (error) message, areturn
containing an empty list, or areturn
containing anything else. - The query number. If it becomes 0, then the
ocamlmerlin-server
has crashed.
About the implementation: The response data is generated in the CI, as opposed to living in the repo, for two reasons: 1. The data is over 20 MB big. 2. The generation of the data takes over 10 minutes, and we don't want to ask our contributors to re-generate it when opening certain PRs.