Conformance testing for linter?

[overlookmotel]

We've noticed while making changes to AstKind (oxc-project/oxc#11490) that it's often possible to introduce bugs in the linter without any tests failing.

The tests for rules are often taken directly from the ESLint/Unicorn/etc rule which they aim to replicate. They often:

1. Are quite a minimal set of test cases.
2. Only cover edge cases which are potentially problematic in the rule's JS implementation. But, given the large differences between Rust and JS, and ESLint and Oxc, these don't necessarily correspond to likely pitfalls in our Rust implementations of these same rules.

I assume that for the vast majority of rules, the Oxc implementation of the rule aims to replicate the behavior of the original ESLint rule exactly (there are some exceptions, where we diverge from ESLint slightly, where we think we can do "better").

Could we add some kind of conformance testing for linter, along the lines of what we have for parser, transformer etc?

• Take a set of test fixtures (either e.g. Test262, or a selection of real-world code taken from Github).
• Run ESLint on these test fixtures, with all rules that Oxlint provides enabled.
• Conformance tester run Oxlint on these same fixtures, with all rules enabled.
• Conformance check that Oxlint produces same errors as ESLint.

The ESLint results could be generated and stored in a separate repo, which is included in Oxc as a submodule (similar to https://github.com/oxc-project/acorn-test262). That way we don't have to run ESLint every time we run conformance.

I imagine comparing the outputs might be a little tricky. Oxlint likely produces different (better?) error messages in some cases, and may use a different span for some errors from ESLint. If we compare Oxlint's and ESLint's output directly, there'd likely be a lot of erroneous mismatches. But we could probably at least compare the number of errors flagged by Oxlint and ESLint for each rule. Those error counts should match if Oxlint's implementation of the rule replicates ESLint's behavior.

[overlookmotel]

Or... is this unnecessary because monitor-oxc already performs this role?