Improve linter testing

[bdovaz]

Conversation comes from #2244

Hypothesis is a phenomenal property-based testing library for Python, but we're a ways off from that today.

Originally posted by @Kurt-von-Laven in #2244 (comment)

Hypothesis is a phenomenal property-based testing library for Python, but we're a ways off from that today.

But we are testing that our wrapping code and calling the linter works, not testing the linter itself. As everyone, learning to better QA software is always a good thing. It's always a weak spot.

Originally posted by @echoix in #2244 (comment)

Hypothesis can help with that too, but it's rarely the case that one would want to write all tests exclusively in Hypothesis. Property-based testing is intended to be used in conjunction with conventional example-based testing.

Originally posted by @Kurt-von-Laven in #2244 (comment)

Right now classes are automatically generated through:

https://github.com/oxsecurity/megalinter/blob/main/.automation/build.py#L644

The generated classes are here:

https://github.com/oxsecurity/megalinter/tree/main/megalinter/tests/test_megalinter/linters

They all inherit from the next class so all linters have exactly the same tests:

https://github.com/oxsecurity/megalinter/blob/main/megalinter/tests/test_megalinter/LinterTestRoot.py#L11

This class uses this utility class:

https://github.com/oxsecurity/megalinter/blob/main/megalinter/tests/test_megalinter/helpers/utilstest.py

And the input for the tests is here:

https://github.com/oxsecurity/megalinter/tree/main/.automation/test

The problem I see as I said, among other things, is that there are no specific tests for certain linters that can apply fixes or format files. Right now there is no testing of any of this.

This class does the trick but we would have to think of a system applicable to all linters:

https://github.com/oxsecurity/megalinter/blob/main/megalinter/tests/test_megalinter/mega_linter_2_fixes_test.py

[bdovaz]

@nvuillam @Kurt-von-Laven @nvuillam do any of you see yourself able to develop an architecture for this case?

With "architecture" I mean something similar to what is done and that I have mentioned in the first message. Generate in the test class of each linter some tests to be able to verify that if a formatting or a fix is applied (if the linters support format/autofix) it matches the expected file.

Create some kind of convention like good/bad folders or _good/_bad suffixes so that you can compare unformatted/formatted and uncorrected/fixed files.

I can lend a hand in preparing the input for the linter test files but I have little experience with python to do the development needed to get to this point.

[nvuillam]

I'm capable... but it requires time that I won't have for a while :/
About the architecture:

• No need to test before/after exact match , just checking that the file has been updated is ok (else we'd have to maintain all before/after sample files for all linters :D )
• I was a newbie in python when I started to code MegaLinter... never even did a hello world before, so don't underestimate yourself, if you understood my architecture, you can understand python :D
• All test classes are automatically generated and inherits from https://github.com/oxsecurity/megalinter/blob/main/megalinter/tests/test_megalinter/LinterTestRoot.py: add a method here and it will be automatically added everywhere ^^
• I think formatters/autofix test could be the following: if in the test folder, there is a file or a folder containing fix , let's
  • read its content and store it in a variable
  • call the linter on it with APPLY_FIXES=true
  • read again the file content and compare: if different, test is successful ^^
  • discard changes on the file ( seems doable like that with gitpython https://stackoverflow.com/a/8249103/7113625 )

All can be tested locally on windows ^^