[TODO] nim-dustmite source code minimization tool for automated bug reduction

[timotheecour]

As I was trying to reduce a compiler error that only triggers in my full project and is hard/tedious to minimize manually, it occurred to me we could try to develop a source code minimization tool to automate that.

in D, that tool exists, see https://github.com/CyberShadow/DustMite and docs https://github.com/CyberShadow/DustMite/wiki ; developped by @CyberShadow
It's used extensively by the community for investigating bugs; it's especially invaluable for those that trigger in your whole project but are hard to isolate to a minimal bug (which is the 1st step to solving it). I've used it myself a lot for great benefit.

It turns out dustmite is not super tied to D language (doesn't need to know about D's semantics) ; in fact it can reduce C++ code too as it only operates at much shallower level.

• First step could be adapting/forking dustmite to work with Nim code

@CyberShadow any help / pointers in this thread would be most welcome!

• second step (optional, not needed for that purpose...) could be porting that forked dustmite to nim

links

• http://delta.tigris.org/
• https://github.com/CyberShadow/DustMite
• also works great in combo with nim-digger nim-digger: program to build nim at any revision (including forks/PRs) (cf D's digger); useful for git bisect timotheecour/Nim#332

[Araq]

IIRC @zielmicha had a tool that does this.

[Araq]

I'm not too happy these tools as they tend to produce "too minimal" test cases that then get fixed but in the real, larger example the problem remains. Minimal test cases cause an "overfit".

[CyberShadow]

@CyberShadow any help / pointers in this thread would be most welcome!

DustMite has, more or less, pluggable splitters:

https://github.com/CyberShadow/DustMite/blob/7403f0d053b96b2c21950df038923b5d335d557f/splitter.d#L75-L82

There is already support for reducing files in a second "syntax" (patch files). D mode should, to some extent, work with any curly-brace languages already though.

I'm not too happy these tools as they tend to produce "too minimal" test cases that then get fixed but in the real, larger example the problem remains. Minimal test cases cause an "overfit".

In our experience this happens very rarely (less than 1% of test cases). In any case, this mistake can be done by humans as well, who are not aware of the exact internal mechanics that cause the bug.

What happens more commonly is that, along the way of reducing the intended bug, DustMite generates a test case that triggers a different bug with similar symptoms (e.g. a compiler segfault), and then gets sidetracked into reducing that bug. In such cases, a more strict test script (e.g. which checks the segfault stack trace) is the solution.

https://github.com/CyberShadow/DustMite/wiki/Detecting-a-specific-segfault

[krux02]

What is a "compiler ICE"?

[krux02]

I added the Won't fix label, because that is how I understand the status quo.

[Araq]

@timotheecour Feel free to submit one, it's not a bad idea, but we don't intent to work on one.

[timotheecour]

What is a "compiler ICE"?

internal compiler error; looks like nim doesn't use this terminology used in some other languages; I rephrasd it as just 'compiler error'

• changed from wontfix to postponed

[timotheecour]

see recent project: disruptek/dust: DUST is Unattended Syntax Truncation

[CyberShadow]

I am obviously extremely biased, but at this point I would recommend implementing similar tools as an add-on to DustMite. I spent some time earlier this year working on some algorithmic improvements to DustMite which made it significantly faster than earlier versions.

If implementing a Nim parser/lexer in D is undesirable and a Nim parser/lexer is already available as a Nim library that could be used for this purpose, then what we can do is make the Nim program produce an intermediary format with a pre-split reducible tree that DustMite can consume, and invoke DustMite with it. I'll be happy to help with the D/DustMite side if there's interest.

CC @disruptek

[disruptek]

Sounds great; gimme a spec and I'll write the Nim side. Thank you!

[CyberShadow]

Alright! How does this look: https://github.com/CyberShadow/DustMite/wiki/JSON-archives

The implementation is in the next branch:
https://github.com/CyberShadow/DustMite/tree/next

[disruptek]

Sorry I missed this. Looks pretty easy; I will reach out in the repo if I have any trouble. Thanks again! Looking forward to seeing this thing work. 🙂

[timotheecour]

a more recent blog about dustmite: https://dlang.org/blog/2020/04/13/dustmite-the-general-purpose-data-reduction-tool/ + related discussion: https://forum.dlang.org/post/pruiwhotmlfselnplztf@forum.dlang.org