Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Even better shrinker! #47

Open
j-hui opened this issue Jul 20, 2021 · 2 comments
Open

Even better shrinker! #47

j-hui opened this issue Jul 20, 2021 · 2 comments
Labels
good first issue Good for newcomers not-urgent Use this labels for issues that are not urgent.

Comments

@j-hui
Copy link
Collaborator

j-hui commented Jul 20, 2021

The shrinker is already really great, and it's producing some excellent counterexamples to populate our regression test suite (thanks @Rewbert !). But it could be even better:

  • There are a lot of skips in the generated code that can should be easily pruned out.
  • For a lot of the test cases, the values don't actually matter---can we try to replace more complicated expressions with constants where possible? And can we try to replace constants with smaller constants (so it's not just a syntactically smaller test case)?
  • Can we try to simplify the types? For instance, if a test case fails with an Int32, does it still fail with a UInt32? And what about a Bool?

I guess the overall theme of this PR is about being more flexible with what constitutes a "smaller" test case---I believe implementing the above may lead to even more intuitive test cases (right now, the test cases are small enough to understand but still require some manual doctoring to be self-explanatory).

@j-hui j-hui added good first issue Good for newcomers not-urgent Use this labels for issues that are not urgent. labels Jul 20, 2021
@Rewbert
Copy link
Collaborator

Rewbert commented Jul 30, 2021

Changing the types of things is very difficult, as the program might not (most likely not) be type correct anymore. And furthermore, if the shrinker can change an Int32 to an Int64, it can probably also do the opposite, and thus the shrinker will never terminate. When we are shrinking a program we are not maintaining any state, so there's no way to know that we've not already tried to give a variable a certain type.

Concerning the constants, the shrinker is already trying to do a syntactic reduction of expressions. If only a single constant remain after this step it could probably be shrunk, yes, but I don't personally think it's worth the effort. We're just trading one constant for another. I did prepare the expression shrinker to do this though, so you'd just need to write a function SSMExp -> [SSMExp] where the resulting list contains mutations of the expression under scrutiny. Then that function can be plugged into the rest of the machinery. Empty list = unsuccessful shrinking.

The skips can be pruned out, but keeping them won't change any semantics or anything. It should be easy to add this as a last shrinking step though, if we think it's worth the time.

@j-hui
Copy link
Collaborator Author

j-hui commented Jul 30, 2021

Ah gotcha. In terms of type errors, @yc2454 's type checker should come in handy, but I do understand your concern.

I think when I have some spare cycles, I'll write these shrinkers myself to get my hands dirty with the shrinker, and see if it's actually useful. For now, I'll punt on this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers not-urgent Use this labels for issues that are not urgent.
Projects
None yet
Development

No branches or pull requests

2 participants