-
Notifications
You must be signed in to change notification settings - Fork 261
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add --test-assumptions option #3185
Add --test-assumptions option #3185
Conversation
I think this looks nice, though I think there will likely be a use for an argument for other modes. I agree that the current |
One thing I'm excited about is a potential mode to even test Given that, I worry that "contracts" is too specific a noun for this, since for me it suggests only things like pre- and post-conditions. Since we talk about how everything Dafny verifies is ultimately some kind of assertion, perhaps
|
Yeah, I think maybe |
I'm confused about the suggestion to call it Do you call a postcondition on an externally implemented method an assertion? I would call that an assumption.
What are the use-cases there? Isn't using externs exposing you to a similar risk as when you're using assume statements? Why would you want to differentiate between those two risks?
You mean even the ones that were statically verified? Interesting. Many of them won't be compilable though. I would put this under an option |
Postconditions, in general, are both assertions and assumptions, depending on what you're doing with them. Because external code isn't being verified, you don't use its postcondition as an assertion during verification. But that's exactly why you'd want to check it at runtime. So, yes, I'd call an extern postcondition an assertion in this contect.
The key use case I imagine is one where internal assumptions aren't effectively testable dynamically, because they heavily use non-executable specifications, but contracts on external code are effectively testable. Although both sorts of assumptions are, indeed, assumptions, I think that in practice they tend to have fairly different characteristics.
The use case I imagine here is that you'd disable verification entirely and then check all compilable assertions dynamically, warning about any that aren't. The goal is to provide a more gradual on-ramp to verification where you can start by simply stating what you think should be true, but don't have to put in any extra effort to verify it. And then, as development proceeds and you become more confident that your specifications are actually provable, you can switch to a mode where you prove everything except what's listed explicitly as an assumption (e.g., anything that's still too hard to prove). |
Sounds good
assert statements have conditions that are verified, while assume statements have conditions that are not verified, but afterwards are taken for granted by the verifier. If you call an external method with a postcondition, then after the call the ensures clause is taken for granted by the verifier, even though it was never verified, so why would you not call the ensure clause an assumption in this case?
Why would the characteristics be different? It could also be the other way around, that you have a compilable assume statement and an uncompilable ensures clause for an {:extern} method. Is there a reason why you want to dynamically test your extern clauses more than your assume statements? Is one less trustworthy than the other? We could have a |
It's both. It's an assumption from the point of the caller, but it's an assertion from the point of the callee (the external code). Since it's not going to be verified and that external code isn't going to test it at runtime, though, it's nice to have the caller test instead.
The key difference is that contracts between internal code are often complex, because they can be (since there's a way to check them statically), and that's more likely to lead to introducing things like quantifiers and infinite mathematical objects. Due to the more limited support for checking, I tend to (and think others tend to) make contracts on externs as simple as possible, and design the code so that those contracts can be simple by avoiding as much coupling as possible. |
I agree that from an implementation perspective that is true, since Dafny uses modular verification and when you do a call it won't verify the callee again just for that call. However I don't think that's the case from a usage perspective. I would say that asserting a condition means asking Dafny to verify that it holds, while assuming a condition means asking Dafny to merely assume that it holds. The ensures clauses of the callee are verified to hold after the call, so in that sense they are asserted, not assumed, from the caller's perspective.
Makes sense, but these are only statistical differences. I think it would be better to allow customers to decide for which assume and How about we introduce
When using |
I think there's more design work we'll need to do to choose the precise set of arguments to include. For the purposes of this PR, I think the key conclusion is that we'll support more than one mode, so an argument is necessary. Since |
Well, I think we could go with Note that this option is marked as experimental, and we'll be free to change it after we add it, so I've made the change you suggested. |
@@ -1,5 +1,6 @@ | |||
using System.Collections.Generic; | |||
using System.CommandLine; | |||
using System.Diagnostics.Tracing; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the code using this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope ^,^ would prefer not to have another review/build cycle again for this though. Maybe we can find a better way of catching this in the future, maybe through dotnet-format
.
@@ -38,6 +38,7 @@ public interface ICommandSpec { | |||
BoogieOptionBag.NoVerify, | |||
CommonOptionBag.EnforceDeterminism, | |||
CommonOptionBag.OptimizeErasableDatatypeWrapper, | |||
CommonOptionBag.TestAssumptions |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still slightly new the the modern options architecture we have here: will this option be available for dafny translate
, too? I suspect it'll be used most often there, though also during dafny run
and dafny test
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Execution options are available for run,build,translate and test
Changes
Add the
--test-assumptions
option to all execution commands: run,build,translate,testNote that I've written the description of the option to be aspirational, so that it indicates what we want this option to do, with the note that it's experimental and after that an explanation of what it currently does. I've also simplified the option so it's just an off/on boolean, instead of allowing using to choose under what conditions they want to add tests, since I don't yet see a use-case for the previous
:TestExterns
option. Feel free to discuss!Testing
Add the new option in a littish test
By submitting this pull request, I confirm that my contribution is made under the terms of the MIT license.