-
Notifications
You must be signed in to change notification settings - Fork 264
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Termination of dynamic dispatch #1588
Comments
The issue and the proposed solution seem consistent and appropriate, I like them. However, rather than the (new) syntax:
woudl it be possible to reuse the same usual trick in the trait, to add one more element in the tuple that decreases to indicate explicit method ordering? It is already part of the specification.
|
I think this is the first time that you mention a decreases-overrides check. What does this check do? What's a use-case in which an overriding method will have a different decreases clause as the trait method? I get the impression you're asking the author of trait method to define a decrease clause on it, which I find confusing. How are you supposed to know what the method will need to call without having an implementation for it yet? |
Following a quick discussion today: I wonder how deep call trees across modules tend to be in real programs. If they are not too deep, it might be possible to abandon the modularity aspect and just verify termination at the whole-program level. If we did that, the current system might actually be sufficient? |
I've realized there's a decent workaround to this problem for shared libraries, which definitely doesn't remove the need for a deeper solution as above, but does allow soundly providing reusable traits in many contexts. Here's the idiom we can provide (using my first attempt to add a trait to // Frames.dfy, in a shared library
abstract module FramesTemplate {
trait Validatable {
// ...
}
}
module Frames refines FramesTemplate {}
// MyStuff.dfy in a consuming project
module MyObjects refines FramesTemplate {
// No warning about extending traits across modules
class MyThing extends Validatable {
// ...
}
// ...
} Inside the This is still clearly limited, and in this particular example this would mean putting a lot of classes in a single module. It also doesn't scale well to using many such sets of reusable traits, and I'm sure there are cases of needing to use more than one at one in a module. This is still better than the current situation of immediately giving up on the soundness of termination checking, though. The concrete |
I've realized I'm not 100% confident the idiom above is sound and would like to double-check before encouraging anyone to use it. I don't have a counterexample handy to try, and it might be that the cross-module trait extension check happens after the contents of an abstract module are copied into a refining module but should be before, etc. |
I want to suggest the following: Each function/method is given a matching height function that takes the same parameters and returns a When you call a function/method Here's an example of how height functions resolve termination issues with traits: trait T {
// If you have a trait with a bodyless method, then the height function of that method is also kept bodyless.
function M_Height()
method M() {}
function FirstCaller_Height(): nat {
M_Height() + 1 // call to abstract height function is allowed because it's in the same trait.
}
method FirstCaller() {
assert M_Height() < FirstCaller_Height(); // implicit assertion, passes because X < X + 1
M();
}
}
class Valid extends T {
function M_Height(): nat { 2 }
method M() {
assert Helper_Height() < M_Height(); // implicit assertion, passes because 1 < 2
Helper();
}
method Helper_Height(): nat { 1 }
method Helper() {
}
}
class Invalid1 extends T {
function M_Height(): nat { FirstCaller_Height() + 1 } // error, cycle detected in call graph of height functions
method M() {
assert FirstCaller_Height() < M_Height();
FirstCaller();
}
}
class Invalid2 extends T {
function M_Height(): nat { 1 }
method M() {
assert FirstCaller_Height() < M_Height(); // implicit assertion, fails because !(2 < 1)
FirstCaller();
}
}
function SecondCaller_Height(a: Valid): nat { 3 }
method SecondCaller(a: Valid) {
assert a.M_Height() < SecondCaller_Height(a) // implicit assertion, passes because 2 < 3
a.M();
}
class AlsoValid extends T {
function M_Height(): nat { 4 }
method M() {
var valid = new Valid();
assert Dangerous_Height([this, valid]) < M_Height(); // implicit assertion, passes because (valid.M_Height()+1) < 4
Dangerous([this, valid]);
}
function Dangerous_Height(ts: seq<T>) { ts[1].M_Height() + 1 }
method Dangerous(ts: seq<T>)
requires |ts| > 1 && ts[1] is Valid
{
ts[0], ts[1] := ts[1], ts[0];
assert ts[1].M_Height() < Dangerous_Height(ts); // implicit assertion, passes because x < x + 1
ts[0].M();
}
} Now lets improve the user experience by introducing trait T {
method M() {}
method FirstCaller()
calls M() // Can be inferred because this is a call on `this` which can not be assigned to.
{
M();
}
}
class Valid extends T {
method M()
calls Helper() // Calls clause can be inferred because this is a non-virtual call
{
Helper();
}
method Helper() {
}
}
class Invalid1 extends T {
method M()
calls FirstCaller() // error, cycle detected in calls clauses. Calls clause can be inferred.
{
FirstCaller();
}
}
class Invalid2 extends T {
method M() {
assert FirstCaller_Height() < M_Height(); // implicit assertion, fails because !(2 < 1)
FirstCaller();
}
}
method SecondCaller(a: Valid)
calls a.M() // If we notice 'a' is never assigned, this can be inferred.
{
a.M();
}
class AlsoValid extends T {
method M()
calls Dangerous([this, new Valid()]) // awkward because I thought calls clauses were just expressions, and now it's not. Have to think about the syntax.
{
var valid = new Valid();
Dangerous([this, valid]);
}
method Dangerous(ts: seq<T>)
requires |ts| > 1 && ts[1] is Valid
calls ts[1].M() // can not be inferred
{
ts[0], t[1] := ts[1], ts[0];
ts[0].M();
}
} |
About the "A possible design" proposal in the details. It mentions having to write export ordering in MyTrait if you want the overriding methods module Library {
export
provides MyTrait.M >> MyTrait.P
provides MyTrait.P == MyTrait.Q
trait MyTrait {
method M(n: nat)
decreases #MyTrait.M, n
method P(n: nat)
decreases n
method Q(n: nat)
}
} But how would the designer of Another thing, if I write module Client {
import opened Library
import UnrelatedLibrary
static method Foo(traitInstance: MyTrait)
// The decreases clause is required because of the call to `M`
decreases #Library.MyTrait.M
{
traitInstance.M(3);
AnyOtherLibrary.Bar(); // would this call always fail because Library does not import UnrelatedLibrary?
}
} |
What if every unimplemented function in a trait or in an abstract class was required to have a decreases clause, and we relax the assumption of inter-module call? I'm taking an example @fabiomadge shared where I simulate such decreases clause. I'm not sure how much it would solve the problem module Reals {
trait {:termination false} Function {
function Decreases(): nat
predicate Valid()
function Apply(r: real, ghost previousDecreases: nat): real reads *
requires Valid()
requires Decreases() < previousDecreases
}
}
module Math {
import Reals
function Derivative(f: Reals.Function, x: real, ghost previousDecreases: nat): real reads *
requires previousDecreases >= 1
requires f.Valid()
requires f.Decreases() < previousDecreases - 1
{
f.Apply(x + 1.0, previousDecreases - 1) - f.Apply(x, previousDecreases - 1)
}
}
module Implementation {
import Reals
import Math
class Linear extends Reals.Function {
var m: real
var t: real
constructor (m: real, t: real) {
this.m := m;
this.t := t;
}
predicate Valid() {
true
}
function Decreases(): nat { 0 }
function Apply(x: real, ghost previousDecreases: nat): real reads *
requires Decreases() < previousDecreases
{
m * x + t
}
}
class Good extends Reals.Function {
const f: Reals.Function
predicate Valid() {
f.Valid() && f.Decreases() < Decreases() - 1
}
function Decreases(): nat { 2 }
constructor () ensures Valid() {
f := new Linear(1.0, -1.0);
}
function Apply(x: real, ghost previousDecreases: nat): real reads *
requires Decreases() < previousDecreases
requires Valid()
{
Math.Derivative(this.f, x, previousDecreases - 1)
}
}
class Bad extends Reals.Function {
constructor () ensures Valid() {}
predicate Valid() {
true
}
function Decreases(): nat { 1 }
function Apply(x: real, ghost previousDecreases: nat): real reads *
requires Decreases() < previousDecreases
requires Valid()
{
Math.Derivative(this, x, previousDecreases - 1) // Cannot prove anything here.
}
}
}
method Main() {
var c := new Implementation.Bad();
print c.Apply(42.0, 2), "\n";
} |
The previous proposal I wrote suffers from the problem that the heights functions themselves need decreases clauses. I think the following proposal is simpler and does not suffer from that. Currently Dafny does not allow implementing traits from another module, because that can cause cycles in the call-graph that cross module boundaries, and to prove termination for these would require doing whole-program verification. We can resolve this problem by changing how dynamic calls are tracked in the call-graph so that it no longer has cycles across modules even when modules implement traits from other modules, but in a way that preserves the graph's ability to be used in termination checking. To be able to construct this call-graph while preserving modularity, functions that call dynamically dispatched methods, must record this in their contract, for which we introduce Example of how it works: module Producer {
trait F {
method Foo()
}
method Dynamic(f: F)
calls f.Foo
{
// No termination check since this call is in the calls clause,
// so the termination check is moved to the caller of Dynamic.
f.Foo();
}
}
module Consumer {
import opened Producer
method M1()
{
var c := new F2;
// Termination check: assert Scc_Height(M1) > Scc_Height(c.Foo)
// This check passes because the verifier determines `c.Foo == F2.Foo`, so the assert becomes
// assert Scc_Height(M1) > Scc_Height(F2.Foo), which becomes
// assert 2 > 1
// Note that this check only looks at the Scc_Height component, but not the decreases clause.
// This is because the calls clause of `Dynamic`,
// does not specify with which arguments the dynamic call to c.Foo is done,
// so we can't use the decreases clause of c.Foo
// A more expressive design with more complicated calls clauses is possible
Dynamic(c);
}
class F1 extends F {
method Foo() {
M1();
}
}
class F2 extends F {
method Foo() { }
}
} Call-graph:
Note that when calling a function with calls clauses, an additional termination check has to be done for each calls clause. It might be possible to always infer the More powerful lambdasFor each function type, In the following example, we create a lambda that current Dafny does not except. function HigherOrder(i: nat, f: nat ?-> nat): nat
calls f.calls
decreases f.decreases - 1
{
1 + f(i - 1)
}
function Recursive(i: nat): int
decreases i, 1
{
var f: nat ?-> nat := j => Recursive(j) decreases j;
// Termination check: assert (Scc_Height(Recursive), i) >= (Scc_Height(Recursive), i - 1)
HigherOrder(i, f)
} |
Having a f.decreases is a very interesting idea; however, it was shown that its relation with requires and reads clauses is extremely dangerous in general #2750 (comment) But in general I like the idea to have a type that indicate that the function should have a decreases clause. |
Every function or method always has a decreases clause, even if you don't specify one. In that case, it's the default. |
Here are the two issues, one minor, and one major, with this proposal: module Tr {
trait {:termination false} T {
function A(other: T): int
function B(other: T): int
}
}
module X {
import Tr
class X extends Tr.T {
constructor () { }
function A(other: Tr.T): int {
0
}
function B(other: Tr.T): int {
other.A(this)
}
}
}
module L {
import Tr
function Tie(t1: Tr.T, t2: Tr.T): int { t1.B(t2) }
}
module Y {
import Tr
class Y extends Tr.T {
constructor () { }
function A(other: Tr.T): int {
other.B(this)
}
function B(other: Tr.T): int {
1
}
}
}
module M {
import Tr
import X
import Y
import L
method Main() {
var x: Tr.T := new X.X();
var y: Tr.T := new Y.Y();
print L.Tie(x, y), "\n";
}
} L doesn't import X, so there's no way of knowing that B calls A when verifying L. You'd have to check it on a program level, which wouldn't end up being too dissimilar from doing a whole program analysis. |
Thanks for your reply! I agree, what I described doesn't seem to work for the Tie program. I feel like at a high level there's two options. One is fully static analysis of the call-graph, which is what we do now, and the other a dynamic analysis that requires the verifier to also construct a type of call-graph, that would accept more valid programs. I think extending the static analysis to work across modules can work. If we include as part of a module signature, for each function/method, the trait functions/methods that it calls, then we apply the static call graph analysis that we currently do within a module, also when traits from other modules are used. In the Tie program above, the termination analysis of module
So the SCC graph is:
And then it could say, "Error: detected a call-graph cycle in methods defined outside module For a cycle where at least one method of the cycle is defined in the current module, then the error can be shown there. module Producer {
trait Trait {
method Foo();
}
method CallsFoo(t: Trait) { // Dafny infers that this calls Trait.Foo and exposes that in the contract of Producer
t.Foo();
}
}
module Consumer {
import Producer
class Bar extends Producer {
method Foo() {
// Error: cannot prove termination; try supplying a decreases clause
Producer.CallsFoo();
}
}
} Call-graph:
I think I've left out some things related to decreases clauses. If you're curious I have thoughts on how to tackle that. I was excited about a call-graph analysis that uses the verifier, because it would admit more valid programs, and I was worried that the static approach wouldn't scale to very large applications. I was thinking that as an application grows, each trait will be have more and more classes that extend it, so the SCCs that its abstract methods are apart of will keep growing. However, given that a module only cares about the trait-extending-classes that it has imported, I think the scaling issue might not be so bad. |
Indeed, based on our discussion after my comment, it looks like what I was looking for is to state something like "When determining a group of mutually recursive definitions, any call to a bodiless function is considered part of the clique and thus requires a decreases clause. An example to make sense of what I say. On the program above, we would end up with an error: module Tr {
trait T {
function A(other: T): int
function B(other: T): int
}
}
module X {
import Tr
class X extends Tr.T {
constructor () { }
function A(other: Tr.T): int {
0
}
function B(other: Tr.T): int {
other.A(this) // Error: could not prove that decreases clauses decrease (calling into a bodiless function automatically requires a decreases clause check)
}
}
}
module L {
import Tr
function Tie(t1: Tr.T, t2: Tr.T): int {
t1.B(t2) // Error: Could not prove that decreases clause decreases. B is a bodiless function so it could get back to Tie.
}
}
module Y {
import Tr
class Y extends Tr.T {
constructor () { }
function A(other: Tr.T): int {
other.B(this) // Error: Could not prove that decreases clause decreases. B is a bodiless function so it could get back to Tie.
}
function B(other: Tr.T): int {
1
}
}
}
module M {
import Tr
import X
import Y
import L
method Main() {
var x: Tr.T := new X.X();
var y: Tr.T := new Y.Y();
print L.Tie(x, y), "\n";
}
} |
Dafny
decreases
clauses were designed before the language had dynamic dispatch viatrait
members. As such,decreases
clauses are made to be as simple as possible within each module, anddecreases
clauses are unnecessary across modules (see below for more details). When dynamic dispatch was added to the language, a draconian restriction was put in place to maintain soundness, namely to disallow dynamic dispatch across module boundaries. This is enforced by insisting that aclass
that implements atrait
is declared in the same module that declares thetrait
.The draconian restriction outlaws the useful case where a
trait
is placed in a library. Indeed, we are seeing this indafny-lang/libraries
now. So, Dafny supports a way for a user to lift the draconian restriction and instead take responsibility themselves for termination of dynamically dispatched calls via atrait
--it is to mark the trait with{:termination false}
.This Issue calls out the need to replace the
{:termination false}
workaround with a sound solution.More details of the current design
Here's how it works today. A
decreases E
declaration (whereE
is a list of expressions) applied to a method (or function) in a program is really a termination metricm, s, E
, wherem
denotes the enclosing module ands
denotes the strongly connected component (SCC) of the method inside its module. The import ordering on modules forms a well-founded order, and so does the connectivity ordering between SCCs in a module's call graph.When a method with termination metric
m, s, E
, calls a method with termination metricm', s', E'
, a proof of termination requires showing thatwhere I'm using
>>
to denote the (well-founded) strict greater-than ordering on lexicographic tuples that Dafny uses. In the absence of dynamic dispatch across module boundaries, note the following:m == m'
, so the verification condition can be optimized by dropping the first components of them, s, E
andm', s', E'
tuples.m > m'
, because the caller's module must import the callee's module in order to make the call. In this case, the whole>>
condition above is known (statically) to be true, so the verification condition can be optimized to justtrue
.s == s'
, so thes
/s'
components of the>>
condition above can be optimized away.s > s'
(because of the way SCCs are defined), so the whole verification condition can be optimized totrue
.What I just described in these bullets is how Dafny works today. That is, the
m
ands
components are always optimized away and are never part of the verification conditions that are passed to the verifier. Also, note that, since them
ands
components are always implicit, there is no need for any syntax for a user to write those components--indeed, Dafny currently does not have any such syntax.One more detail: Dafny builds the intra-module call graph for each module and computes the SCCs of each such call graph. Since no global information is needed for this, this technique is compatible with modular verification. In the call graph of a module, there is an edge from a trait method
T.m
to each class methodC.m
that overrides it. That is, it is as dynamic dispatch is achieved through a synthetic implementation ofT.m
that does the following:This synthetic implementation is a true representation of the effect of dynamic dispatch (but it's done more efficiently at run time by the use of v-tables or the like). To argue once-and-for-all about the termination of the calls that emanate from the synthetic implementation, one can change the
m, s, E
ofT.m
tom, s, E, 1
and change all otherm, s, E
triples intom, s, E, 0
.A possible design
To soundly allow dynamic dispatch across module boundaries, we need to do three things:
m
component and ans
components of eachdecreases
clauses, and should then look at the associated components when determining whether or not the optimizations can be applied.m
ands
components are those that are implicitly used today. This will keep the specification clutter low.I'm suggesting that a
decreases
clause can start with an optional component that looks like#X
where#
is concrete syntax andX
is the qualified name of a name of a method/function/whatever. For example,#X
would typically have the form#M.T.F
whereM
names a module,T
names a type within that module, andF
names a member of that type. Since it starts with#
, the optional#X
component is easy to distinguish from the otherE
components of thedecreases
clause. Them
component discussed above will be (explicit or implicit) module part ofX
, and thes
component discussed above will be the SCC ofX
in its declaring module.Example: Consider a trait
I'll use
R(_)
as a function that gives a representative identifier of the SCC containing a given entity. The termination metric forM
is thenLibrary, R(Library.MyTrait.M), n
, the termination metric forP
isLibrary, R(Library.MyTrait.P), n
(note that the first two components are implicit), and the termination metric forQ
isLibrary, R(Library.MyTrait.Q), n
(note that all three components are implicit).Now, consider a class in a different module that implements
MyTrait
:The termination metric of
MyClass.M
is the one given explicitly, namelyLibrary, R(Library.MyTrait.M), n
. This is the same as the method it overrides, so thedecreases
-overrides check passes trivially.The override
MyClass.P
does not give a#
component in itsdecreases
clause. Since it is an override, I suggest that Dafny fill in the#
component in the same way as the method being overridden. This means the the termination metric ofMyClass.P
isLibrary, R(Library.MyTrait.P), n
. Note, it would not be friendly to pickClient, R(Client.MyClass.P), n
as the default termination metric, because that's certain to fail thedecreases
-overrides check.Similarly, the override
MyClass.Q
, which does not have an explicitdecreases
clause at all, defaults to a termination metric ofLibrary, R(Library.MyTrait.Q), n
.Method
S
is not an override, so it defaults to a termination metricClient, R(Client.MyClass.S), n
.As given in the example, all of the
decreases
-overrides check will pass. But consider what happens to calls in the implementations of the methods inMyClass
. IfS
calls into one of the other methods, then things are fine, becauseClient >> Library
. But if one of the other methods, sayP
, tries to callS
, then the verifier will flag it with a termination error. To correct such an error, the recourse would be to change the specification ofS
to saydecreases #Library.MyTrait.P
.If
P
inMyClass
callsQ
, then the verifier will also report a termination error, because it does not know howR(Library.MyTrait.P)
andR(Library.MyTrait.Q)
compare--these refer to SCCs insideLibrary
, which are (because of modular verification) not available inClient
. Instead, the solution here is thatLibrary
must export something that says how these compare. I suggest extending the language to allow export clauses to export ordering information. In particular, I suggest the syntaxto say that
M
lies aboveP
inLibrary
's SCC ordering and thatP
andQ
are in the same SCC. If there's noexport
declaration, then no such orderings are exported--in other words, any such ordering has to be exported explicitly. There needs to be a check for all export sets to see if the orderings they claim are consistent. A good way to do that check seems to be add an edge fromM
toP
for>>
export, and to add an edge fromP
toQ
and an edge fromQ
toP
for the==
export, and then to check thatM
is indeed aboveP
in the final ordering. (No additional check is needed for the==
export, since adding the two edges ensures thatP
andQ
will be in the same SCC.)(One will also need to express that certain modules lie above other modules in the import ordering. That can probably be done with this same syntax.)
An explicit
decreases
clause can give just the#
component. That is, if a program says justdecreases #M.X
, then theE
components are filled in as usual in Dafny. For example, four ways forMyClass.Q
above to have the same termination metric would be todecreases
clause altogether (as was done above)decreases #Library.MyTrait.Q
and let theE
components default ton
decreases n
and let the#
component default to#Library.MyTrait.Q
decreases #Library.MyTrait.Q, n
One thing I have not detailed--in fact, I'm not sure of its answer--is how to construct the call graph inside a module and compute its SCCs. I'm guessing this should be done in the same way as always, without regard to any (implicit or explicit)
decreases
clauses.Other designs
I know of only one other design that solves this problem and achieves modular verification. It is by Bart Jacobs and colleagues, and it makes use of a multiset ordering of method names. (I think it is implemented in VeriFast.) I haven't worked out a detailed comparison, but I'm guessing Jacobs's design is more flexible than what I have sketched above. Nevertheless, I'm hoping the design sketched above might be good enough, might be lighter in required syntax, and might fit into what Dafny programmers are used to today. If not, we may need to adopt Jacobs's design more directly.
The text was updated successfully, but these errors were encountered: