non-exhaustive tests

[Average-user]

As explained here #80 there is a way to write tests that is not exhaustive enough. As an example, if we have a simple program like:

add(X,Y,Z) :-
 Z >= X+Y.

A test like:

?- add(1,2,3).
true.

succeeds. When is obviously not correct. nevertheless if we do this:

?- add(1,2,X).
ERROR: Arguments are not sufficiently instantiated
ERROR: In:
ERROR:    [9] _2658>=1+2
ERROR:    [7] <user>
ERROR: 
ERROR: Note: some frames are missing due to last-call optimization.
ERROR: Re-run your program in debug mode (:- debug.) to get more detail.

throws an error. In this case, this predicate should be tested like so add(1,2,X), x =:= 3.

This is a particular characteristic of Prolog, and that is why is particularly confusing.

This are all tests that have this flaw:

• anagram
• binary
• complex-numbers
• grains
• hamming
• nucleotide-count
• pascals-triangle
• rna-transcription
• spiral-matrix
• sum-of-multiples
• triangle

If you submit a PR to fix one of them, please refer to this issue. And if you find another exercise with this problem add a comment here.

[neenjaw]

I am working on the triangle problem's tests, and at the same time referencing https://github.com/exercism/problem-specifications/blob/master/exercises/triangle/canonical-data.json to make the tests congruent to the canonical tests. In this test I am concern about how the example implementation limits itself to a single answer because the test days proposes the following test case:

        {
          "description": "equilateral triangles are also isosceles",
          "property": "isosceles",
          "input": {
            "sides": [4, 4, 4]
          },
          "expected": true
        },

which if we test for explicitly triangle(4,4,4, "isosceles") returns true, but under this proposed change with the example solution triangle(4, 4, 4, Result), Result == "isosceles" returns false. If I just remove the ! from the equilateral predicate, then it will succeed with a choicepoint.

Is there a way to accept the test with a choicepoint without raising a warning? Or should the test be changed to findall(R, triangle(4,4,4, R), Rs), member("isosceles", Rs).

I think we should aim to be conformant to the canonical tests, so this requires an interface change somewhere. Thoughts?

[Average-user]

Maybe triangle does not need to be fixed.

[neenjaw]

I think it all depends on how we want to view the query. triangle(+SideA, +SideB, +SideC, +Type) or triangle(+SideA, +SideB, +SideC, -Type) or triangle(+SideA, +SideB, +SideC, ?Type). If we want a deterministic answer (which may be the purest prolog solution), then maybe we shouldn't fix it.

(edit) Had more time to think/read. Changed my mind.

[ErikSchierboom]

I think it all depends on how we want to view the query. triangle(+SideA, +SideB, +SideC, +Type) or triangle(+SideA, +SideB, +SideC, -Type) or triangle(+SideA, +SideB, +SideC, ?Type). If we want a deterministic answer (which may be the purest prolog solution), then maybe we shouldn't fix it.

(edit) Had more time to think/read. Changed my mind.

@neenjaw What do you suggest we do with triangle?

[neenjaw]

I haven't really thought about this issue for a long time, but I lean towards the problem set being written in such a way to promote deterministic solutions. I think those are better supported from the unit test library (if I recall correctly).

[cpmachado]

#289
Just linking this here for triangle.