Over-reporting of coverage

[timholy]

In working on JuliaLang/julia#7464, I've taken a closer look at how this package scores coverage. I think there's a problem of over-reporting.

Suppose we create the following file:

module Functions

export f1, f2

function f1(a, b)
    println(a, " ", b)
end

function f2(a, b)
    println(a, " ", a + b)
end

end

Now we run the following test script:

using Functions

f1(3,7)

as julia --code-coverage test.jl. We get the following coverage file:

        - module Functions
        - 
        - export f1, f2
        - 
        - function f1(a, b)
        1     println(a, " ", b)
        - end
        - 
        - function f2(a, b)
        -     println(a, " ", a + b)
        - end
        - 
        - end
        - 

As expected, f1 is marked as having run, but f2 is not. IIUC, this package will mark all lines that have a - in front of them with (javascript) null, and that is interpreted as not being relevant for coverage. So this "package" would be listed as having 100% coverage, despite the fact that half of the functions aren't tested.

To me it seems that the problem is (unfortunately) much harder, in that you have to parse the difference between lines like end that definitely shouldn't count against your coverage total vs. genuine lines of code that should have coverage but don't.

[IainNZ]

This is a dupe of #4, I'm not sure what I can do about it without using JuliaParser.jl which seems intimidating. I feel like it should probably be fixed in Julia.

[IainNZ]

If you figure how to fix this in Julia you'll be my hero :D

[timholy]

I'll look into it. Won't be for a couple of days at least. And it's not obvious that this is easy to fix there, either.