Update coverage tool to manually filter some Fortran90 continuation lines #604
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR checklist
Update the coverage implementation, manually filter out some lines from the coverage tool
apcraig
One source code line was changed to help the filter. Ran a quick suite on cheyenne with 3 compilers, those test pass fine, https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks#a09e51fb161b59af72f63f39c6247c9698a2ace2
The gcov/lcov tool occasionally generates false negatives or incorrect
statistics with respect to coverage for Fortran 90 continuation lines (using "&").
In some cases, within a single continuation block, some lines have hits which
are correctly counted while other lines have misses which are incorrect and
also counted. This seems to be partly associate with the lcov -a feature that we
use to aggregate multiple tests, but it also probably is ultimately created by gcov
because it doesn't seem to handle skipped vs missed lines entirely consistently.
With this modification, --coverage creates a temporary sandbox where the tests are
run. Prior to running the tests, lcov_modify_source.sh is invoked.
" ! LCOV_EXCL_LINE" (a special defined string in lcov) is added to the end
of some source code lines to request that gcov exclude them. The lines excluded
are Fortran 90 continuation lines. In all cases the first and last line of a
continuation block and lines that continue after an "if" are never excluded.
This seems to improve the accuracy of the coverage output overall.
Several other attempts were made to modify the lcov and geninfo perl scripts to
handle the Fortran90 continuation line filtering there without success, so this
solution was implemented.
To see a before and after for the same code,
before: https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html
after: https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html
A good place to see the error/differences is after line 255 of
https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html
and
https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html
In the first case, 20+ lines of code are recorded as missed even though they are hit. In the updated coverage output, most of those lines are skipped and the statistics are cleaned up significantly. Coverage for the routine goes from 73% to 90%, and I think more accurately reflects the true coverage of the routine. In addition, manually searching for missing coverage is easier if some of the false negatives are removed from the output. The total coverage went from 70.3% to 71.6% with this change which suggest this problem is fairly localized.