Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update coverage tool to manually filter some Fortran90 continuation lines #604

Merged
merged 2 commits into from
May 27, 2021

Conversation

apcraig
Copy link
Contributor

@apcraig apcraig commented May 26, 2021

PR checklist

  • Short (1 sentence) summary of your PR:
    Update the coverage implementation, manually filter out some lines from the coverage tool
  • Developer(s):
    apcraig
  • Suggest PR reviewers from list in the column to the right.
  • Please copy the PR test results link or provide a summary of testing completed below.
    One source code line was changed to help the filter. Ran a quick suite on cheyenne with 3 compilers, those test pass fine, https://github.com/CICE-Consortium/Test-Results/wiki/cice_by_hash_forks#a09e51fb161b59af72f63f39c6247c9698a2ace2
  • How much do the PR code changes differ from the unmodified code?
    • bit for bit
    • different at roundoff level
    • more substantial
  • Does this PR create or have dependencies on Icepack or any other models?
    • Yes
    • No
  • Does this PR add any new test cases?
    • Yes
    • No
  • Is the documentation being updated? ("Documentation" includes information on the wiki or in the .rst files from doc/source/, which are used to create the online technical docs at https://readthedocs.org/projects/cice-consortium-cice/. A test build of the technical docs will be performed as part of the PR testing.)
    • Yes
    • No, does the documentation need to be updated at a later time?
      • Yes
      • No
  • Please provide any additional information or relevant details below:

The gcov/lcov tool occasionally generates false negatives or incorrect
statistics with respect to coverage for Fortran 90 continuation lines (using "&").
In some cases, within a single continuation block, some lines have hits which
are correctly counted while other lines have misses which are incorrect and
also counted. This seems to be partly associate with the lcov -a feature that we
use to aggregate multiple tests, but it also probably is ultimately created by gcov
because it doesn't seem to handle skipped vs missed lines entirely consistently.

With this modification, --coverage creates a temporary sandbox where the tests are
run. Prior to running the tests, lcov_modify_source.sh is invoked.
" ! LCOV_EXCL_LINE" (a special defined string in lcov) is added to the end
of some source code lines to request that gcov exclude them. The lines excluded
are Fortran 90 continuation lines. In all cases the first and last line of a
continuation block and lines that continue after an "if" are never excluded.
This seems to improve the accuracy of the coverage output overall.

Several other attempts were made to modify the lcov and geninfo perl scripts to
handle the Fortran90 continuation line filtering there without success, so this
solution was implemented.

To see a before and after for the same code,

before: https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html

after: https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/index.html

A good place to see the error/differences is after line 255 of

https://apcraig.github.io/lcov_cice_210520-185156:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html

and

https://apcraig.github.io/lcov_cice_210526-003153:a09e51fb16:8:first,base,travis,decomp,reprosum,io,quick,unittest/cicecore/cicedynB/dynamics/ice_dyn_evp.F90.gcov.html

In the first case, 20+ lines of code are recorded as missed even though they are hit. In the updated coverage output, most of those lines are skipped and the statistics are cleaned up significantly. Coverage for the routine goes from 73% to 90%, and I think more accurately reflects the true coverage of the routine. In addition, manually searching for missing coverage is easier if some of the false negatives are removed from the output. The total coverage went from 70.3% to 71.6% with this change which suggest this problem is fairly localized.

The gcov/lcov tool occasionally generates false negatives or incorrect
statistics with respect to coverage for Fortran 90 continuation lines (using "&").
In some cases, within a single continuation block, some lines have hits which
are correctly counted while other lines have misses which are incorrect and
also counted.  This seems to be partly associate with the lcov -a feature that we
use to aggregate multiple tests, but it also probably is ultimately created by gcov
because it doesn't seem to handle skipped vs missed lines entirely consistently.

With this modification, --coverage creates a temporary sandbox where the tests are
run.  Prior to running the tests, lcov_modify_source.sh is invoked.
"  ! LCOV_EXCL_LINE" (a special defined string in lcov) is added to the end
of some source code lines to request that gcov exclude them.  The lines excluded
are Fortran 90 continuation lines.  In all cases the first and last line of a
continuation block and lines that continue after an "if" are never excluded.
This seems to improve the accuracy of the coverage output overall.

Several other attempts were made to modify the lcov and geninfo perl scripts to
handle the Fortran90 continuation line filtering there without success, so this
solution was implemented.
Copy link
Contributor

@eclare108213 eclare108213 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting!

@apcraig apcraig merged commit bd512d4 into CICE-Consortium:master May 27, 2021
@phil-blain
Copy link
Member

Nice approach @apcraig, simple and effective. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants