gh-94759: Collect C-level coverage using llvm-cov #94760
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| name: C coverage | ||
|
|
||
| on: | ||
| schedule: | ||
| # Run this daily at 2400 UTC | ||
| - cron: '0 0 * * *' | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| jobs: | ||
| c-coverage: | ||
| name: 'C-level coverage (using llvm-cov)' | ||
| runs-on: ubuntu-20.04 | ||
| env: | ||
| OPENSSL_VER: 1.1.1n | ||
| PYTHONSTRICTEXTENSIONBUILD: 1 | ||
| steps: | ||
| - name: Install Dependencies | ||
| run: | | ||
| sudo ./.github/workflows/posix-deps-apt.sh | ||
| - name: Configure OpenSSL env vars | ||
| run: | | ||
| echo "MULTISSL_DIR=${GITHUB_WORKSPACE}/multissl" >> $GITHUB_ENV | ||
| echo "OPENSSL_DIR=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}" >> $GITHUB_ENV | ||
| echo "LD_LIBRARY_PATH=${GITHUB_WORKSPACE}/multissl/openssl/${OPENSSL_VER}/lib" >> $GITHUB_ENV | ||
| - name: Restore OpenSSL build | ||
| id: cache-openssl | ||
| uses: actions/cache@v3 | ||
| with: | ||
| path: ./multissl/openssl/${{ env.OPENSSL_VER }} | ||
| key: ${{ runner.os }}-multissl-openssl-${{ env.OPENSSL_VER }} | ||
| - name: Install OpenSSL | ||
| if: steps.cache-openssl.outputs.cache-hit != 'true' | ||
| run: python3 Tools/ssl/multissltests.py --steps=library --base-directory $MULTISSL_DIR --openssl $OPENSSL_VER --system Linux | ||
| - name: Add ccache to PATH | ||
| run: | | ||
| echo "PATH=/usr/lib/ccache:$PATH" >> $GITHUB_ENV | ||
| - name: Configure ccache action | ||
| uses: hendrikmuhs/ccache-action@v1 | ||
| - name: Configure clang (with coverage) | ||
| run: | | ||
| echo "CC=clang" >> $GITHUB_ENV | ||
| echo "CXX=clang++" >> $GITHUB_ENV | ||
| - name: Configure CPython | ||
| run: ./configure --with-pydebug --with-openssl=$OPENSSL_DIR | ||
| - name: Generate coverage report | ||
| # Using "-j1" is important, or the Github Action runs out of memory | ||
| run: EXTRATESTOPTS=-j1 xvfb-run make coverage-report | ||
| - name: Publish coverage-report | ||
|
There was a problem hiding this comment. Does every run clobber the previous run's report? What about CI runs on PRs and CI runs from release branches? Can this be configured to commit the coverage results into a branch in the repo matching the reponame+branchname? Or do branches not render in gh-pages? in which case it'd need to be subdirectories in the repo. There was a problem hiding this comment. Currently, this clobbers the previous run. That is easily changed with a flag, but then we could run into Github repository size limits (each result is around 100MB of HTML). It's probably possible to keep the last N commits, but the tool I'm using to publish doesn't support that directly. This is currently configured to just run on the There was a problem hiding this comment. If we would get some funding to run a persistent VM with public hostname from ... let's say Azure, then it would be trivial to create a buildbot worker and serve the LCOV results from HTTP server. They are static HTTP and JS files on the file system after all. Just saying :) There was a problem hiding this comment. I don't mind Regarding a buildbot configured to host the results, while I could simply set one up it'd probably make more sense for mdboom to do that and be an admin given who's driving this work. :) There was a problem hiding this comment. Yeah, I'd be happy to admin that if it comes to it. I think it's fine to go with the Github Action here as an MVP, and if the amount of history or frequency of runs isn't good enough, we can revisit migrating to our own VM down the road. |
||
| uses: JamesIves/github-pages-deploy-action@v4 | ||
| with: | ||
| folder: coverage-report | ||
| repository-name: '' # TODO Destination | ||
| token: ${{ secrets.COVERAGE_DEPLOY_TOKEN }} # TODO: Use an organization-level token | ||
|
There was a problem hiding this comment. What permissions does the token need? There was a problem hiding this comment. That's a good question, and it isn't terribly well-documented: https://github.com/JamesIves/github-pages-deploy-action#required-setup I think the permission to push to the coverage results repo is all that's needed, but we may not know until we try. There was a problem hiding this comment. Does it need two separate tokens then, one (with read permissions) for the cpython repo, and another (with write permissions) for the coverage results repo? There was a problem hiding this comment. We don't need a token for read permissions to the cpython repo. Just one to write to a repo under the python org. |
||
| single-commit: true | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -23,6 +23,7 @@ | ||
| *.gc?? | ||
| *.profclang? | ||
| *.profraw | ||
| *.profdata | ||
| *.dyn | ||
| .gdb_history | ||
| .purify | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| The ``coverage-report`` Makefile now supports both the ``gcc/lcov`` and ``clang/llvm-profdata`` stacks to generate coverage reports, and will select the correct one based on the compiler in use. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason for using 20.04 instead of 22.04?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is what the current
build.ymluses, and I think there is value in being consistent with it.build.ymlis run in every PR, this will be in a nightly cronjob, so it would be annoying if a theoretical Ubuntu-version-specific bug crept in. (That said, we could consider upgrading all of the Ubuntu's in tandem in a separate PR, of course...)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1