WIP Coverage reports

[chris48s]

Closes #120

This build is probably going to fail too when it tries to submit the coverage report.

I've set up this repo on coveralls at https://coveralls.io/github/codebuddies/backend

This missing piece of the jigsaw on this one is I we need to set COVERALLS_REPO_TOKEN in the repo secrets so that we can POST the coverage report, but I don't have access to do this.

Can someone with access to modify the repo secrets:

• Go to https://coveralls.io/ and "sign in with github"
• Go to https://coveralls.io/github/codebuddies/backend when you're signed in.
• Copy the repo token
• Go to https://github.com/codebuddies/backend/settings/secrets and add a secret with the name COVERALLS_REPO_TOKEN and the value as the token we copied

Then let me know. I'll confirm this works and we can wrap this one up.

Alternatively, if you'd rather use a different coverage service, let me know and we can revisit it..

Edit: gah - meant to create this as a draft PR 🙄

[aonomike]

Hi @chris48s thanks for starting work on this, I am not sure we need to add COVERALLS_REPO_TOKEN for open source projects? I was thinking maybe coveralls is able just to read that directly from the setup in coverrals.io but am not sure. I am tryna get some reference on this. But I may be wrong

[chris48s]

The only CI coveralls has that integration with is travis CI. For any other CI (e.g: Circle CI, or in our case GitHub actions) I think we do need the token to post the coverage report.

I've pushed 2332c2f to demonstrate this. Note the build fails with

Starting db ... 

Starting db ... done
Not on TravisCI. You have to provide either repo_token in .coveralls.yml or set the COVERALLS_REPO_TOKEN env var.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/coveralls/cli.py", line 61, in main
    service_name=options['--service'])
  File "/usr/local/lib/python3.7/site-packages/coveralls/api.py", line 64, in __init__
    self.ensure_token()
  File "/usr/local/lib/python3.7/site-packages/coveralls/api.py", line 73, in ensure_token
    self.config_filename))
coveralls.exception.CoverallsException: Not on TravisCI. You have to provide either repo_token in .coveralls.yml or set the COVERALLS_REPO_TOKEN env var.
##[error]Process completed with exit code 1.

[aonomike]

Yeah @chris48s you are correct, I was basing my reasoning on this post that I read a while back and I could vaguely remember the content, and this was travis, so yeah, you are correct. Thanks!

[BethanyG]

@chris48s @aonomike : Just added the secret to the repo. Not sure I did it correctly. Please give it a go, and see if it works now?? Thanks!!

[BethanyG]

@chris48s - humm. This is failing still. Should that key have been in a string? ..or is there something here that we need to do in addition??

[chris48s]

Right then. Coveralls is going to be a pain because we:

• Can't generate LCOV format from pytest-cov (or other common python tooling), which means we can't use the GH action
• Want to report code coverage on PRs from forks (this is an open contribution project), but tokens (sensibly) aren't expose to builds from forks

For more details see:

• Using for Python code and coverage.py coverallsapp/github-action#4
• Can I use only LCOV files? coverallsapp/github-action#30

Easy solution is: use codecov instead. Codecov has a github action which allows us to upload an XML coverage report without a token as long as the repo is public.
I've got that working. See https://codecov.io/gh/codebuddies/backend/pull/122/tree/project and

  _____          _
 / ____|        | |
| |     ___   __| | ___  ___ _____   __
| |    / _ \ / _` |/ _ \/ __/ _ \ \ / /
| |___| (_) | (_| |  __/ (_| (_) \ V /
 \_____\___/ \__,_|\___|\___\___/ \_/
                              Bash-tbd


==> GitHub Actions detected.
    project root: .
    Yaml not found, that's ok! Learn more at http://docs.codecov.io/docs/codecov-yaml
    -> Found 1 reports
==> Detecting git/mercurial file structure
==> Reading reports
    + ./project/coverage.xml bytes=18190
==> Appending adjustments
    http://docs.codecov.io/docs/fixing-reports
    -> No adjustments found
==> Gzipping contents
==> Uploading reports
    url: https://codecov.io
    query: branch=issue120&commit=5446dbcc24450e21f742577802ff1d0bf77f3f1d&build=66008864&build_url=http%3A%2F%2Fgithub.com%2Fcodebuddies%2Fbackend%2Factions%2Fruns%2F66008864&name=&tag=&slug=codebuddies%2Fbackend&service=github-actions&flags=&pr=122&job=
    -> Pinging Codecov
https://codecov.io/upload/v4?package=bash-tbd&token=secret&branch=issue120&commit=5446dbcc24450e21f742577802ff1d0bf77f3f1d&build=66008864&build_url=http%3A%2F%2Fgithub.com%2Fcodebuddies%2Fbackend%2Factions%2Fruns%2F66008864&name=&tag=&slug=codebuddies%2Fbackend&service=github-actions&flags=&pr=122&job=
    -> Uploading
    -> View reports at https://codecov.io/github/codebuddies/backend/commit/5446dbcc24450e21f742577802ff1d0bf77f3f1d

I've submitted a request to enable the codecov integration on this repo which will allow codecov to report on pull requests (I've not used codecov before, but hopefully once the app is enabled we should be able to test that with another push to this branch even though we don't have a cov report from master to compare to yet 🤞 )
If an admin can let me know once that's enabled, I'll push another commit..

[lpatmo]

I've submitted a request to enable the codecov integration on this repo

Saw and approved this just now; please let me know if you need anything else, @chris48s!

[BethanyG]

Looks like we're at 78%. WOOT! (and...I gotta get busy getting that number up!!). Here is the URL I am seeing: https://codecov.io/gh/codebuddies/backend/commits. Nice Work @chris48s ! Dunno if we want to play with a way we can use their SVG visualizations to report on coverage..but they're kinda cool: https://codecov.io/gh/codebuddies/backend/commit/5446dbcc24450e21f742577802ff1d0bf77f3f1d/graphs

[BethanyG]

Nice work. Thank you for doing this (twice!) and for debugging it!

[chris48s]

Dunno if we want to play with a way we can use their SVG visualizations to report on coverage..but they're kinda cool:

If you set up a codecov.yml, you can customise the pull request comment:
https://docs.codecov.io/docs/pull-request-comments#section-layout
I think if you enable reach, that will post the "grid" graph in each PR comment.

Personally I find the diff most useful, but have a play with it..

[chris48s]

actually, having looked at the comments on other issues, reach is on by default