Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Coverage trend checker #17

Merged
merged 2 commits into from
Oct 25, 2024
Merged

feat: Coverage trend checker #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
90b60e8
feat: Coverage trend workflow
aborgna-q Oct 22, 2024
1d4e95b
Update README
aborgna-q Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
124 changes: 124 additions & 0 deletions .github/workflows/coverage-trend.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
name: Check coverage trend on Codecov
on:
workflow_call:
secrets:
CODECOV_GET_TOKEN:
description: 'The token to query the codecov API.'
required: true
outputs:
should_notify:
description: 'Returns true if the coverage has changed.'
value: ${{ jobs.check-coverage.outputs.should_notify }}
msg:
description: 'The message describing the change. Suitable to post on Slack.'
value: ${{ jobs.check-coverage.outputs.msg }}

jobs:
check-coverage:
runs-on: ubuntu-latest
outputs:
msg: ${{ steps.make_msg.outputs.msg }}
should_notify: ${{ steps.get_coverage.outputs.should_notify }}
steps:
- name: Download commit sha of the most recent successful run
uses: dawidd6/action-download-artifact@v6
with:
# Downloads the artifact from the most recent successful run
workflow: 'coverage-trend.yml'
name: head-sha.txt
if_no_artifact_found: ignore
- name: Get today's and last run's coverage trends from codecov
id: get_coverage
# API reference: https://docs.codecov.com/reference/repos_totals_retrieve
run: |
# Get the previous commit coverage, if the last sha is available
if [ ! -f head-sha.txt ]
then
echo "No previous coverage found."

# Update the head-sha.txt file with the current sha,
# so next time we campare against the current coverage.
echo ${{ github.sha }} > head-sha.txt

echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi

PREV_SHA=$( cat head-sha.txt )
echo "Previous sha: \"$PREV_SHA\""

# Check if the sha has changed
if [ "$PREV_SHA" == "${{ github.sha }}" ]
then
echo "No new commits since last run."
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi

# Query the previous coverage from codecov
curl --request GET \
--url "https://api.codecov.io/api/v2/github/${{ github.repository_owner }}/repos/${{ github.event.repository.name }}/totals/?sha=$PREV_SHA" \
--header 'accept: application/json' \
--header "authorization: Bearer ${{ secrets.CODECOV_GET_TOKEN }}" \
> coverage-prev.json
cat coverage-prev.json | jq ".totals.coverage" > coverage-prev.txt
echo "Previous coverage query result:"
cat coverage-prev.json | jq "del(.files)"
echo

# Query the current coverage from codecov
curl --request GET \
--url "https://api.codecov.io/api/v2/github/${{ github.repository_owner }}/repos/${{ github.event.repository.name }}/totals/?sha=${{ github.sha }}" \
--header 'accept: application/json' \
--header "authorization: Bearer ${{ secrets.CODECOV_GET_TOKEN }}" \
> coverage.json
cat coverage.json | jq ".totals.coverage" > coverage.txt
echo "Current coverage query result:"
cat coverage.json | jq "del(.files)"
echo

echo
echo "Previous coverage: `cat coverage-prev.txt`%"
echo "Current coverage: `cat coverage.txt`%"

# A `null` in either coverage means that the coverage is not available,
# so we don't want to notify about that.
if [ "$( cat coverage-prev.txt )" == "null" ]
then
echo "Previous coverage not available."
echo ${{ github.sha }} > head-sha.txt
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "$( cat coverage.txt )" == "null" ]
then
echo "Current coverage not available."
# Note that we don't update the head-sha.txt file here,
# so next time we compare against the one that had coverage data.
echo "should_notify=false" >> "$GITHUB_OUTPUT"
exit 0
fi

echo ${{ github.sha }} > head-sha.txt
echo "should_notify=true" >> "$GITHUB_OUTPUT"
- name: Compare with previous summary and make message
id: make_msg
if: steps.get_coverage.outputs.should_notify == 'true'
run: |
prev=`cat coverage-prev.txt`
current=`cat coverage.txt`
change=`printf "%.2f%% --> %.2f%%" $prev $current`
codecov="https://codecov.io/gh/${{ github.repository }}?search=&trend=7%20days"
if (( $(echo "$prev < $current + 0.04" | bc -l) ))
then
MSG="msg=Coverage check for ${{ github.repository }} shows no regression (${change}). ✅ ${codecov}"
else
MSG="msg=Coverage check for ${{ github.repository }} shows regression (${change}). ❌ ${codecov}"
fi
echo $MSG
echo $MSG >> "$GITHUB_OUTPUT"
- name: Upload current HEAD sha
uses: actions/upload-artifact@v4
with:
name: head-sha.txt
path: head-sha.txt
133 changes: 89 additions & 44 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,99 @@ access the GitHub API. For these we [generate fine-grained access
tokens](https://github.com/settings/personal-access-tokens/new) with the
@hugrbot bot account, which must be stored in the repository secrets.

## Workflows

The following workflows are available:

- [`add-to-project`](#add-to-project): Adds new issues to a GitHub project board when they are created.
- [`coverage-trend`](#coverage-trend): Checks the coverage trend for the project, and produces a summary that can be posted to slack.
- [`drop-cache`](#drop-cache): Drops the cache for a branch when a pull request is closed.
- [`pr-title`](#pr-title): Checks the title of pull requests to ensure they follow the conventional commits format.
- [`rs-semver-checks`](#rs-semver-checks): Runs `cargo-semver-checks` on a PR against the base branch, and reports back if there are breaking changes.
- [`add-to-project`](#add-to-project): Adds new issues to a GitHub project board when they are created.

### [`drop-cache`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/drop-cache.yml)
## [`add-to-project`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/add-to-project.yml)

Adds new issues to a GitHub project board when they are created.

### Usage
```yaml
name: Add issues to project board
on:
issues:
types:
- opened

jobs:
add-to-project:
uses: CQCL/hugrverse-actions/.github/workflows/add-to-project.yml@main
with:
project-url: https://github.com/orgs/{your-org}/projects/{project-id}
secrets:
GITHUB_PAT: ${{ secrets.ADD_TO_PROJECT_PAT }}
```

### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Projects | Read and write |
| Pull requests | Read |

Note that fine-grained access tokens cannot grant permissions to projects and repositories in different organisations simultaneously.
In those cases, you will need an unrestricted _classical_ github token instead.

## [`coverage-trend`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/coverage-trend.yml)

Compares the project coverage on [Codecov](https://codecov.io/) against the last workflow run,
and produces a summary of the changes that can be posted to slack.

If the project didn't have new commits that changed the coverage since the last run,
the `should_notify` output will be set to `false` and the `msg` output will be empty.

### Usage
```yaml
name: Notify coverage changes
on:
schedule:
# 04:00 every Monday
- cron: '0 4 * * 1'
workflow_dispatch: {}

jobs:
coverage-trend:
uses: CQCL/hugrverse-actions/.github/workflows/coverage-trend.yml@main
secrets:
CODECOV_GET_TOKEN: ${{ secrets.CODECOV_GET_TOKEN }}
# Post the result somewhere.
notify-slack:
needs: coverage-trend
runs-on: ubuntu-latest
if: needs.coverage-trend.outputs.should_notify == 'true'
steps:
- name: Send notification
uses: slackapi/slack-github-action@v1.27.0
with:
channel-id: "SOME CHANNEL ID"
slack-message: ${{ needs.coverage-trend.outputs.msg }}
env:
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
```

### Outputs

- `should_notify`: Whether there has been a change in coverage since the last run, which we can post about.
- `msg`: A message summarising the coverage changes. This is intended to be posted to slack.

### Token Permissions

`CODECOV_GET_TOKEN` is a token generated by Codecov to access the repository's coverage data.

## [`drop-cache`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/drop-cache.yml)

Drops the cache for a branch when a pull request is closed. This helps to avoid
cache pollution by freeing up some of github's limited cache space.

#### Usage
### Usage
```yaml
name: cleanup caches by a branch
on:
Expand All @@ -37,14 +115,14 @@ jobs:
uses: CQCL/hugrverse-actions/.github/workflows/drop-cache.yml@main
```

### [`pr-title`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/pr-title.yml)
## [`pr-title`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/pr-title.yml)

Checks the title of pull requests to ensure they follow the [conventional
commits](https://www.conventionalcommits.org/en/v1.0.0/) format. If the title
does not follow the conventional commits, a comment is posted on the PR to help
the user fix it.

#### Usage
### Usage
```yaml
name: Check Conventional Commits format
on:
Expand All @@ -67,21 +145,21 @@ jobs:
GITHUB_PAT: ${{ secrets.GITHUB_PAT }}
```

#### Token Permissions
### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Pull requests | Read and write |

### [`rs-semver-checks`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/rs-semver-checks.yml)
## [`rs-semver-checks`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/rs-semver-checks.yml)

Runs `cargo-semver-checks` on a PR against the base branch, and reports back if
there are breaking changes.
Suggests adding a breaking change flag to the PR title if necessary.

#### Usage
### Usage
```yaml
name: Rust Semver Checks
on:
Expand All @@ -98,44 +176,11 @@ jobs:

The workflow compares against the base branch of the PR by default. Use the `baseline-rev` input to specify a different base commit.

#### Token Permissions
### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Pull requests | Read and write |


### [`add-to-project`](https://github.com/CQCL/hugrverse-actions/blob/main/.github/workflows/add-to-project.yml)

Adds new issues to a GitHub project board when they are created.

#### Usage
```yaml
name: Add issues to project board
on:
issues:
types:
- opened

jobs:
add-to-project:
uses: CQCL/hugrverse-actions/.github/workflows/add-to-project.yml@main
with:
project-url: https://github.com/orgs/{your-org}/projects/{project-id}
secrets:
GITHUB_PAT: ${{ secrets.ADD_TO_PROJECT_PAT }}
```

#### Token Permissions

The fine-grained `GITHUB_PAT` secret must include the following permissions:

| Permission | Access |
| --- | --- |
| Projects | Read and write |
| Pull requests | Read |

Note that fine-grained access tokens cannot grant permissions to projects and repositories in different organisations simultaneously.
In those cases, you will need an unrestricted _classical_ github token instead.