Commit based scorecard

[KPostOffice]

Is your feature request related to a problem? Please describe.
I would like to be able to create a score card for either specific releases or commits on GitHub so that I can meaningfully show how a project's health has changed on a release by release basis.

I understand that some scorecard indicators might not work because they interact directly with the GitHub api and are not dependent on the current head commit. Is there a subset of checks that this could work with though?

[laurentsimon]

That's an interesting feature to have, thanks for the suggestion. Any checks that are looking at file content (frozen dependencies, binary artifacts checked-in, etc) would work in this model. In the future, we would like to support other type of code source: not only online repos, but also files/directories on disk, e.g. scorecard --repo=file://. This would make commit-based checks feasible.

We are also working on a BigQuery end point to expose the results of our cron run of scorecard. The projects we crawl are in cron/data/projects.csv.

@azeemshaikh38 who's leading the above and will have more comments.

[laurentsimon]

as part of the work to create a GitHub action that works on pull requests (content-based checks), I'm going to work on this soon. What I plan to do is create a new repoClient that takes as input a folder containing code that is already downloaded. The new repo client implementation will only support content-based checks and will use the folder passed as argument by users. Scorecard will then be run as:

$git clone some/repo
$scorecard ... --source-code=./repo

I will re-use this functionality for the GitHub action on pull request events.

Any comment?

[goern]

I'd love to see a Tekton task for this too ;) Let me know if you need some help with it or if Project Thoth shall go ahead and publish on on the tekton catalog.

[laurentsimon]

Hi @goern , thanks for the interest in the project. Having a tekton task sounds like an awesome thing to have!

Would you need all the checks for the task (like we do for our GitHub actions on cron and push triggers #1074), or a subset of checks that run on source-code only (basically, this PR)?

What output would you need? (we have JSON and SARIF). SARIF is based on a policy, whereas JSON is the raw output with scores for each check and an aggregate score for the repo.

If you have cycles to work on it, we'd be more than happy for you to own it.

[azeemshaikh38]

Will give this a shot. Might be something worth pursuing for v5.

[laurentsimon]

do you know which checks won't be supported? I suppose those using "out-of-band" data not tied to commits, like Fuzzing, branch protection, CI-Badges. Signed Release may be hard too. Just curious.

[azeemshaikh38]

Below checks are not commit-dependent. For those, we would just report the current status (same as HEAD).

Branch-Protection
CII-Best-Practices
Fuzzing
Signed-Releases
Maintained (because we want to see if it is maintained today, not at the commit we are checking for)
Packaging

[laurentsimon]

could be confusing to report results for HEAD vs a commit from a user's perspective. Any idea whether we should take a similar approach to --local, ie only report what's supported?

[azeemshaikh38]

Well, these checks are not commit-based in the first place. Even today, when we run these checks at HEAD, there is no correlation between the HEAD commit and the results of these checks i.e, the results of these checks can change over time even if HEAD commit hasn't changed. So I think it's ok to keep the behavior same whether user runs at HEAD or at a particular commit.

[laurentsimon]

Implicitly, running scorecard at HEAD looks for the most recent signals, whereas commits are in the past. I personally would be confused about what the results mean if it mixed commit-based results and "recent" results.

cc @naveensrinivasan @chrismcgehee @justaugustus @olivekl @oliverchang

[ristomcgehee]

I think we should use the same approach as with --local. When running for a specific commit, we just show the results for what's supported. The results for a commit should be deterministic; it shouldn't be that running against the same commit at different times gives different results (assuming the implementation of the checks does not change).

[azeemshaikh38]

Fair enough. Currently exploring the feasibility of implementing this. If I can do that, will update the CLI to not support non-commit based checks.