Missing checks in API response

[UlisesGascon]

Describe the bug

While comparing the scoring with the API:

• Request1: Nodejs (bd31d489615b424202553ffe1a5093a19d5e5e1f)
• Request2: Nodejs (1fbd908497ece0c5b08c7cc8f7494d7a12e7001f)

In the Request2 the payload does not include all the checks, for example "CI-Tests" is missing inside the checks.

Reproduction steps
Steps to reproduce the behavior:

1. You can check this in action at: https://kooltheba.github.io/openssf-scorecard-api-visualizer/#/projects/github.com/nodejs/nodejs.org/compare/1fbd908497ece0c5b08c7cc8f7494d7a12e7001f/bd31d489615b424202553ffe1a5093a19d5e5e1f (explore the network request in the chrome dev tools) or do the HTTP request directly in the browser and compare the payloads

Expected behavior

I will expect both payloads to include all the checks included in the scorecard, or at least an error code that explains this behaviour, so the API consumers can control the error.

Additional context

This bug is affecting:

• openssf-scorecard-api-visualizer cc: @KoolTheba
• OpenSSF Scorecard Monitor

[spencerschrock]

I believe this is a matter of where the result is sourced. The API pulls from the GitHub action published results if available, or the weekly cron if not.
https://github.com/ossf/scorecard-webapp/blob/e7be60069b18df32c7d3b003dd9daca6a622e6c5/app/server/get_results.go#L70-L88

diff checks1 checks2
4d3
< "CI-Tests"
6d4
< "Contributors"
8d5
< "Dependency-Update-Tool"

These are the checks disabled in the weekly cron:

scorecard/cron/config/config.yaml

Line 48 in b68ef18

blacklisted-checks: CI-Tests,Contributors,Dependency-Update-Tool,Webhooks

The output is basically tied to the checks requested, so

scorecard --repo foo/bar --checks Vulnerabilities

will only have the Vulnerabilities check in its result. Which is why those 3 are missing.

Your second request comes from before the action was installed, so there's no other data to grab.

I believe this is working as intended.

[UlisesGascon]

Thanks @spencerschrock this makes sense! Thanks for the complete response 👍