BUG: Dependency-Pinning Scoring

[balteravishay]

Describe the bug

The Dependency-Pinning Scoring is taking into account all ecosystems, even if there is no evidence for them in a repository.

For instance, if a repo only contains a single shell script with a single line "pip install -r requirements.txt", the pinned-dependency score for that repo will by 8/10.
To get to a score of 0/10, means that all ecosystems that are known by scorecard exist in the repo, at the same time, and all checks for unpinned dependencies for those ecosystems report for unpinned dependencies. That seems like an edge case, at best.

Reproduction steps
Steps to reproduce the behavior:

1. Have a repo with only one ecosystem that is checked and scored by the unpinned dependency check (at the time of opening this issue the supported checks are: docker files, docker downloads, pip or scripts download, PR 🐛 Add npm installs to Pinned-Dependencies score #2960 by @gabibguti adds npm support)
2. run Scorecard with --checks=Pinned-Dependencies and the repo name
3. The result is 8/10.

Expected behavior
It is expected that the scoring is normalized only for the ecosystems that are actually found in the repository. so that a repository is only measured on the technology stacks that the repo actually implements.

Examples:

• A repository that uses python. pip install commands are unpinned: expected result is 0/10
• A repository that uses python. pip install commands are pinned: expected result is 10/10
• A repository that uses python and docker. pip install commands are pinned (i.e. use --require-hashes) while docker base images are not pinned: expected result is 5/10

Additional context
The root cause seems to be that raw dependency checks do not differentiate between not finding unpinned dependencies for an ecosystem (score 10), and not finding any evidence of using dependencies of the ecosystem the check is for.

[laurentsimon]

/cc @gabibguti

[spencerschrock]

+1 I think this could be converted to some proportional logic (e.g. x / y dependencies are pinned), or keep the equal split approach we have now, but only consider repos with dependencies present.

[spencerschrock]

Similarly I don't think we need an Info statement that dependencies are pinned for an ecosystem that isn't used.

[gabibguti]

I will work on this next.

As commented:

The root cause seems to be that raw dependency checks do not differentiate between not finding unpinned dependencies for an ecosystem (score 10), and not finding any evidence of using dependencies of the ecosystem the check is for.

First, I will work on getting the dependencies count for each ecossystem.
e.g. we have 20 pip installs, 20 go installs and 0 npm installs in this repository

Second, use this counting to know if the project depends or not on that ecosystem and only count ecossystems it depends on.
e.g. of 20 pip installs all were pinned, so that's a 10 for pip installs, of 20 go installs, one or all were not pinned, so that's a 0 for go installs, but 0 npm installs were found, so that's gonna be disconsidered in the aggregated score.

As a next step we could do a proportional score for each ecossystem as described in #2582.
e.g. of 20 go installs, 10 were not pinned, so that becomes 10/20 = 5 instead of 0.

[spencerschrock]

Hmm, I lean slightly towards the proportional approach. Consider a NPM repo with one Go tool they install in their CI:

Let's say there are 1000 dependencies, and none are pinned. And the Go dependency is pinned. Should that score a 0 (1/1001), or a 5 (average of 0 and 10)? Happy to hear other thoughts

[gabibguti]

Let's say there are 1000 dependencies, and none are pinned. And the Go dependency is pinned. Should that score a 0 (1/1001), or a 5 (average of 0 and 10)? Happy to hear other thoughts

Ah, good point, I believe 0 represents better the scenario. So I agree on averaging all dependencies.