Customization of results

[laurentsimon]

Some ideas

• criticality of checks for Action prioritization. Requires Feature: Make existing results more structured scorecard#1874
• Ignore checks
• Ignore paths
• Reintroduce a way to silence individual warnings #143 (see also Won't Fix alerts are still shown github/codeql-action#986 (comment))

[laurentsimon]

/cc @raghavkaul

[naveensrinivasan]

This should be more of a config option. But isn't this supposed to be in Scorecard rather than action? That way it is consistent and Scorecard action is just another way to run scorecard.

[laurentsimon]

The Scorecard Action can build a policy on top of the results, in which case this feature need / should not be in Scorecard, especially if we don't want to enforce / support a universal policy style / language / framework. That's what I had in mind. This keeps the Scorecard results "universal", same as the cron results. The policy could be applied on top of cron results, for example, using whatever logic the consumer wants.

Wdut?

[azeemshaikh38]

Another customization to consider is improving Wont Fix behavior for Scorecard checks. see github/codeql-action#986 (comment).

[laurentsimon]

yes, that's the 143 issue above. I'll update the description. Thanks

[azeemshaikh38]

yes, that's the 143 issue above. I'll update the description. Thanks

Ah sorry, didn't see it. My bad.

[naveensrinivasan]

The Scorecard Action can build a policy on top of the results, in which case this feature need / should not be in Scorecard, especially if we don't want to enforce / support a universal policy style / language / framework. That's what I had in mind. This keeps the Scorecard results "universal", same as the cron results. The policy could be applied on top of cron results, for example, using whatever logic the consumer wants.

Wdut?

Can you please unpack what you mean with an example? For example, I envisioned that end-users can apply a policy on top scorecard results to fit their needs.

For example

1. Score Customization for a given repository and a given check
2. Ignoring certain checks from certain folders.

[laurentsimon]

pretty much what you said. Not sure that score customization is useful if we have more structured input, but maybe.

They could also customize the criticality of results, or ignore certain "details" using the detailID field suggested in ossf/scorecard#1874 (comment)

Let me know if this is a bit clearer or not. Thanks for asking

[naveensrinivasan]

pretty much what you said. Not sure that score customization is useful if we have more structured input, but maybe.

They could also customize the criticality of results, or ignore certain "details" using the detailID field suggested in ossf/scorecard#1874 (comment)

Let me know if this is a bit clearer or not. Thanks for asking

Yes, it is! Thanks. Then why not have this functionality in Scorecard versus here. That is what I don't understand.

[laurentsimon]

Thanks @naveensrinivasan for pushing on this.

Initially I thought we would not want to commit to a particular policy format in the main scorecard repo, and let consumers decide how they want to handle it. You're making a good point that we could support a simple policy format in the main scorecard repo, and it would benefit everyone. So I think I'm fine with that.

@azeemshaikh38 any further thought about this?

[laurentsimon]

fyi, here's a possible policy file:

Version: X
BasePolicy: BuiltInAllChecksEnforced | BuiltInAllChecksEnforced | __path_to_file__
IgnoredPaths:
IgnoredFiles:
EnforcedPaths:
EnforcedFiles:
Checks:
  - CheckName:
    - Status: enforced | ignored
    - Criticality: XXX
    - IgnoredPaths:
      - a_path_
      - a/**/path
    - IgnoredFiles:
      - a_file
      - another_file
    - EnforcedPaths:
     ...
    - EnforcedFiles:
    ...
    - Rules:
      # RuleName is the same as `DetailID` in https://github.com/ossf/scorecard/issues/1874#issuecomment-1178259870
      - RuleName: 
         - Status: enforced | ignored
         - Criticality: XXX
         - IgnoredPaths:
         - IgnoredFiles:
         - EnforcedPaths:
         - EnforcedFiles:

[naveensrinivasan]

The community is leaning towards OPA https://github.com/open-policy-agent/opa.

[laurentsimon]

Yes. Kind of why I suggested earlier to have something simple only in the Action and not commit to something complex / to main in the main CLI as well.

I have not looked into OPA or the complexity for both us or the user. I think a simple solution first would help users and would not require them to learn a new programming language. AllStar takes this approach and is able to express many policies this way.

Later we can provide template OPA policy (I think we discussed this a few months ago but did not have bandwidth to look more closely).

Wdut?

[naveensrinivasan]

That would be breaking a change in Policy.
Scorecard should avoid breaking changes and we would have to support both of them.
We should rather do it right that is my opinion. Is there a rush for this feature?

If I am not wrong @developer-guy was working on cosign with OPA.

[laurentsimon]

That would be breaking a change in Policy. Scorecard should avoid breaking changes and we would have to support both of them. Is there a rush for this feature?

A lot of people want ways to opt out. Having a simple policy and actually delivering it seems like a good step forward to me. It's not always viable to do everything at once, and iterations are always necessary. We don't even know what other policies would want, so betting on OPA seems overkill.

[naveensrinivasan]

We should have some to measure lot of people want ways to opt out

This should be a discussion topic for bi-weekly

[laurentsimon]

I went thru the issues last week and found 3-4 issues of people asking for customization like this. It's also something people have been complaining about in conversations.

[naveensrinivasan]

OK, Go ahead if you have heard from people in conversations 👍 . I just wanted to be sure that we are building what customers want. Thanks for the explanation!

[laurentsimon]

I'd rather hear from more people, and also talk about it in the bi-weekly as you mentioned. Was just trying to say there's demand, but I agree with you on discussing it to decide on prioritization.

Thanks again for all the questions you asked

[laurentsimon]

/cc @ethanent @jeffmendoza may be relevant to the work on AllStar policies for Scorecard.

[jonasbb]

I requested a way to silence reports in #143. In that case, I want to relax the requirements for CI code. Not all code needs to uphold the same quality measures, mainly test code and CI code. The proposed policy file would work for me. It would allow suppressing the Pinned-Dependency check for the GitHub Actions code, while keeping it for the rest. It also looks flexible enough to further differentiate between different projects in the same repo or other parts.

In the proposed policy file, there are both IgnoredPaths and IgnoredFiles. This seems somewhat redundant and a single option might work, similar to the syntax of gitignore files.