From f8ce10a455fa740c6b4b9862dd387d241a4012bb Mon Sep 17 00:00:00 2001 From: Spencer Schrock Date: Thu, 20 Jun 2024 14:52:59 -0700 Subject: [PATCH] add probe lifecycle to documentation Signed-off-by: Spencer Schrock --- docs/probes.md | 94 +++++++++++++++++++++++++++ docs/probes/internal/generate/main.go | 1 + 2 files changed, 95 insertions(+) diff --git a/docs/probes.md b/docs/probes.md index ca9dfebdb70..16c7db619ad 100644 --- a/docs/probes.md +++ b/docs/probes.md @@ -7,6 +7,8 @@ please [contribute](../CONTRIBUTING.md)! ## archived +**Lifecycle**: stable + **Description**: Check that the project is archived **Motivation**: An archived project will not received security patches, and is not actively tested or used. @@ -19,6 +21,8 @@ If the project is not archived, the outcome is OutcomeFalse. ## blocksDeleteOnBranches +**Lifecycle**: stable + **Description**: Check that the project blocks non-admins from deleting branches. **Motivation**: Allowing non-admins to delete project branches has a similar effect to performing force pushes. @@ -30,6 +34,8 @@ If the project is not archived, the outcome is OutcomeFalse. ## blocksForcePushOnBranches +**Lifecycle**: stable + **Description**: Check that the project blocks force push on its branches. **Motivation**: Allowing force pushes to branches could allow those with write access to make insecure changes to the behavior of the project. @@ -42,6 +48,8 @@ Returns OutcomeNotAvailable if Scorecard cannot fetch the data from the reposito ## branchProtectionAppliesToAdmins +**Lifecycle**: stable + **Description**: Check that the project's branch protection rules apply to project admins. **Motivation**: Admins may be able to bypass branch protection settings which could defeat the purpose of having them. @@ -53,6 +61,8 @@ Returns OutcomeNotAvailable if Scorecard cannot fetch the data from the reposito ## branchesAreProtected +**Lifecycle**: stable + **Description**: Check that the project uses protected branches. **Motivation**: Unprotected branches may allow actions that could compromise the project's security. @@ -64,6 +74,8 @@ Returns OutcomeNotAvailable if Scorecard cannot fetch the data from the reposito ## codeApproved +**Lifecycle**: stable + **Description**: Check that all recent changesets have been approved by someone who is not the author of the changeset. **Motivation**: To ensure that the review process works, the proposed changes should have a minimum number of approvals. @@ -77,6 +89,8 @@ If there are no changes, the probe returns OutcomeNotApplicable ## codeReviewOneReviewers +**Lifecycle**: experimental + **Description**: Check that at least one reviewers review a change before merging. **Motivation**: To ensure that the review process works, the proposed changes should have a minimum number of approvals. @@ -89,6 +103,8 @@ If the changes had fewer than one reviewers, the prove returns OutcomeFalse (0) ## contributorsFromOrgOrCompany +**Lifecycle**: experimental + **Description**: Checks whether a project has a contributions from users associated with a company or organization. **Motivation**: This probe tries to determine if the project has recent contributors from multiple organizations. For some projects, having a diverse group of contributors is an indicator of project health. @@ -101,6 +117,8 @@ If the project has contributing organizations, the probe returns 1 OutcomeTrue p ## createdRecently +**Lifecycle**: stable + **Description**: Checks if the project was created in the last 90 days. **Motivation**: Recently created repositories have been used for malicious forks / typosquatting attacks in the past. A newly created repo is not a strong signal on its own, but can be a useful piece of information. @@ -113,6 +131,8 @@ If the project is older than 90 days, the outcome is OutcomeFalse. The finding w ## dependencyUpdateToolConfigured +**Lifecycle**: stable + **Description**: Check that a dependency update tool config is present. **Motivation**: Out-of-date dependencies make a project vulnerable to known flaws and prone to attacks. Tools can help the process of updating dependencies by scanning for outdated or insecure requirements, and opening a pull request to update them if found. @@ -125,6 +145,8 @@ If no tool is detected, the probe returns OutcomeFalse. ## dismissesStaleReviews +**Lifecycle**: stable + **Description**: Check that the project dismisses stale reviews when new commits are pushed. **Motivation**: When a project does not dismiss stale reviews, contributors can bring their pull requests to an approved state and then make unreviewed commits. @@ -136,6 +158,8 @@ If no tool is detected, the probe returns OutcomeFalse. ## fuzzed +**Lifecycle**: stable + **Description**: Check that the project is fuzzed **Motivation**: Fuzzing, or fuzz testing, is the practice of feeding unexpected or random data into a program to expose bugs. Regular fuzzing is important to detect vulnerabilities that may be exploited by others, especially since attackers can also use fuzzing to find the same flaws. @@ -148,6 +172,8 @@ If no fuzzing tool is found, or the project uses a tool we don't detect, one fin ## hasBinaryArtifacts +**Lifecycle**: stable + **Description**: Checks if the project has any binary files in its source tree. **Motivation**: Binary files are not human readable so users and reviewers can't easily see what they do. @@ -160,6 +186,8 @@ If the probe finds no binary files, it returns a single OutcomeFalse. ## hasDangerousWorkflowScriptInjection +**Lifecycle**: stable + **Description**: Check whether the project has GitHub Actions workflows that enable script injection. **Motivation**: Script injections allow attackers to use untrusted input to access privileged resources (code execution, secret exfiltration, etc.) @@ -172,6 +200,8 @@ If no dangerous patterns are found, the probe returns one finding with OutcomeFa ## hasDangerousWorkflowUntrustedCheckout +**Lifecycle**: stable + **Description**: Check whether the project has GitHub Actions workflows that does untrusted checkouts. **Motivation**: GitHub workflows triggered with pull_request_target or workflow_run have write permission to the target repository and access to target repository secrets. Combined with a dangerous checkout of PR contents, attackers may be able to compromise the repository, for example, by using build scripts controlled by the PR author. @@ -184,6 +214,8 @@ The probe returns one finding with OutcomeFalse if no untrusted checkouts are de ## hasFSFOrOSIApprovedLicense +**Lifecycle**: stable + **Description**: Check that the project has an FSF or OSI approved license. **Motivation**: A license can give users information about how the source code may or may not be used. Using a recognized license facilitates security or legal reviews for potential users. @@ -197,6 +229,8 @@ If the license is not of an approved format, the probe returns a single OutcomeF ## hasLicenseFile +**Lifecycle**: stable + **Description**: Check that the project has a license file **Motivation**: A license can give users information about how the source code may or may not be used. The lack of a license will impede any kind of security review or audit and creates a legal risk for potential users. @@ -209,6 +243,8 @@ If a license file is not found, the probe returns a single OutcomeFalse. ## hasNoGitHubWorkflowPermissionUnknown +**Lifecycle**: experimental + **Description**: Checks that GitHub workflows have workflows with unknown permissions **Motivation**: Unknown permissions may be a result of a bug or another error from fetching the permission levels. @@ -221,6 +257,8 @@ The probe returns 1 true outcome if the project has no workflows with unknown pe ## hasOSVVulnerabilities +**Lifecycle**: stable + **Description**: Check whether the project has known vulnerabilities **Motivation**: This check determines whether the project has open, unfixed vulnerabilities in its own codebase or its dependencies using the OSV (Open Source Vulnerabilities) service. An open vulnerability may be exploited by attackers and should be fixed as soon as possible. @@ -233,6 +271,8 @@ If there are no known vulnerabilities detected, the probe returns one false outc ## hasOpenSSFBadge +**Lifecycle**: stable + **Description**: This check determines whether the project has an OpenSSF (formerly CII) Best Practices Badge. **Motivation**: The OpenSSF Best Practices badge indicates whether or not the project uses a set of security-focused best development practices for open source software. @@ -245,6 +285,8 @@ If the project does not have a badge, the probe returns one OutcomeFalse. ## hasPermissiveLicense +**Lifecycle**: stable + **Description**: Check that the project has an permissive license. **Motivation**: A permissive license allows users to use the analyzed component to be used in derivative works. Non-permissive licenses (as copyleft licenses) might be a legal risk for potential users. @@ -258,6 +300,8 @@ If the license is not permissive, the probe returns a single OutcomeFalse. ## hasRecentCommits +**Lifecycle**: stable + **Description**: Check whether the project has at least one commit per week over the last 90 days. **Motivation**: A project which is not active might not be patched, have its dependencies patched, or be actively tested and used. A lack of active maintenance should signal that potential users should investigate further to judge the situation. A project may not need further features or maintenance; In this case, the probe results can be disregarded. @@ -270,6 +314,8 @@ If the project does not have commits in the last 90 days, the probe returns a si ## hasReleaseSBOM +**Lifecycle**: experimental + **Description**: Check that the project publishes an SBOM as part of its release artifacts. **Motivation**: An SBOM can give users information about how the source code components and dependencies. They help facilitate sotware supplychain security and aid in identifying upstream vulnerabilities in a codebase. @@ -282,6 +328,8 @@ If an SBOM artifact is not found, the probe returns a single OutcomeFalse. ## hasSBOM +**Lifecycle**: experimental + **Description**: Check that the project has an SBOM file **Motivation**: An SBOM can give users information about how the source code components and dependencies. They help facilitate sotware supplychain security and aid in identifying upstream vulnerabilities in a codebase. @@ -294,6 +342,8 @@ If an SBOM file is not found, the probe returns a single OutcomeFalse. ## hasUnverifiedBinaryArtifacts +**Lifecycle**: stable + **Description**: Checks if the project has binary files in its source tree. The probe skips verified binary files which currently are gradle-wrappers. **Motivation**: Binary files are not human readable so users and reviewers can't easily see what they do. @@ -306,6 +356,8 @@ If the probe finds no unverified binary files, it returns OutcomeFalse. ## issueActivityByProjectMember +**Lifecycle**: stable + **Description**: Checks that a collaborator, member or owner has participated in issues in the last 90 days. **Motivation**: A project which does not respond to issues may not be actively maintained. A lack of active maintenance should signal that potential users should investigate further to judge the situation. However a project may simply not have any recent issues; In this case, the probe results can be disregarded. @@ -318,6 +370,8 @@ If collaborators, members or owners have NOT participated in issues in the last ## jobLevelPermissions +**Lifecycle**: experimental + **Description**: Checks that GitHub workflows do not have "write" permissions at the "job" level. **Motivation**: In some circumstances, having "write" permissions at the "job" level may enable attackers to escalate privileges. @@ -330,6 +384,8 @@ The probe returns 1 true outcome if the project has no workflows "write" permiss ## packagedWithAutomatedWorkflow +**Lifecycle**: stable + **Description**: Checks whether the project uses automated packaging. **Motivation**: Packages give users of a project an easy way to download, install, update, and uninstall the software by a package manager. In particular, they make it easy for users to receive security patches as updates. @@ -342,6 +398,8 @@ If the project doesn't use automated packaing we can detect, the outcome is nega ## pinsDependencies +**Lifecycle**: stable + **Description**: Check that the project pins dependencies to a specific digest. **Motivation**: Pinned dependencies ensure that checking and deployment are all done with the same software, reducing deployment risks, simplifying debugging, and enabling reproducibility. They can help mitigate compromised dependencies from undermining the security of the project (in the case where you've evaluated the pinned dependency, you are confident it's not compromised, and a later version is released that is compromised). @@ -355,6 +413,8 @@ If the project has no supported dependencies, the probe returns OutcomeNotApplic ## releasesAreSigned +**Lifecycle**: stable + **Description**: Check that the projects GitHub and GitLab releases are signed. **Motivation**: Signed releases allow consumers to verify their artifacts before consuming them. @@ -368,6 +428,8 @@ If the project has no releases, the probe returns OutcomeNotApplicable. ## releasesHaveProvenance +**Lifecycle**: stable + **Description**: Check that the projects releases on GitHub and GitLab have provenance. **Motivation**: Provenance give users security-critical, verifiable information so that consumers can verify their artifacts before consuming them. @@ -381,6 +443,8 @@ If the project has no releases, the probe returns OutcomeNotApplicable. ## releasesHaveVerifiedProvenance +**Lifecycle**: experimental + **Description**: Checks if the project releases with provenance attestations that have been verified **Motivation**: Package provenance attestations provide a greater guarantee of authenticity and integrity than package signatures alone, since the attestation can be performed over a hash of both the package contents and metadata. Developers can attest to particular qualities of the build, such as the build environment, build steps or builder identity. @@ -393,6 +457,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## requiresApproversForPullRequests +**Lifecycle**: stable + **Description**: Check that the project requires approvers for pull requests. **Motivation**: Requiring approvers for pull requests makes it harder to introduce vulnerable code to the project. @@ -404,6 +470,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## requiresCodeOwnersReview +**Lifecycle**: stable + **Description**: Check that the project requires dedicated code owners to review PRs. **Motivation**: Code owners are expected to have deep knowledge about a code; Having experienced reviewers for PRs is expected to prevent security issues. @@ -415,6 +483,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## requiresLastPushApproval +**Lifecycle**: stable + **Description**: Check that the project requires approval of the most recent push. **Motivation**: Requiring approval of the most recent push prevents contributors from sneaking malicious commits into a PR after it has been approved. @@ -426,6 +496,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## requiresPRsToChangeCode +**Lifecycle**: stable + **Description**: Check that the project requires pull requests to change code. **Motivation**: Changing code through pull requests promotes testing and reviews of the suggested change. @@ -437,6 +509,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## requiresUpToDateBranches +**Lifecycle**: stable + **Description**: Check that the project requires PRs to be in sync with the base branch. **Motivation**: Requiring PRs to be in sync with the base branch is good practice. @@ -448,6 +522,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## runsStatusChecksBeforeMerging +**Lifecycle**: stable + **Description**: Check that the project runs required status checks **Motivation**: Required status checks can check for common errors and resolve issues in PRs. @@ -459,6 +535,8 @@ If we didn't find a package or didn't find releases, return OutcomeNotAvailable. ## sastToolConfigured +**Lifecycle**: stable + **Description**: Check that the project uses a SAST tool **Motivation**: SAST is testing run on source code before the application is run. Using SAST tools can prevent known classes of bugs from being inadvertently introduced in the codebase. @@ -471,6 +549,8 @@ If the project does not use a SAST tool, or uses a tool we dont currently detect ## sastToolRunsOnAllCommits +**Lifecycle**: stable + **Description**: Checks that a SAST tool runs on all commits in the projects CI. **Motivation**: SAST is testing run on source code before the application is run. Using SAST tools can prevent known classes of bugs from being inadvertently introduced in the codebase. @@ -484,6 +564,8 @@ If the project does not run any SAST tools successfully on every pull request be ## securityPolicyContainsLinks +**Lifecycle**: stable + **Description**: Check that the security policy contains web or email links. **Motivation**: URLs point users to additional information as well as online disclosure forms. Emails provide a point of contact for vulnerability disclosure. @@ -497,6 +579,8 @@ If no security policy files are found, one finding with OutcomeFalse is returned ## securityPolicyContainsText +**Lifecycle**: stable + **Description**: Check that the security policy contains enough text and not just links. **Motivation**: Telling security researchers how to privately disclose problems with your project is important. The more details available, the better. @@ -510,6 +594,8 @@ If no security policy is found, one finding with OutcomeFalse is returned. ## securityPolicyContainsVulnerabilityDisclosure +**Lifecycle**: stable + **Description**: Check that the security policy indicates a vulnerability disclosure process. **Motivation**: If someone finds a vulnerability in the project, it is important for them to be able to communicate it to the maintainers. @@ -523,6 +609,8 @@ If no security policy is found, the probe returns one finding with OutcomeFalse. ## securityPolicyPresent +**Lifecycle**: stable + **Description**: Check if a security policy is defined in the repository or in the org's .github repository. **Motivation**: A security policy (typically a SECURITY.md file) can give users information about what constitutes a vulnerability and how to report one securely so that information about a bug is not publicly visible. If you have a large organization, having a unified security policy across all your repositories may simplify the vulnerability disclosure response. @@ -535,6 +623,8 @@ If no security file is found, one finding with OutcomeFalse is returned. ## testsRunInCI +**Lifecycle**: stable + **Description**: Checks that the project runs tests in the CI for example with GitHub Actions or Prow. **Motivation**: Running tests helps developers catch mistakes early on, which can reduce the number of vulnerabilities that find their way into a project. @@ -547,6 +637,8 @@ The probe returns a single OutcomeNotApplicable if the projects has had no pull ## topLevelPermissions +**Lifecycle**: experimental + **Description**: Checks that the project does not have any top-level write permissions in its workflows. **Motivation**: In some circumstances, having "write" permissions at the "top" level may enable attackers to escalate privileges. @@ -559,6 +651,8 @@ The probe returns 1 true outcome if the project has no workflows "write" permiss ## webhooksUseSecrets +**Lifecycle**: experimental + **Description**: This check determines whether the webhooks defined in the repository have secrets configured to authenticate the origins of requests. **Motivation**: Webhooks without secret authorization have the potential to make projects accessible to third-parties. diff --git a/docs/probes/internal/generate/main.go b/docs/probes/internal/generate/main.go index 0ee461f1675..126a1a17fc4 100644 --- a/docs/probes/internal/generate/main.go +++ b/docs/probes/internal/generate/main.go @@ -39,6 +39,7 @@ func printField(w io.Writer, name string, value any) { func printProbe(w io.Writer, p *pyaml.Probe) { // short, motivation, implementation, outcome, remediation, ecosystem fmt.Fprint(w, "\n"+"## "+p.ID+"\n\n") + printField(w, "Lifecycle", p.Lifecycle) printField(w, "Description", p.Short) printField(w, "Motivation", p.Motivation) printField(w, "Implementation", p.Implementation)