Allowing users to integrate external checks via blank-imports

[spencerschrock]

Original proposal

In #3055 @naveensrinivasan proposed using the plugin functionality (https://pkg.go.dev/plugin) to handle:

It would be nice for users to be able to run their own custom checks in addition to those available in Scorecard.

While plugin exists, it comes with many constraints and warnings.

Runtime crashes are likely to occur unless all parts of the program (the application and all its plugins) are compiled using exactly the same version of the toolchain, the same build tags, and the same values of certain flags and environment variables.

Similar crashing problems are likely to arise unless all common dependencies of the application and its plugins are built from exactly the same source code.

Together, these restrictions mean that, in practice, the application and its plugins must all be built together by a single person or component of a system. In that case, it may be simpler for that person or component to generate Go source files that blank-import the desired set of plugins and then compile a static executable in the usual way.

blank-import

Instead, I want to focus on the blank-import approach, as that's what scorecard does today for its own checks:

scorecard/checks/binary_artifact.go

Lines 24 to 36 in bfd5357

	// CheckBinaryArtifacts is the exported name for Binary-Artifacts check.
	const CheckBinaryArtifacts string = "Binary-Artifacts"
	//nolint
	func init() {
	supportedRequestTypes := []checker.RequestType{
	checker.CommitBased,
	}
	if err := registerCheck(CheckBinaryArtifacts, BinaryArtifacts, supportedRequestTypes); err != nil {
	// this should never happen
	panic(err)
	}
	}

If we make the necessary changes to allow external modules to do the same, custom checks becomes a possibility:

Imagine some develops a custom, external check in their repo: github.com/foo/mycustomcheck

package mycustomcheck // github.com/foo/mycustomcheck

import (
	"github.com/ossf/scorecard/v4/checker"
	"github.com/ossf/scorecard/v4/checks"
)

const name = "My-Custom-Check"

func init() {
	checks.RegisterCheck(name, MyCustomCheck, nil)
}

func MyCustomCheck(r *checker.CheckRequest) checker.CheckResult {
	return checker.CreateMaxScoreResult(name, "my check found nothing wrong!")
}

If a consumer uses a small wrapper around scorecard's main function, the check could be integrated just by adding blank-imports, without modifying the scorecard code themselves:

package main


import (
    "log"

    "github.com/ossf/scorecard/v4/cmd"
    "github.com/ossf/scorecard/v4/options"

    _ "github.com/foo/mycustomcheck"
    _ "github.com/bar/someothercheck"
)


func main() {
    opts := options.New()
    if err := cmd.New(opts).Execute(); err != nil {
        log.Fatalf("error during command execution: %v", err)
    }
}

Changes required

• Export RegisterCheck
• Figure out an alternative to checkDocs.GetCheck(check.Name) which will work with external checks
  • need to get things like the risk to determine score weight, documentation links, etc.

[spencerschrock]

With some hacky local modifications (see Changes required), the above code works at a PoC level for incorporating external checks:

go run main.go --repo ossf/scorecard --checks My-Custom-Check --format json | jq
{
  "date": "2023-05-31T12:30:54-07:00",
  "repo": {
    "name": "github.com/ossf/scorecard",
    "commit": "bfd535755379bbe13d156b262fc9dd8ec33e23a4"
  },
  "scorecard": {
    "version": "",
    "commit": "unknown"
  },
  "score": 10,
  "checks": [
    {
      "details": null,
      "score": 10,
      "reason": "my check found nothing wrong!",
      "name": "My-Custom-Check",
      "documentation": {
        "url": "",
        "short": ""
      }
    }
  ],
  "metadata": null
}

[naveensrinivasan]

Amazing! Thanks @spencerschrock

[spencerschrock]

Export RegisterCheck

Instead of exporting registerCheck, it may make more sense to have a RegisterExternalCheck to handle any differences we want to enforce between standard and external checks.

[github-actions]

Stale issue message