Skip to content

Latest commit

 

History

History
243 lines (178 loc) · 10.9 KB

gomod.md

File metadata and controls

243 lines (178 loc) · 10.9 KB
Raw

gomod

https://go.dev/ref/mod

Specifying modules to process

cachi2 fetch-deps \
  --source ./my-repo \
  --output ./cachi2-output \
  '<modules JSON>'

Module1 JSON:

{
  // "gomod" tells Cachi2 to process a go module
  "type": "gomod",
  // path to the module (relative to the --source directory)
  // defaults to "."
  "path": "."
}

The main argument accepts alternative forms of input, see usage: pre-fetch-dependencies.

Using fetched dependencies

See also usage.md for a complete example of Cachi2 usage.

Cachi2 downloads the required modules into the deps/gomod/ subpath of the output directory (cachi2-output/deps/gomod). Further down the file tree, at cachi2-output/deps/gomod/pkg/mod, is a directory formatted as the Go module cache.

cachi2-output/deps/gomod/pkg/mod
└── cache
    └── download
        ├── github.com
        │   └── ...
        └── golang.org
            └── ...

To use this module cache during your build, set the GOMODCACHE environment variable. Cachi2 generates GOMODCACHE along with other expected environment variables for you. See usage: generate environment variables for more details.

For more information on Go's environment variables:

go help environment

Note that the deps/gomod/ layout described above does not apply when using vendoring. With vendoring enabled, deps/gomod/ will be an empty directory. Instead, dependencies will be inside the vendor subdirectory of your module.

my-repo
└── vendor
    ├── github.com
    │   └── ...
    ├── golang.org
    │   └── ...
    └── modules.txt

Go will use the vendored dependencies automatically, but it's not a bad idea to set the environment variables generated by Cachi2 anyway.

gomod flags

The cachi2 fetch-deps command accepts the following gomod-related flags:

--cgo-disable

Makes Cachi2 internally disable cgo while processing your Go modules. Typically, you would want to use this flag if your modules do use C code and Cachi2 is failing to process them. Cachi2 will not attempt to disable cgo in your build (nor should you disable it yourself if you rely on C).

Disabling cgo should not prevent Cachi2 from fetching your Go dependencies as usual. Note that Cachi2 will not make any attempts to fetch missing C libraries. If required, you would need to get them through other means.

--force-gomod-tidy

Makes Cachi2 run go mod tidy after downloading dependencies.

⚠ This flag is questionable and may be removed in the future.

Vendoring

Go supports vendoring to store the source code of all dependencies in the vendor/ directory alongside your module. Before go 1.17, go mod vendor used to download fewer dependencies than go mod download. Starting with 1.17, that is no longer true - see the overview in the README.

We generally discourage vendoring, but Cachi2 does support processing repositories that contain vendored content. In this case, instead of a regular prefetching of dependencies, Cachi2 will only validate if the contents of the vendor directory are consistent with what go mod vendor would produce.

Deprecated flags

Cachi2's behavior towards vendoring used to be governed by two flags:

  • --gomod-vendor
  • --gomod-vendor-check

Both are deprecated and will have no effect when set. They are only kept for backwards compatibility reasons.

Understanding reported dependencies

Cachi2 reports two (arguably three) different types of dependencies in the generated SBOM for your Go modules:

  • gomod dependencies (Go modules)
  • go-package dependencies (Go packages)

gomod vs go-package

Best explained by the Go modules documentation:

A module is a collection of packages that are released, versioned, and distributed together.

Your Go code imports individual packages, which come from modules. You might import a single package from a module that provides many, but Go (and Cachi2) has to download the whole module anyway. Effectively, modules are the smallest "unit of distribution." Go does have the ability to list the individual packages that your project imports. Cachi2 makes use of this ability to report both the downloaded modules and the required packages.

The list of go-package dependencies reported by Cachi2 is the full set of packages (transitively) required by your project. ⚠ If any of your module dependencies is missing a checksum in go.sum, the list may be incomplete.

The list of gomod dependencies is the set of modules that Cachi2 downloaded to satisfy the go-package dependencies.

Note that versioning applies to modules, not packages. When reporting the versions of Go packages, Cachi2 uses the version of the module that provides the package.

How to match a package to a module?

Borrowing from the modules documentation again:

For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html"

The name of a package starts with the name of the module that provides it.

In the source tree, what are modules? What are packages?

To simplify a little:

  • Does the directory have a go.mod file? It's a module (provides packages).
  • Does the directory have any *.go files? It's a package (is importable).
  • Does it have both? It's both a module and a package.

stdlib dependencies

Go is able to list even the standard library packages that your project imports. Cachi2 exposes these as go-package dependencies, with caveats. Cachi2 uses some version of Go to list the dependencies. This may or may not be the same version that you will use to build your project. We do not presume that the versions would be the same, hence why:

  • the reported stdlib packages may be slightly inaccurate (e.g. new packages in new Go versions)
  • the versions of stdlib packages are not reported

What identifies stdlib dependencies in the go-package list?

  • does not have a version
  • the name does not start with a hostname
    • io/fs - standard library
    • golang.org/x/net - external

Missing checksums

Go stores the checksums of all your dependency modules in the go.sum file. Go typically manages this file entirely on its own, but if any of your dependencies do end up missing, it can cause issues for Cachi2 and for Go itself.

For Cachi2, a missing checksum means that the offending module gets downloaded without checksum verification (or with partial checksum verification - Cachi2 does consult the Go checksum database). Due to go list behavior, it also means that the go-package dependency listing may be incomplete2.

For Go, a missing checksum will cause the go build or go run commands to fail.

Please make sure to keep your go.sum file up to date, perhaps by incorporating the go mod tidy command in your dev workflow.

Go 1.21+ (since cachi2-v0.5.0)

Starting with Go 1.21, Go changed the meaning of the go 1.X directive in that it now specifies the minimum required version of Go rather than a suggested version as it originally did. The format of the version string in the go directive now also includes the micro release and if you don't include the micro release in your go.mod file yourself (i.e. you only specify the language release) Go will try to correct it automatically inside the file. Last but not least, Go 1.21 also introduced a new keyword toolchain to the go.mod file. What this all means in practice for end users is that you may not be able to process your go.mod file with an older version of Go (and hence older cachi2) as you could in the past for various reasons. Many projects bump their required Go toolchain's micro release as soon as it becomes available upstream (i.e. not waiting for distributions to bundle them properly). This caused problems for cachi2-v0.5.0 because the container image's version simply may not have been high enough to process a given project's go.mod file. Therefore, cachi2-v0.7.0 implemented a mechanism to always rely on the origin 0th release of a toolchain (e.g. 1.21.0) and use the GOTOOLCHAIN=auto setting to instruct Go to fetch any toolchain as specified by the go.mod file automatically, hence allowing us to keep up with frequent micro version bumps. Note that such a language version would still need to be officially marked as supported by cachi2, i.e. we'd not allow Go to fetch e.g. a 1.22 toolchain if the maximum supported Go version by cachi2 were 1.21!

Footnotes

  1. You may have noticed a slight naming issue. You use the main argument, also called PKG, to specify a module to process. Even worse, Go has packages as well (see gomod vs go-package). What gives? As far as we know, most languages/package managers use the opposite naming. For example, in Python, modules are *.py files, packages are collections of modules. In npm, modules are directories/files you can require(), packages are the top-level directories with package.json. In Cachi2, we stick to the more common naming. ↩

  2. When a module does not have a checksum in go.sum, the go list command returns only basic information and an error for the packages from said module. Go doesn't return any information about the dependencies of the affected packages. This can cause Cachi2 to miss the transitive package dependencies of packages from checksum-less modules. ↩