cmd/go: allow verifying vendored code

[krancour]

go mod verify is extremely useful for validating the integrity of modules in the local cache.

It would be great if projects that choose to vendor their modules (then presumably building with go build -mod vendor ...) had a similar command to verify the integrity of modules in that directory.

This would satisfy a major requirement that many projects need to account for in their CI process-- ensuring that vendored code hasn't been tampered with.

[bcmills]

This is still on our radar, but probably not happening for 1.13. (We have a lot to do this cycle!)

I'm hoping to get to it in 1.14, but we don't have a 1.14 milestone defined yet.

[bcmills]

No, it's definitely not happening for 1.15.

[ianlancetaylor]

Moving milestone to Backlog.

[iholder101]

This is indeed valuable! Projects that choose to vendor their modules usually do not want to count on every package to be available 100% of the time, therefore validation that enforces re-downloading the package is problematic.

Would be happy to see this implemented.

[ArsenArsen]

There seems to be a fundamental issue here, vendor directories exclude many files not used in the actual build. Verifying against go.sum partially becomes impossible.

I'd like to automatically generate vendored tarballs from sources, while being able to verify them, so this issue becomes relevant to me.

Is there any discussed solution that went unrecorded in this thread?

Thanks in advance

[bcmills]

@ArsenArsen, the plan at the moment is to have go mod verify download the full module source code for each dependency.

Then we can demonstrate the integrity of the vendor tree by showing that, for each package in the vendor tree, the directory for that package contains exactly the expected subset of source files, and the contents of each of those files match the contents of the corresponding file in the module cache.

[bcmills]

(However, note that go mod verify cannot be implemented for vendor trees in a way that is independent of the module cache.)

[ArsenArsen]

@bcmills Hm, while that works and will produce valid results, the thing we're implementing requires us to be able to do so without the network.

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment? I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Thanks in advance, and for the quick response.

[bcmills]

Would it be possible to create a vendor archive that contains all the code required for building and verification to run in this constrained environment?

In theory you could checksum each package, but the go.sum file (and sum.golang.org) only contains checksums for modules and I don't see that changing.

I was considering shipping a tarred up version of GOMODCACHE, however that is likely unsupported.

Actually, that would work (once we implement go mod verify for vendored code at all).
You can set GOPROXY to point to $GOMODCACHE/cache/download to treat a module cache as a module proxy. (See #43646.)

[petar-dochev-f3]

A problem we recently hit is that due to some recursive patterns in .gitignore some files under vendor were not committed to git, which resulted in some strange errors in CI which could not be reproduced locally. It would be very helpful if there is a command we could use use in our CI to verify vendor is intact. Also sometimes people make changes in vendor by accident.

[qmuntal]

@bcmills what if go mod vendor creates a vendor.sum file next to modules.txt and go mod verify checks the integrity of the vendored modules against vendor.sum?

The downside is that the verify command wouldn't have any external source (e.g., GOSUMDB) to validate vendor.sum checksums, but AFAIK that's not happening neither for non-vendored modules (see #47752). vendor.sum could be recreated from the module cache using go mod vendor, so I don't see this as a blocker.

[bcmills]

I guess a vendor.sum could help defend against stray edits, at least. That's an interesting thought.

[jpbetz]

Since this has been open for almost 5 years, maybe recommend a mitigation? Kubernetes performs this check with https://github.com/kubernetes/kubernetes/blob/master/hack/verify-vendor.sh. I'm looking to do the same for github.com/google/cel-go, and having a recipe would be a huge time saver.

[bcmills]

@jpbetz, you can run the following:

• go mod vendor (to redownload and checksum the dependencies in the vendor tree)
• Then check the vendor directory for diffs (to make sure it matches what was downloaded).
• Then run go mod verify -mod=readonly (to verify that the contents of downloaded modules match their checksums).

[jpbetz]

Thanks @bcmills

In cel-go we ended up using this script: https://github.com/google/cel-go/blob/master/scripts/verify-vendor.sh

[aep-sunlife]

This is a security concern. An attacker lying about the contents of vendor/ in go.{mod,sum} can inject malware.