cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

kevinburke · 2018-09-25T23:07:49Z

Please answer these questions before submitting your issue. Thanks!

What did you do?

Clone the repo at https://github.com/kevinburke/go-mod-nondeterministic.

Running GO111MODULE=on go mod tidy modifies go.sum.
Running GO111MODULE=on go mod vendor changes go.sum back to the original value.

What did you expect to see?

I expect go.sum to reach a steady state at some point, where further command invocations make no changes to the file.

What did you see instead?

go.sum changes after running each of go mod tidy and go mod vendor. There's no "steady state" since the file state depends on the last command that was run.

Does this issue reproduce with the latest release (go1.11)?

Yes

System details

go version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kevin/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kevin"
GOPROXY=""
GORACE=""
GOROOT="/Users/kevin/go"
GOTMPDIR=""
GOTOOLDIR="/Users/kevin/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
GOROOT/bin/go version: go version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000 darwin/amd64
GOROOT/bin/go tool compile -V: compile version devel +93ad702251 Tue Sep 25 22:07:55 2018 +0000
uname -v: Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G65
lldb --version: lldb-1000.11.37.1
  Swift-4.2

The text was updated successfully, but these errors were encountered:

kevinburke · 2018-09-25T23:22:16Z

In addition, I get the following error when running go mod why:

$ GO111MODULE=on go mod why github.com/sourcegraph/zoekt
go: finding github.com/sourcegraph/zoekt latest
go: github.com/sourcegraph/zoekt@v0.0.0-20180925141536-852b3842c11d: parsing go.mod: unexpected module path "github.com/google/zoekt"
go: error loading module requirements

Please let me know if I should file a second issue for that.

myitcv · 2018-09-26T06:55:01Z

The delta appears within github.com/sourcegraph/zoekt so this is almost certainly some function of the replace directive that exists in the go.mod:

diff --git a/go.sum b/go.sum
index 1d2e68e..034fb23 100644
--- a/go.sum
+++ b/go.sum
@@ -26,7 +26,6 @@ github.com/pelletier/go-buffruneio v0.2.0/go.mod h1:JkE26KsDizTr40EUHkXVtNPvgGtb
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
-github.com/sourcegraph/zoekt v0.0.0-20180925141536-852b3842c11d h1:82xTRg5XBKlVsvLvgcgQzcG54AFTYhS2DdMVT5U5AWw=
 github.com/sourcegraph/zoekt v0.0.0-20180925141536-852b3842c11d/go.mod h1:f1Qnbu4glSRTb9A/H7qLv4AhudFf3eStkDbpyb21KYQ=
 github.com/src-d/gcfg v1.3.0/go.mod h1:p/UMsR43ujA89BJY9duynAwIpvqEujIH/jFlfL7jWoI=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=

cc @rsc @bcmills

kevinburke · 2018-09-28T23:10:28Z

FWIW, I'm manually editing go.sum multiple times per day on a (soon to be, large, probably popular) open source project to work around this issue.

During a user test today I also spoke with someone who said they were sticking with go1.10 because they had a bad experience with Go modules.

odiferousmint · 2018-10-02T08:44:49Z

I never run go build in the project's directory that I am going to git push because it changes the go.sum and adds lots of duplicates of dependencies with a different checksum (IIRC), otherwise I would have to remove go.sum and do go mod verify every time I want to git push. It is an annoyance and I am not sure why it does that when the vendor directory is intact.

dmitris · 2018-10-08T13:55:27Z

I'm having an issue with go.sum being modified as a side-effect of running go install ./... (or go mod why - but not with go mod tidy) - not sure if the same issue, let me know if it should go in a separate one. Go version is go1.11.1.

I have a go.mod file with 15 replace lines in the form:
replace github.com/ardielle/ardielle-go => mirror.foobar.com/ardielle/ardielle-go.git v1.5.1
replace github.com/aws/aws-sdk-go => mirror.foobar.com/aws/aws-sdk-go.git v1.15.49 // indirect
github.com/coreos/go-oidc => mirror.foobar.com/coreos/go-oidc.git v2.0.0+incompatible
github.com/golang/protobuf => mirror.foobar.com/golang/protobuf.git v1.2.0
etc.
I run go mod tidy and happily checking go.mod and go.sum. Re-running go mod tidy of course does not product any changes, as expect. But! - running go install -v ./... has a side effect of modifying go.sum which grows from 96 to 110 lines with the following lines being added:
+mirror.foobar.com/ardielle/ardielle-go.git v1.5.1 h1:s1uPWrHpT/9tcsN8To9KFgq1iOFrRn/0Db5bn7rmOWQ=
+mirror.foobar.com/aws/aws-sdk-go.git v1.15.49 h1:2wqyo+X4SGkh67paS1vqTCmS2+n/UmSYEOfAyhcnZvY= etc.
the old lines like with /go.mod suffix next to the version - like mirror/foobar.com/aws/aws-sdk-go.git v1.15.49/go.mod h1:mFuSZ37Z9YOHbQEwBWztmVzqXrEkub65tZoCYDt7FT0= - are still preserved

dmitris · 2018-10-08T14:09:39Z

@thepudds asked on Slack to post the samples of go.sum - attaching 3 files: go.mod.txt, go.sum.txt, and go.sum.modified.txt
with modifications described above introduced by running go install ./... (or go mod why -m <pkg>). Removed internal dependencies (not involved in the diffs) and replaced the mirror domain name with the fake mirror.foobar.com value.

go.mod.txt
go.sum.txt
go.sum.modified.txt

myitcv · 2018-10-19T16:03:27Z

go mod tidy appears to be at fault here (there may be other issues at play), but here's a reproduction:

$ export GOPATH=$(mktemp -d)
$ export PATH=$GOPATH/bin:$PATH
$ cd $(mktemp -d)
$ go mod init example.com/hello
go: creating new go.mod: module example.com/hello
$ cat <<EOD >main.go
package main
import _ "github.com/gopherjs/gopherjs/js"
import _ "rsc.io/quote"

func main() {}
EOD
$ go mod edit -require=github.com/gopherjs/gopherjs@latest -replace=github.com/gopherjs/gopherjs=github.com/myitcv/gopherjs@go1.11
$ go mod tidy
go: finding github.com/gopherjs/gopherjs latest
...

Look at the number of lines in go.sum for both rsc.io/quote (not replaced) and GopherJS (replaced):

$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

The single line for github.com/myitcv/gopherjs looks wrong.

"Fix" this using go list all:

$ go list all >/dev/null
go: finding github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3

and check again:

$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3 h1:gHMnSFk1naYj5G8j9hhiYQ7kKWbsHjjGSPI8sM+Fqt4=
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

Run go mod tidy again to see if our "fix" persists:

$ go mod tidy
$ cat go.sum | grep quote
rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3Y=
rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPXsUe+TKr0=
$ cat go.sum | grep gopherjs
github.com/myitcv/gopherjs v0.0.0-20181007174709-d1c34fc540e3/go.mod h1:X83RgTaopDGeqahgXrQ120BVA5Ztn+oW4kxNueqOja4=

It doesn't; hence we conclude go mod tidy is over aggressively pruning go.sum.

The "fix" here could involve the use of any go command that effectively invokes a build step, hence install, build, list etc all work.

dmitris · 2018-11-08T11:04:00Z

@bcmills - do you (or anyone from the Go team) agree that go mod tidy needs to be rectified? Is anyone working on this? What are the chances this can be fixed in 1.12?

dmitris · 2018-11-08T11:06:43Z

maybe as a workaround it would be possible for go mod tidy to "internally" run an equivalent of go list all >& /dev/null to "effectively invoke a build step" and "untidy" its over-aggressive purning?

bcmills · 2018-11-08T13:23:38Z

@dmitris, I'm planning to investigate this for 1.12 during the freeze. If the fix is not too invasive, it should make the 1.12 release.

gopherbot · 2018-12-12T13:59:21Z

Change https://golang.org/cl/153817 mentions this issue: cmd/go: retain sums for replacement modules in 'go mod tidy'

Signed-off-by: Sylvain Rabot <sylvain@abstraction.fr>

The correct content was generated with Go 1.12beta2. `go mod tidy` should not be run using previous versions of Go. See golang/go#27868 for context.

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>

kevinburke added the modules label Sep 25, 2018

kevinburke mentioned this issue Sep 26, 2018

cmd/go: go mod tidy, vendor hide module load errors #27063

Closed

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 26, 2018

bcmills added this to the Go1.12 milestone Sep 26, 2018

bcmills self-assigned this Sep 26, 2018

bcmills changed the title ~~cmd/go: "go mod tidy" / "go mod vendor" can't agree on correct go.sum~~ cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' Oct 9, 2018

myitcv mentioned this issue Oct 19, 2018

cmd/go: go mod tidy always prints "finding" log messages #27123

Closed

thepudds mentioned this issue Oct 21, 2018

cmd/go: go mod tidy misses adding certain lines to go.sum #28129

Closed

myitcv mentioned this issue Oct 25, 2018

cmd/go: "go mod tidy" removes an entry added to go.sum by "go build" for a replacement module #27176

Closed

myitcv mentioned this issue Nov 7, 2018

'go mod tidy' removes entries related to the 'replace' instruction #28632

Closed

rarkins mentioned this issue Nov 8, 2018

Go modules - vendoring support renovatebot/renovate#2580

Closed

myitcv mentioned this issue Nov 11, 2018

cmd/egrunner: simplify checking for shell asserts myitcv/x#102

Merged

bcmills mentioned this issue Nov 14, 2018

cmd/go: use go.sum hashes from all downloaded modules #28802

Closed

This was referenced Dec 10, 2018

cmd/go: go mod tidy revoke the modify about go.sum made by go build #29163

Closed

cmd/go: 'go mod download' of a replaced module downloads the go.mod for its replacement #29181

Open

gopherbot closed this as completed in cc8ae42 Dec 12, 2018

sylr added a commit to sylr/prometheus-azure-exporter that referenced this issue Dec 13, 2018

travis: fix golang/go#27868

ea3e7fa

Signed-off-by: Sylvain Rabot <sylvain@abstraction.fr>

rliebz added a commit to rliebz/tusk that referenced this issue Jan 24, 2019

Fix go.sum file with correctly tidied content

5235dce

The correct content was generated with Go 1.12beta2. `go mod tidy` should not be run using previous versions of Go. See golang/go#27868 for context.

fenollp added a commit to FuzzyMonkeyCo/monkey that referenced this issue Feb 10, 2019

weaken restrictions on spec filename + hit golang/go#27868

228bc67

Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>

markfuller mentioned this issue May 10, 2019

Add build checks for go fmt and go mod tidy lyraproj/lyra#268

Closed

golang locked and limited conversation to collaborators Dec 12, 2019

gopherbot added the FrozenDueToAge label Dec 12, 2019

rsc unassigned bcmills Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

kevinburke commented Sep 25, 2018

kevinburke commented Sep 25, 2018

myitcv commented Sep 26, 2018

kevinburke commented Sep 28, 2018 •

edited

Loading

odiferousmint commented Oct 2, 2018 •

edited

Loading

dmitris commented Oct 8, 2018 •

edited

Loading

dmitris commented Oct 8, 2018 •

edited

Loading

myitcv commented Oct 19, 2018

dmitris commented Nov 8, 2018

dmitris commented Nov 8, 2018

bcmills commented Nov 8, 2018

gopherbot commented Dec 12, 2018

cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

cmd/go: incorrect sum for replaced module in either 'mod tidy' or 'mod vendor' #27868

Comments

kevinburke commented Sep 25, 2018

What did you do?

What did you expect to see?

What did you see instead?

Does this issue reproduce with the latest release (go1.11)?

System details

kevinburke commented Sep 25, 2018

myitcv commented Sep 26, 2018

kevinburke commented Sep 28, 2018 • edited Loading

odiferousmint commented Oct 2, 2018 • edited Loading

dmitris commented Oct 8, 2018 • edited Loading

dmitris commented Oct 8, 2018 • edited Loading

myitcv commented Oct 19, 2018

dmitris commented Nov 8, 2018

dmitris commented Nov 8, 2018

bcmills commented Nov 8, 2018

gopherbot commented Dec 12, 2018

kevinburke commented Sep 28, 2018 •

edited

Loading

odiferousmint commented Oct 2, 2018 •

edited

Loading

dmitris commented Oct 8, 2018 •

edited

Loading

dmitris commented Oct 8, 2018 •

edited

Loading