cmd/go: go build ignores installed packages in GOPATH

[yuzhichang]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11rc1 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/abclab/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/abclab/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build545148080=/tmp/go-build -gno-record-gcc-switches"

What did you do?

1. Build and install https://github.com/infinivision/vectodb.
   Note this is cgo wrapper of some C++ code, and requires custom build script and some third-party C++ shared libraries. It cannot be built with standard "go build".
   vectodb is installed to /home/abclab/go/pkg/linux_amd64/github.com/infinivision/vectodb.a.

2. Try build https://github.com/infinivision/filesyncer (branch replace_dep) with build.sh. This script invokes "GO111MODULE=on go build...".
   I got the error:
   github.com/infinivision/vectodb
   /opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: cannot find -lfaiss
   collect2: error: ld returned 1 exit status

It tried to build vectodb with standard "go build", failed with missing an third-party C++ shared library.

What did you expect to see?

"GO111MODULE=on go build..." shall reuse installed packages in GOPATH.

What did you see instead?

"GO111MODULE=on go build..." ignored vectodb.a in GOPATH, tried to build vectodb with standard "go build", and failed.

[mvdan]

The error you got is related to ld, supposedly your system's linker. Are you sure that's not the issue?

Try running go build -x to see what commands were run.

[andybons]

@bcmills @rsc

[yuzhichang]

I'm sure system's linker works since https://github.com/infinivision/vectodb (a cgo wrapper of libfaiss and vectodb.cpp) compilation uses this linker.

https://github.com/infinivision/filesyncer (branch remove_dep) "GO111MODULE=on go build -x" output:

~/go/src/github.com/infinivision/filesyncer/cmd/faceserver ~/go/src/github.com/infinivision/filesyncer
WORK=/tmp/go-build813107023
mkdir -p $WORK/b174/
cd /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a
CGO_LDFLAGS='"-g" "-O2" "-L/home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a/faiss" "-lboost_thread" "-lboost_filesystem" "-lboost_system" "-lglog" "-lgflags" "-lfaiss" "-lopenblas" "-lgomp" "-lstdc++"' /usr/local/go/pkg/tool/linux_amd64/cgo -objdir $WORK/b174/ -importpath github.com/infinivision/vectodb -- -I $WORK/b174/ -g -O2 ./vectodb.go
cd $WORK
gcc -fno-caret-diagnostics -c -x c - || true
gcc -Qunused-arguments -c -x c - || true
gcc -fdebug-prefix-map=a=b -c -x c - || true
gcc -gno-record-gcc-switches -c -x c - || true
cd $WORK/b174
TERM='dumb' gcc -I /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b174=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x001.o -c _cgo_export.c
TERM='dumb' gcc -I /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b174=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_x002.o -c vectodb.cgo2.c
cd $WORK
g++ -fno-caret-diagnostics -c -x c - || true
g++ -Qunused-arguments -c -x c - || true
g++ -fdebug-prefix-map=a=b -c -x c - || true
g++ -gno-record-gcc-switches -c -x c - || true
cd /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a
TERM='dumb' g++ -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b174=/tmp/go-build -gno-record-gcc-switches -I $WORK/b174/ -g -O2 -I/home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a -o $WORK/b174/_x003.o -c vectodb.cpp
cd $WORK/b174
TERM='dumb' gcc -I /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b174=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a
TERM='dumb' g++ -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b174=/tmp/go-build -gno-record-gcc-switches -o $WORK/b174/_cgo_.o $WORK/b174/_cgo_main.o $WORK/b174/_x001.o $WORK/b174/_x002.o $WORK/b174/_x003.o -g -O2 -L/home/abclab/go/pkg/mod/github.com/infinivision/vectodb@v0.0.0-20180725101234-ba0c4fd4103a/faiss -lboost_thread -lboost_filesystem -lboost_system -lglog -lgflags -lfaiss -lopenblas -lgomp -lstdc++
# github.com/infinivision/vectodb
/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/ld: cannot find -lfaiss
collect2: error: ld returned 1 exit status

My point is, vectodb compilation uses custom build script, filesyncer cannot build that package with standard "GO111MODULE=on go build". There shall be some way to tell filesyncer that fact.

At filesyncer master branch, there's following line in Gopkg.toml:
ignored = ["github.com/infinivision/vectodb"]
So that filesyncer compilation reuses installed vectodb in GOPATH instead of trying to build it.

[andybons]

@ianlancetaylor

[ianlancetaylor]

There is some difference in the build ID between your installed package and the computation of the build ID done when building filesyncer. The go tool has to decide when a package is out of date, and has to rebuild the package when it is out of date. This works poorly with a package that can not be built with go build.

It might possibly help to treat this package as a binary-only package after you build it. See https://golang.org/cmd/go/#hdr-File_types . I haven't tried this so I don't know whether it will work.

[yuzhichang]

vectodb was built from git revision ba0c4fd4103ad3957bedf12d00b1c6b43fafa0bc, and installed as /home/abclab/go/pkg/linux_amd64/github.com/infinivision/vectodb.a.
In filesyncer go.mod, github.com/infinivision/vectodb v0.0.0-20180725101234-ba0c4fd4103a refers to just that tip revision.
Why installed package and the computation of the build ID when building filesyncer differ?

[rsc]

It cannot be built with standard "go build".

How does it get built?

[rsc]

Also you have not said explicitly that GO111MODULE=off go build works. Does it?
If so, then probably you just need to run your install script with GO111MODULE=on as well.
If not, then this is not a modules bug and we should unlabel it modules.

[yuzhichang]

vectodb uses custom build script SConstruct. It invokes gcc and "go build".

It works if both vectodb and filesyncer use GO111MODULE=off go build. That's what vectodb master and filesyncer master just do.

I edited vectodb and filesyncer to both use GO111MODULE=on go build. See vectodb master branch and filesyncer remove_dep branch on github.
Then I run SConstruct to build and install this module, and tried to build filesyncer. However filesyncer still failed with the same error(ignored vectodb.a in GOPATH, tried to build vectodb with standard "go build", and failed.)

[bcmills]

It might possibly help to treat this package as a binary-only package after you build it.

That will not work in module mode today: see #28146.

[bcmills]

As far as I can tell the go command is working as designed here: it correctly notices that the built artifact does not match what go build would produce from scratch for the corresponding target, and rebuilds it.

Injecting a pre-built copy of that artifact essentially treats the package as a binary-only package, for which we're dropping support (#28152) — in large part due to the difficulty of detecting and diagnosing these sorts of version skew issues.

The alternatives discussed in that proposal likely apply here as well: in particular, you may be able to compile the library separately as a Go plugin, or compile platform-specific .syso files (see #28152 (comment)).

[bcmills]

I notice that a go.mod file was added to github.com/infinivision/filesyncer on December 2. Were you able to find a workaround to get it to build?

[yuzhichang]

@bcmills For the project need, an earlier commit f5439f8e92f017daafffe38ef1c5f3aeec189b69 of github.com/infinivision/filesyncer removed dependency to github.com/infinivision/vectodb.
This issue is still valid while the procedure of the original post doesn't apply.

[bcmills]

The procedure of the original post — manually installing to GOPATH/pkg — is not going to work in the long term. I don't think there is anything more to be done for this.

(As mentioned above, the long-term alternatives that I know of are to use plugins or .syso files.)