cmd/link: Go tip fails to build with lld linker

[aykevl]

While building go tip (at commit 3403ee5), I got the following error:

~/src/go/src$ ./make.bash
Building Go cmd/dist using /usr/local/go.
Building Go toolchain1 using /usr/local/go.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/amd64.
# cmd/trace
runtime/cgo(.text): relocation target pthread_mutex_lock not defined for ABI0 (but is defined for ABI0)
runtime/cgo(.text): relocation target pthread_mutex_unlock not defined for ABI0 (but is defined for ABI0)
# cmd/go
runtime/cgo(.text): relocation target pthread_mutex_lock not defined for ABI0 (but is defined for ABI0)
runtime/cgo(.text): relocation target pthread_mutex_unlock not defined for ABI0 (but is defined for ABI0)
# cmd/pprof
runtime/cgo(.text): relocation target pthread_mutex_lock not defined for ABI0 (but is defined for ABI0)
runtime/cgo(.text): relocation target pthread_mutex_unlock not defined for ABI0 (but is defined for ABI0)
go tool dist: FAILED: /home/ayke/src/go/pkg/tool/linux_amd64/go_bootstrap install -gcflags=all= -ldflags=all= std cmd: exit status 2

This is because I use the lld linker from LLVM:

/usr/bin$ ll ld
lrwxrwxrwx 1 root root 8 mrt 22 19:26 ld -> ld.lld-8*

When I change the symlink to point to ld.gold or ld.bfd, compiling Go works as expected. I don't know what the error exactly means (pthread_mutex_* is both defined and not defined for ABI0?) but would expect Go to compile just fine with lld as the system linker.

My system is Debian stretch with LLVM 8 installed from here: http://apt.llvm.org/

What version of Go are you using (go version)?

$ go version
go version go1.12 linux/amd64

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ayke/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ayke"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="/usr/bin/gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build176331235=/tmp/go-build -gno-record-gcc-switches"

[ianlancetaylor]

CC @aclements

It's hard to understand why changing to lld would cause this error to appear. This error comes from the Go linker before the C linker is ever invoked.

[aykevl]

To be really sure, I tried again with ld.lld-8, ld.bfd, and again ld.lld-8. The error persists, and only when ld points to ld.lld.

[cuonglm]

It works fine with ld.lld-6.

The command executed during make.bash:

[pid 22896] execve("/usr/bin/ld", ["/usr/bin/ld", "-plugin", "/usr/lib/gcc/x86_64-linux-gnu/5/liblto_plugin.so", "-plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper", "-plugin-opt=-fresolution=/tmp/ccpfuiXA.res", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "-plugin-opt=-pass-through=-lpthread", "-plugin-opt=-pass-through=-lc", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "--sysroot=/", "--build-id", "--eh-frame-hdr", "-m", "elf_x86_64", "--hash-style=gnu", "--as-needed", "-dynamic-linker", "/lib64/ld-linux-x86-64.so.2", "-z", "relro", "-o", "/tmp/go-build852796803/b085/_cgo_.o", "/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crt1.o", "/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crti.o", "/usr/lib/gcc/x86_64-linux-gnu/5/crtbegin.o", "-L/usr/lib/gcc/x86_64-linux-gnu/5", "-L/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu", "-L/usr/lib/gcc/x86_64-linux-gnu/5/../../../../lib", "-L/lib/x86_64-linux-gnu", "-L/lib/../lib", "-L/usr/lib/x86_64-linux-gnu", "-L/usr/lib/../lib", "-L/usr/lib/gcc/x86_64-linux-gnu/5/../../..", "/tmp/go-build852796803/b085/_cgo_main.o", "/tmp/go-build852796803/b085/_x001.o", "/tmp/go-build852796803/b085/_x002.o", "/tmp/go-build852796803/b085/_x003.o", "/tmp/go-build852796803/b085/_x004.o", "/tmp/go-build852796803/b085/_x005.o", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "-lpthread", "-lc", "-lgcc", "--as-needed", "-lgcc_s", "--no-as-needed", "/usr/lib/gcc/x86_64-linux-gnu/5/crtend.o", "/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/crtn.o"], [/* 109 vars */]) = 0

[aclements]

I can reproduce this on my linux/amd64 Debian testing-ish machine by changing /usr/bin/ld to point to ld.lld-8. (Curiously, GO_LDFLAGS='-extld ld.lld-8' ./make.bash was not enough to reproduce it.)

I think the problem occurs when the cgo tool invokes the linker when building the cgo object for runtime/cgo. In the final link step that fails, like Ian mentioned, we haven't invoked the external linker at all yet. The only time we invoke the external linker is from the cgo tool, and in the build of cmd/go that only happens when building the runtime/cgo and net packages.

To demonstrate:

./make.bash
go install cmd/buildid cmd/pack
strace -f -e execve -s 256 go build -a -x -o /dev/null cmd/go >& /tmp/st

Search /tmp/st for /usr/bin/ld. The surrounding context shows it being invoked by cgo.

[aclements]

@cherrymui pointed out that _cgo_import.go is missing //go:cgo_import_dynamic pragmas for the two pthread_mutex_* symbols, even though it appears to have them for every other pthread symbol we use. If we put the bfd linker back, we get those //go:cgo_import_dynamic pragmas.

go build -a -x -work runtime/cgo
cat $WORK/b001/_cgo_import.go

This appears to be because the _cgo_.o file (which is actually a linked binary, not an object file) contains weak references to pthread_mutex_* when linked with lld:

$ readelf --dyn-syms $WORK/b001/_cgo_.o | grep pthread_
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (3)
    12: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_cond_wait@GLIBC_2.3.2 (4)
    13: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (3)
    14: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_cond_broadcast@GLIBC_2.3.2 (4)
    16: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_create@GLIBC_2.2.5 (3)
    17: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_detach@GLIBC_2.2.5 (3)
    22: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_attr_init@GLIBC_2.2.5 (3)
    23: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_attr_getstacksize@GLIBC_2.2.5 (3)
    24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_attr_destroy@GLIBC_2.2.5 (3)
    27: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_sigmask@GLIBC_2.2.5 (3)

The cgo tool uses debug/elf.File.ImportedSymbols to iterate over the imported symbols in _cgo_.o and emit //go:cgo_import_dynamic pragmas, but this function specifically skips weak symbols.

Ping @ianlancetaylor @cherrymui for thoughts.

[ianlancetaylor]

The functions pthread_mutex_lock and pthread_mutex_unlock are STB_WEAK in /lib/x86_64_linux_gnu/libpthread.so.0, although they are STB_GLOBAL in /lib/x86_64-linux-gnu/libc.so.6. That must somehow be causing lld to emit them as STB_WEAK symbols, although GNU ld and gold act differently.

Unfortunately, elf.File.ImportedSymbols explicitly skips STB_WEAK symbols.

It seems to me that lld is acting incorrectly here. Since we link against libc.so.6, and the symbols are STB_GLOBAL there, lld should emit them as STB_GLOBAL, not STB_WEAK. I don't know if this has changed in lld-9. CC @rui314.

In the meantime I think we need to change cgo to call DynamicSymbols, and we need to add a new exported API to debug/elf to do a version of File.gnuVersion that returns the library and version for a dynamic symbol index.

[cherrymui]

I just built LLD tip from source and it still generates weak symbols for pthread_mutex_lock/unlock.

[aclements]

I have a fix for the bootstrap, though it appears this isn't the only problem with lld 8:

--- FAIL: TestScript (0.00s)
    --- FAIL: TestScript/vendor_complex (1.24s)
        script_test.go:191: 
            # smoke test for complex build configuration (1.240s)
            > go build -o complex.exe complex
            > [exec:gccgo] go build -compiler=gccgo -o complex.exe complex
            [stderr]
            # complex
            ld: error: undefined symbol: pthread_sigmask
            >>> referenced by generic-morestack.o:(__morestack_block_signals) in archive /usr/lib/gcc/x86_64-linux-gnu/7/libgcc.a
            
            ld: error: undefined symbol: pthread_sigmask
            >>> referenced by generic-morestack.o:(__morestack_unblock_signals) in archive /usr/lib/gcc/x86_64-linux-gnu/7/libgcc.a
            
            ld: error: undefined symbol: pthread_sigmask
            >>> referenced by generic-morestack.o:(__morestack_block_signals) in archive /usr/lib/gcc/x86_64-linux-gnu/7/libgcc.a
            
            ld: error: undefined symbol: pthread_sigmask
            >>> referenced by generic-morestack.o:(__morestack_unblock_signals) in archive /usr/lib/gcc/x86_64-linux-gnu/7/libgcc.a
            collect2: error: ld returned 1 exit status
            [exit status 2]
            FAIL: testdata/script/vendor_complex.txt:5: unexpected command failure
            
FAIL
FAIL	cmd/go	43.183s

(This happens with or without my fix, as long as I apply my fix to get past bootstrap.)

[aclements]

@ianlancetaylor, are there going to be other consequences of importing weak symbols in the cgo tool? I don't really understand the semantics here. Is the cgo tool just wrong to omit these right now?

[ianlancetaylor]

The error you show above suggests that somehow the version of gccgo you are using is not passing -lpthread to the linker. I don't think that has anything to do with any change you made to cgo.

[aclements]

It definitely doesn't have anything to do with my change to cgo because it happens if I revert my change.

[gopherbot]

Change https://golang.org/cl/184100 mentions this issue: cmd/cgo: accept weak dynamic imports

[gopherbot]

Change https://golang.org/cl/184099 mentions this issue: debug/elf: add version information to all dynamic symbols

[ianlancetaylor]

What version of gccgo do you have installed? What is go env GCCGO?

[aclements]

$ gccgo -v
Using built-in specs.
COLLECT_GCC=gccgo
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 7.3.0-18' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.3.0 (Debian 7.3.0-18)

$ go env GCCGO
/usr/bin/gccgo

[brooksmoses]

Ian, I believe that your "what the linker should do" analysis in the previous comment is a bit erroneous by being overcomplicated. The STB_GLOBAL symbols in libc.so.6 are a red herring.

If I have an object file with a STB_GLOBAL reference to "foo", and link it against a shared library that exposes a definition of foo, I should get a STB_GLOBAL reference in the resulting file -- regardless of whether the definition of foo is weak or strong. The weakness of the reference and the weakness of the definition are distinct things, and there's no reason they need to match.

Testing this with various linkers:

$ cat libweakfoo.c
int __attribute__((weak)) foo() {
  return 23;  // fnord
}
$ gcc libweakfoo.c -shared -o libweakfoo.so
$ readelf --dyn-syms libweakfoo.so | grep ' foo'
     6: 000000000000057a    11 FUNC    WEAK   DEFAULT   11 foo

$ cat callfoo.c
int foo();
int main() {
  return foo();
}
$ gcc callfoo.c -c -o callfoo.o
$ readelf -s callfoo.o | grep ' foo'
    10: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND foo

$ gcc callfoo.o -L. -lweakfoo -o callfoo -fuse-ld=bfd
$ readelf --dyn-syms callfoo | grep ' foo'
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND foo

$ gcc callfoo.o -L. -lweakfoo -o callfoo -fuse-ld=gold
$ readelf --dyn-syms callfoo | grep ' foo'
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND foo

(Apologies for the compiler switch here; I don't have my gcc set up to use lld.)
$ clang callfoo.o -L. -lweakfoo -o callfoo -fuse-ld=lld
$ readelf --dyn-syms callfoo | grep ' foo'
     6: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND foo

Looks like in this simple case, LLD (via Clang) is doing the right thing. So it must be something other than simply the presence of weak definitions in libpthread.o that's causing LLD to emit the weak references in _cgo_.o. (Or else this was fixed sometime between @cherrymui's tip-of-trunk LLD from 16 days ago, and mine from a few days more recently.)

FWIW, I also repeated this analysis adding an __attribute__((weak)) to the reference in callfoo.c, and determined that the "UND" symbol in the linked binary is weak regardless of whether I link with a library that has a strong definition or a weak definition, and regardless of which linker I use.

[brooksmoses]

Addendum: I checked things with a somewhat older build of LLD, to rule out the "this was fixed between @cherrymui's tip-of-trunk version and my slightly-newer one", and this still works the same way with the older LLD versions.

[cherrymui]

I just build tip LLD and it is the same.

$ ld.lld --version
LLD 9.0.0 (http://llvm.org/git/lld.git 5a41ae1ba61fd6c3e5bb15e8d04c8deb9f63079d) (compatible with GNU linkers)

$ CGO_LDFLAGS=-fuse-ld=lld go build -work -x -a runtime/cgo
...
TERM='dumb' gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o $WORK/b001/_cgo_.o $WORK/b001/_cgo_main.o $WORK/b001/_x001.o $WORK/b001/_x002.o $WORK/b001/_x003.o $WORK/b001/_x004.o $WORK/b001/_x005.o $WORK/b001/_x006.o $WORK/b001/_x007.o $WORK/b001/_x008.o $WORK/b001/_x009.o $WORK/b001/_x010.o $WORK/b001/_x011.o $WORK/b001/_x012.o -fuse-ld=lld -lpthread
...

The line above is how we invoke the linker.

The linker output contains weak references.

$ readelf --dyn-syms $WORK/b001/_cgo_.o | grep pthread_mutex
    11: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (3)
    13: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (3)

The input to the linker seems normal.

$ readelf -s /tmp/go-build382217838/b001/_x*.o | grep pthread_mutex
    22: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND pthread_mutex_lock
    24: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND pthread_mutex_unlock

[cherrymui]

Interestingly, if we don't pass -lpthread to gcc, it still works and actually generates strong symbols.

$ TERM='dumb' gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o $WORK/b001/_cgo_.o $WORK/b001/_cgo_main.o $WORK/b001/_x001.o $WORK/b001/_x002.o $WORK/b001/_x003.o $WORK/b001/_x004.o $WORK/b001/_x005.o $WORK/b001/_x006.o $WORK/b001/_x007.o $WORK/b001/_x008.o $WORK/b001/_x009.o $WORK/b001/_x010.o $WORK/b001/_x011.o $WORK/b001/_x012.o -fuse-ld=lld 
$ readelf --dyn-syms $WORK/b001/_cgo_.o | grep pthread_mutex
    11: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (3)
    13: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (3)

Note that gcc will still pass -lpthread to the linker even without -lpthread in the command line.

$ TERM='dumb' gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK/b001=/tmp/go-build -gno-record-gcc-switches -o $WORK/b001/_cgo_.o $WORK/b001/_cgo_main.o $WORK/b001/_x001.o $WORK/b001/_x002.o $WORK/b001/_x003.o $WORK/b001/_x004.o $WORK/b001/_x005.o $WORK/b001/_x006.o $WORK/b001/_x007.o $WORK/b001/_x008.o $WORK/b001/_x009.o $WORK/b001/_x010.o $WORK/b001/_x011.o $WORK/b001/_x012.o -fuse-ld=lld -v
...
 /usr/lib/gcc/x86_64-linux-gnu/7/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/7/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper -plugin-opt=-fresolution=/tmp/ccvdFN99.res -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lgcc_s --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -fuse-ld=lld -o /tmp/go-build549226297/b001/_cgo_.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o -L/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/7/../../.. /tmp/go-build549226297/b001/_cgo_main.o /tmp/go-build549226297/b001/_x001.o /tmp/go-build549226297/b001/_x002.o /tmp/go-build549226297/b001/_x003.o /tmp/go-build549226297/b001/_x004.o /tmp/go-build549226297/b001/_x005.o /tmp/go-build549226297/b001/_x006.o /tmp/go-build549226297/b001/_x007.o /tmp/go-build549226297/b001/_x008.o /tmp/go-build549226297/b001/_x009.o /tmp/go-build549226297/b001/_x010.o /tmp/go-build549226297/b001/_x011.o /tmp/go-build549226297/b001/_x012.o -lgcc --push-state --as-needed -lgcc_s --pop-state -lpthread -lc -lgcc --push-state --as-needed -lgcc_s --pop-state /usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o
...

(there is -lpthread near the end)

[ianlancetaylor]

@brooksmoses What you say makes perfect sense.

That said, here is how to recreate the problem.

> cat foo.c
#include <pthread.h>

static pthread_mutex_t mu = PTHREAD_MUTEX_INITIALIZER;

void f1(void) {
	pthread_mutex_lock(&mu);
}

void f2(void) {
	pthread_mutex_unlock(&mu);
}

int main() { return 0; }
> gcc -g -c -fPIC -pthread foo.c
> gcc -o x foo.o -lpthread # Using GNU ld
> readelf -s --wide x | grep 'pthread_mutex.*lock'
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (3)
     7: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (3)
    63: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_unlock@@GLIBC_2.2.5
    72: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND pthread_mutex_lock@@GLIBC_2.2.5
> gcc -o x foo.o -lpthread # Using lld
> readelf -s --wide x | grep 'pthread_mutex.*lock'
     6: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (3)
     7: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (3)
    35: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_lock
    37: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_unlock

I tested this using an lld built at Google, not one I built myself.

[ianlancetaylor]

@aclements I have no explanation for what you are seeing with gccgo. It does work for me. I guess it's unrelated to this issue, though, so should probably be dealt with elsewhere if at all. It might help to see the go build -x output.

[brooksmoses]

Thanks, @ianlancetaylor -- having a nice minimal reproducer like that is very helpful, and I'll dig into that tomorrow and see if I can boil this down to a clean easily-fixable LLD bug report.

The difference in symbol-versioning there is also weird (and perhaps significant). I wonder if that's also related to getting the symbols from different places -- iirc, they're versioned in one library and not the other.

[brooksmoses]

Quick confirmation: @ianlancetaylor, I can reproduce what you're seeing with my Clang-based setup -- including the fact that, without -lpthread, the LLD link gets strong symbols.

[brooksmoses]

Some more reduction, using @ianlancetaylor's foo.c above. (I'm omitting paths and -L options for clarity.)

ld.lld -o foo foo.o -lpthread -lgcc_s links and produces the weak references.
ld.lld -o foo foo.o -lpthread links and produces the strong references.

(Both produce a warning about cannot find entry symbol _start; defaulting to 0x201000 because I've simplified the link line, but that's expected and irrelevant to this issue; we're not running these!)

[brooksmoses]

And, as one might start to expect from this data, libgcc_s.so.1 has these definitions:

readelf --dyn-sym libgcc_s.so.1 | grep pthread_mutex
   102: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_unlock@GLIBC_2.2.5 (13)
   165: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND pthread_mutex_lock@GLIBC_2.2.5 (13)

[brooksmoses]

A complete, simple reproducer:

$ cat foo.c
// A program with a strong reference to bar()
int bar();
int main() { return bar(); }

$ cat bar.c
// A strong definition of bar().
int bar() { return 23; }

$ cat baz.c
// An unused function with a weak reference to bar()
int __attribute__((weak)) bar();
int baz() { return bar ? bar() : 0; }

$ gcc foo.c -c -o foo.o
$ gcc bar.c -shared -o libbar.so
$ gcc baz.c -shared -o libbaz.so

First, to confirm foo.o has a strong reference:

$ readelf -s foo.o | grep bar
    10: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND bar

If we link without libbaz.so, we retain a strong reference, as expected:

$ ld.lld foo.o libbar.so -o foo1
ld.lld: warning: cannot find entry symbol _start; defaulting to 0x201000
$ readelf --dyn-sym foo1 | grep bar
     1: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND bar

However, if we append libbaz.so to the end of the command line, it's a weak reference:

$ ld.lld foo.o libbar.so libbaz.so -o foo1
ld.lld: warning: cannot find entry symbol _start; defaulting to 0x201000
$ readelf --dyn-sym foo1 | grep bar
     1: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND bar

That seems to pretty clearly be an LLD bug. The program still has code that was written to require a strong reference to bar, and appending something with a weak reference shouldn't change that.

Interestingly, if we invert the order, making it libbaz.so libbar.so, we get a strong reference again.

I'll go file this in the LLVM bugtracker.

[brooksmoses]

This is now filed as https://bugs.llvm.org/show_bug.cgi?id=42442.

[ianlancetaylor]

Thanks @brooksmoses

[MaskRay]

Thanks to Ian and Brooks for analysis. I am really sad to see pthread weak undef in libgcc_s.so.1... Anyway, proposed fix of the bug: https://reviews.llvm.org/D63974

% clang -fuse-ld=lld -o x foo.o -pthread -Wl,-y,pthread_mutex_lock
foo.o: reference to pthread_mutex_lock
/usr/lib/gcc/x86_64-linux-gnu/8.0.1/libgcc_s.so.1: reference to pthread_mutex_lock
/usr/lib/gcc/x86_64-linux-gnu/8.0.1/../../../x86_64-linux-gnu/libpthread.so: shared definition of pthread_mutex_lock

% clang -fuse-ld=lld -o x foo.o -lpthread -Wl,-y,pthread_mutex_lock
foo.o: reference to pthread_mutex_lock
/usr/lib/gcc/x86_64-linux-gnu/8.0.1/../../../x86_64-linux-gnu/libpthread.so: shared definition of pthread_mutex_lock
/usr/lib/gcc/x86_64-linux-gnu/8.0.1/libgcc_s.so.1: reference to pthread_mutex_lock
### Without D63974, the weak undef in libgcc_s.so.1 sets the result binding to STB_WEAK.

At runtime, setting the binding of a symbol from STB_GLOBAL to STB_WEAK dosn't cause issues.
You just miss an ld.so error: symbol lookup error: ... undefined symbol: (or Error relocating %s: %s: symbol not found) if the symbols happens to be unavailable. (There is also LD_DYNAMIC_WEAK in ancient glibc's). I will be surprised if a Go tool does something special if the binding is accidentally changd from STB_GLOBAL to STB_WEAK.

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[aclements]

Whoops, we have the info we need, but forgot to remove the label. Reopening.

[gopherbot]

Change https://golang.org/cl/189458 mentions this issue: api/go1.13: add debug/elf.Symbol fields added in CL 184099