cmd/go/internal/work, cmd/cgo: duplicate libobjc library with 2 or more cgo packages with Objective-C

[dmitshur]

Go version

go version go1.22.3 darwin/arm64

Output of go env in your module/workspace

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/gopher/Library/Caches/go-build'
GOENV='/Users/gopher/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/gopher/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/gopher/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.3'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/tmp.ElsgkzNn19/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/_0/h0671fcn4rgb5pn9c745dx2h0000gn/T/go-build1736924935=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Consider a minimal program that involves at least two cgo packages, both of which with Objective-C code:

cd $(mktemp -d)
go mod init test
mkdir p1 p2
echo 'package p1; import "C"' > p1/p1.go
echo 'package p2; import "C"' > p2/p2.go
touch p1/p1.m p2/p2.m
echo 'package main; import (_ "test/p1"; _ "test/p2"); func main() {}' > main.go

I tried to build it on macOS Sonoma 14.5 with Xcode 15.4 without custom flags:

go build

What did you see happen?

A warning is printed:

# test
ld: warning: ignoring duplicate libraries: '-lobjc'

What did you expect to see?

No warning printed.

---

I believe this is a minified reproduce for an issue like fyne-io/fyne#4502.

The duplicate library outcome for 2+ cgo packages with Objective-C has likely been happening for a while, but what's changed recently is that the new linker in Xcode 15 causes the duplicate library to be printed as a warning unless -ldflags=-extldflags=-Wl,-no_warn_duplicate_libraries is used to disable duplicate library warnings. See [1], [2].

[1]: issue #61229 ("cmd/link: issues with Apple's new linker in Xcode 15 beta")
[2]: https://indiestack.com/2023/10/xcode-15-duplicate-library-linker-warnings/

CC @matloob, @cherrymui.

[ianlancetaylor]

In the general case we can't drop duplicate libraries, because traditional Unix linkers search the library at a specific point on the command line. It's normal for people to write things like -lgcc -lc -lgcc, which would perhaps generate a warning with this new linker. So while it would be fine to change -lobjc -lobjc to just -lobjc, I'm inclined to pass the option to disable the warning.

[matloob]

Okay, sounds like on darwin hosts we should pass in the flags to disable the warnings. Is this something that would need to be done by the next release?

[dmitshur]

Note that if the linker in older Xcode 14 doesn't support the same flag to disable the warning, such that passing it unconditionally causes an error (as mentioned in [2]), then that needs to be taken into account.

This is a warning that doesn't prevent successful compilation of programs, and it's been around since last year (Xcode 15 was publicly released in September 2023). I think this would be good to fix when possible, since go build printing warnings on valid Go programs isn't a good experience, but it doesn't need to block the next release.

[cherrymui]

We can pass the flag in the Go linker to the C linker only when the flag is supported.

That said, as -lobjc is passed from cmd/go, can it just pass it once? That is, only add -lobjc if not already added.

[matloob]

Is that the preferred solution? To just pass in -lobjc once?

[cherrymui]

Personally I would prefer passing -lobjc just once, for a more targeted fix, if that isn't too complicated.

The duplicate libraries warning doesn't seem to be really helpful anyway. But as the Apple linker decides to emit it, if we don't have a strong reason to suppress it, we may well just keep it. If there is a reason that it is hard for us to avoid passing a -l flag multiple times, I'd be also happy to just suppress it.

[matloob]

If I'm understanding this correctly, this seems like something we'd have to fix in cmd/link, right?

My understanding is that the go command passes in -lobjc to the ldflags flag when invoking the cgo command. cgo in turn places its ldflags into a //go:cgo_ldflag directives in a generated go file. They in turn end up in the object files and are collected by the linker to pass to the host linker. To fix this in the go command we'd essentially have to not pass in the ldflag to each of the cgo invocations but just pass it in once at the end to the linker? I'm not too familiar with the guts of cgo but leaving that ldflag out seems wrong to me? So it seems the correct thing to do is for the linker would to see if it's about to pass in -lobjc more than once and remove the duplicates when it's calling the host linker?

[cherrymui]

My understanding is that the go command passes in -lobjc to the ldflags flag when invoking the cgo command. cgo in turn places its ldflags into a //go:cgo_ldflag directives in a generated go file.

Instead of writing a cgo_ldflag to all packages in the linker, can the go command just pass the flag as -extldflag to cmd/link? In general, if a flag is specified in CGO_LDFLAGS, I think we should pass it once for the build, instead of once per each cgo-using package.

As @ianlancetaylor mentioned above, generally it is not always okay to dedup C linker flags. But for the case of -l flags on macOS, it is okay (Mach-O linking uses a different model). But, if the user really wants to pass a flag several times, I'm not sure we want to stop them from doing that. For the ones that we pass, I think it would be better we pass it only once to begin with, instead of dedup'ing.