-
Notifications
You must be signed in to change notification settings - Fork 17.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: cmd/go: build tag in filename suffix for matching of syso files #42477
Comments
cc @eliasnaur |
This seems like a lot of work to support .syso files. Why do you need them? Is it to avoid the need for SDKs and system compilers/linkers? If so, it seems to me the effort is better spent on that. Shameless plug: #38917. |
Kind of an odd idea, but could fat .syso files solve the problem? Not sure whether those are even a thing, but if it were possible then that would allow us to transfer the problem to the loader. |
A lot of work for a contribution for a CL to address this, or a lot of work for a 3rd party developer who might make use of this?
We use syso files for two purposes:
|
That's actually a great suggestion. I just tried it and unfortunately hit this with
Clang's linker can handle "fat objects", but each fat object can only host dissimilar architectures, even if each component is built for a different platform. This becomes pretty clear when you look at the Mach-O fat header format: each slice is keyed on a CPU and sub-CPU architecture magic, and has no reference to a platform magic. That means we cannot, for example, have a fat object that targets both iOS (arm64 device) and the iOS-simulator (arm64 Apple Silicon Mac); or iOS (arm64 device) and tvOS (arm64 device) and/or the respective simulators (arm64 Apple Silicon Mac). |
Currently there isn't anything stop you from doing this. But this is not really "supported". |
@cherrymui, you wrote something similar back in July last year. As I replied back then, the Go Wiki documents this pattern since at least 2014 [1]. I spent a lot of time to contribute a CL to "support" that late last year, and I'm offering to support it again here. Separate to the motivation for the proposal, I'd be grateful for any comments or support for the concrete use of #tags in a syso filename suffix. With some blessing, or at least no objections, I'd be happy to start working on a CL. [1] See wiki commit |
I know the Wiki page mentions it, as a "trick". But I wouldn't say it is "supported". I agree that we probably don't want to intentionally break it, and probably will keep it working if it doesn't take much effort. But I don't think this kind of use is a good motivation for new proposals. |
I'd prefer to avoid an off-topic discussion about the semantics of what "supported" means. This proposal is rather minor: it adds an optional feature, limited in scope to just syso files, so that it keeps an existing compiler feature working for new architectures that are being adopted more broadly (i.e. #38485). The community effort here will mainly be review of a CL I am volunteering to develop. I don't expect the CL to be overly complex. Are there any objections to the concrete tag naming scheme for the filename suffix? |
I think it might be a good idea to have some mechanism to match build tags for syso files (although I don't necessarily agree with the motivation). I think the proposal needs to be a little clearer about the matching semantics. Is the tag after the If we do this, I think supporting multiple tags is fine. |
I would propose:
I'm happy to update the proposal to make this more clear. |
This seems very advanced and specific, and I'm a bit reluctant to make syso and file selection far more complicated just to support this rare case. It seems like you can already make this work today. Suppose you have a package p, and you put two (or more) files like:
and:
and so on. Then the p/iphone_syso being imported can be an empty package (with a 'package iphone_syso' and nothing else in a single Go file) along with a syso. On iPhone, p/tv_syso won't be imported and the syso won't go to the linker. But p/iphone_syso will, and that syso will go to the linker? Thoughts? |
Yeah, this is what I suggested in #38485 (comment) |
Thank you for reviewing the proposal, @rsc. My outline could have been more clear on your workaround, also suggested by @cherrymui: it was referred to the "sub-package" approach without further detail. I've updated the proposal at the top of this thread, and also included more clarity on the treatment of multiple build tags. Compared to the workaround available today, both @cherrymui and I agree that it is not pretty. As outlined in the update, it requires two additional Go source files and one new sub-package directory for every unique syso file. I have several packages supporting macOS (amd64 + soon arm64), iOS (arm64), iOS-simulator (amd64 + soon arm64), Android (arm64), Linux (amd64, arm64), and Windows (amd64, 386). I hope to support tvOS and the tvOS simulator in time. For each platform object, I have Release and Debug builds. The macOS, iOS (tvOS), iOS(tvOS)-simulator, Android, and Linux syso files can't be uniquely selected for a build without the workaround which makes the package a mess, while the Windows syso files remain in the package root. (This happens because GOOS uses non-unique aliases for macOS, iOS/tvOS, Android, and Linux.) This proposal would unify and simplify that layout significantly. I appreciate the flexibility that I'm about to start looking at a CL for this, and I expect the change to be small. I hope you're able to keep an open mind. |
Change https://golang.org/cl/269280 mentions this issue: |
Using subpackages is not pretty, but using build tags in the file name starting with "#" is also not pretty. Subpackages, while verbose, do not require any new features and are thus clear to most Go programmers. Build tags starting with "#" will always be a very obscure feature that seems likely to trip people up. Given that very very few people will ever want to use this feature, and given that it can be supported without adding any new features, I personally think there is a strong argument for not adding any new features here. |
The lesser of the two evils, and a much better developer experience, is surely just naming one file The filename
I think more people would use this functionality if:
The feature is rather trivial: visible only in a syso filename and nowhere else, and the implementation is also minimal as you can see in the simple CL I submitted 1/2-hour ago. Thanks for taking the time to read the proposal, @ianlancetaylor. I hope you'll reconsider with the above arguments in mind, and have a quick look at the associated CL. |
Today, the I think it would be a mistake to add a separate naming convention that applies only to On the flip-side, users are already confused enough about the implicit build constraints for ordinary Go source files. If this new naming scheme did not apply only to Given how rarely |
Thanks for chiming in. I agree with this, and the proposal doesn't change the
If consistency is important to you, I'd be happy to see the proposed
I agree that developers can be easily confused about the explicit But the implicit build constraints in the filename are straightforward because they are so rigid and simplistic: you just summarized it with a simple string pattern in your post! Can you point out instances where developers get confused with this?
The prospect of living with this messy workaround for "a few years" is unappealing. We already have syso files in over 15 packages and are just getting started. Based on the discussion so far, what interest (if any!) is there for:
|
I wonder whether that linker error is the only problem? If so, the Go linker may be able to help by only presenting object files to the system linker that have matching LC_BUILD_VERSION load commands. The linker already tries to select an appropriate version, see for example hostobjMachoPlatform. |
@jpap It seems to me that you are arguing in favor of a change that 1) very few people will want to use; 2) you can already do today in a different, albeit more awkward, way. A good argument here would be pointing to several different popular packages, by different people, that would use this feature if it were available. The argument that more people are going to want to use this feature in the future doesn't seem particularly convincing to me. It's particularly unconvincing if the reason that people want to use this feature is to bypass cgo safety in the name of speed. If that is the problem that people are trying to address, then we should address that, not encourage increasingly complex use of a workaround. |
Yes -- the Go linker could filter all of the syso files with In the code you've cited, it takes the first object file involved in the build, extracts the platform, and if it is macOS, inserts the required Even if you were to use some magic in the Go linker to solve the above; you still have the problem with Android and Linux. When you build for Android, the The fundamental problem here is that Elias, did you have have an opinion on the proposal itself? I appreciate the thoughts going into other workarounds, but would love to know if someone else who is knee-deep in non-Go code and platform interop can see the benefits proposed here. |
Thanks for painting a path forward. I can't help being a little disappointed that it isn't to discuss the technical merits of the proposal, or some of the proposed alternatives, but to judge it solely on its apparent popularity, especially given how minor this feature is and how small the CL turned out to be. :) I agree that syso usage is not an everyday use-case for the majority of Go users. That is because of the demographic: the Go Survey clearly shows that the vast majority of Go use-cases are API/RPC services and CLIs. Those using Go for Desktop and Mobile apps are in the tiny minority, but that is where syso shines. This situation is chicken-and-egg: Go developers don't write GUI apps because Go lacks great UI packages. That will change as one of more Go UI projects matures. I feel it is a bit disingenuous to ask for references from "several different popular packages, by different people" when the number of people working in this space is miniscule. It's great to have Elias in this thread, and I've put it to him in my last post. His Gio project may not benefit as much from improved syso support, as his goal is to avoid interfacing with the (UI-component part of the) platform as much as possible, by rendering a completely custom UI into a blank canvas. When you write GUI apps that interface into platform SDKs and platform UI components, you tend to rely on cgo much more frequently, and leveraging syso leads to a much better developer experience.
The stain of "let's use syso to bypass cgo" is unwarranted here. Syso files are the only way to host a "Go gettable" package that uses or wraps a nontrivial non-Go library that gets automatically and statically compiled into the final Go executable while using cgo as the glue. I've already linked to a public example in a previous post. I think this approach isn't popular because it isn't well documented. A lot of projects that provide cgo wrappers for a non-Go library typically assume that library is pre-installed on the system. It is a much better developer and end-user experience, and more "Go like", to have the library statically linked to the executable. I have successfully used the statically compiled syso approach for interop with many 3rd party non-Go libraries:
I have also used the approach to statically link many 1st party non-Go libraries that interface with platform SDKs. I am not going to list those here. All but one of these packages are currently private; that might change in the future, but the work being done here is commercially focused. All of these packages use cgo. |
I want to clarify that this is not a strong argument. Every feature must be documented, maintained, and tested. Small features are not better from a long-term maintenance perspective. They are worse, because they are less used, less tested, and less understood, but they must be fixed when they break. Thanks for the comments about cgo. I wonder if this is an argument for having better interaction between |
I agree on the importance of documentation, testing, and maintenance in a large, long term project. One could argue that my CL included the first two things: it included tests, and added both godoc and Broad generalizations are helpful but there are always special cases. What you say about small features may be true in general, but in this case, having never previously looked at the relevant code, I was able to understand and produce the CL, including testing, within an hour. While some of the Go codebase is very complex for newcomers, perhaps some parts brittle to breakage when unrelated changes are made, after working on it I feel this CL is far from both of these things. :)
Based on my experience, I would recommend bolstering syso support or something like that, where you are ingesting precompiled objects or archives. The reason for this is that for many nontrivial non-Go projects, the build process is complex and sometimes bespoke. They might use CMake, Autoconf, GYP and Ninja, etc. On platforms like Android, you not only have the C-JNI glue, but corresponding Java sources as well. In all of these cases, I am not suggesting syso objects be shipped in isolation; in my projects I also ship a Makefile or script to fetch the non-Go project, cross compile it where possible, and then (re)produce the objects that are checked into the repository with Git-LFS. One of the great things about Go is not only the ability to target across platforms, but also the ability to cross-compile easily. Doing so with cgo is possible, but requires more care. Doing so with cgo + a non-Go project build system is even harder. Cross-compiling some of the 3rd party libraries I mentioned in my last post was not always an easy task. In some sense that is what Elias is looking for in #38917 -- if you are linking prebuilt objects, there's no need to worry about SDK headers that you are licensed to only use on a certain platform. ;) It is much easier to compartmentalize 3rd party non-Go code than work directly with its source files, especially when that code was not originally intended for consumption by Go. In recent months I have also spent time working out how to pragmatically support cross-compilation of cross-platform apps that make extensive use of a FFI, for example, calling Objective-C APIs on Apple's platforms. I think I'm converging on something that works really well, but some of it hinges on prebuilt objects for calls into platform SDKs to make cross-compilation a breeze. Syso files are at the core of that, which is another reason I am so keen to see this proposal through now and not in "a few years". (Again, this is all using cgo as the glue.) I have a separate viable path to do all of that without this proposal, while still maintaining a good developer experience, but unfortunately that path involves much more work on my end and a departure from the standard Go project structure. It would require a new build tool that sits above If there's any interest in discussing some of my earlier suggestions, I'd be all for it. Otherwise I fear we're not making the best use of our time here. For that, I apologize, no hard feelings, and thanks for the discussion. I can't wait for you to see what I've been working on. :) |
Build tags in filenames is a blunt instrument. I believe there are lighter weight options available. Off the top of my head:
It's true that Android support for the above scheme seems harder. Without knowing more about your particular setup, I don't have any recommendations other than an explicit linker flag.
I would love easy linking of static libraries. I'll probably need it for integrating something like Harfbuzz in Gio. However, this proposal adds public facing machinery that's not clear is necessary, in particular when there's a (messy) workaround. For something as obscure as syso files, I believe there is value in searching for better solutions before resorting to build tags in filenames. There's ample time; the next freeze is 5 months away. |
Appreciate your reply, Elias. Some comments inline below.
You could invoke the compiler to produce a dummy object, then extract the platform from it. The extra round-trip would also elongate build times.
That is possible, however how can you deal with Debug vs. Release object selection? Build tags offer a lot more flexibility because their meaning is user-defined, and not hard-coded into a flag.
What information would you like to know about my setup? Based on an explicit linker flag, and the inability to discern Android ELF from "Desktop Linux" ELF, how can we filter syso files in this case?
I look forward to your suggestions, and any feedback you might have on my other alternative proposals. |
Having separate debug vs release builds is unusual in Go, so I'm not sure how far the Go tools should be stretched to accommodate that variant axis. The ugly workaround seems good enough here.
"How to filter syso files" smells like an XY problem to me. Why do you need different syso files for Android vs Linux? Can the static libraries be built in a way they don't refer to Android-specific APIs?
If we decide filtering syso files is required, then I like your magic comments better. Perhaps add them to
|
Go doesn't suffer from the debug-vs-release dichotomy because it doesn't perform extensive optimizations, favoring faster build times (and a simpler compiler?). There's also good reason for keeping debugging information in the binary for panic stack traces at runtime. These things are fine if you're deploying a backend service on your own/rented network; but for user interactive apps that run on foreign machines it's a bit different:
I am really looking forward to gollvm maturing so we can get the best of both worlds: fast compilation during development with
We need native code that uses the C(++) ABI on:
That native code is most easily written in C(++), interfaced to Go via cgo (+ SWIG, or an If you use { I see X == "need native code that hits platform-specific APIs" and Y == "need syso".
I would caution against associating syso with cgo here: they are independent features, even though they might work well together under many circumstances. |
FWIW, I appreciate that it's a little more work in this use case, but sometimes that's preferable to complicating the general case for something that most people will never see. It becomes one more thing people have to understand when they do see it. If you know about regular build tags and subdirectories (which most people do), then you know what you're looking at when you see the subdirectory solution. On the other hand, if you see a file named |
Based on the discussion above, this seems like a likely decline. |
Given the current state of this proposal, I've instead implemented a custom build tool with a nonstandard Go project layout in favor of a better DX over the proposed workaround. Thank you for the consideration and detailed discussion by all. |
A rather unfortunate side effect of using multiple packages is that you'll need to instruct the linker to look up symbols at runtime. So, this program that previously wouldn't compile now crashes: https://play.golang.org/p/zLi2iNfwiZO This is a problem I've run into when creating .syso files for x86-64 Android and Linux. Since the @rsc I'd like to reopen this issue (or make another if you'd prefer, I'm not a fan of the "#" syntax). .syso files are a little esoteric, but incredibly useful. |
As far as I can tell you can address that issue using the approach described at #42477 (comment). If not, why not? |
@ianlancetaylor yes, multiple packages works. But using a tennis racket as a canoe paddle also works. :) In particular, cmd/cgo does not pull in other packages, so linking fails unless you instruct the linker to ignore undefined symbols. This could cause the program to explode at runtime instead of compile time. (See the playground link above.) Or, you have to manually link against the .syso files in #cgo directives, which mostly defeats the purpose of using .syso files. |
I think we discussed all the tradeoffs above, and came to a decision. If multiple packages works, then I don't see a need to reconsider. I understand that this is more work, but it is possible, and it's for an unusual case. Thanks. |
Motivation
We have several Go packages that targets multiple platforms, including macOS, iOS, tvOS, iOS-simulator, tvOS-simulator, windows, linux, etc. The package has one or more syso file dependencies.
Syso files can be selected for a build by a GOOS/GOARCH filename suffix but not build tags, unlike Go source files that use
// +build
or//go:build
lines. If we ignore tvOS for the moment, some time ago we could build a syso file fordarwin/amd64
and include that in both macOS and iOS-simulator builds just fine. With recent Xcode 12, that approach no longer works:The situation worsens when we support tvOS (+ the tvOS-simulator), and the new ARM-based Apple Silicon Macs because we can no longer use a GOOS/GOARCH filename build constraint to uniquely select the appropriate syso variant.
Workaround
In #38485 @cherrymui suggested using build-tags that selectively import sub-packages containing the associated syso file(s). That can work, but is messy as outlined next.
In more detail, the idea is to write a Go source file with a
// +build
or//go:build
line that is conditional on one or more tags of interest. The conditional build of said source file then imports another sub-package that contains the syso file to use in the build.For every syso file, it would require one additional sub-package and two additional Go source files. (One for with the conditional build line, and another to define the otherwise empty package containing the syso file.) This balloons when there might be a Debug and Release version of each syso file, requiring
4n
Go source files and2n
sub-package directories forn
platforms.A better approach would be to support matching build tags in syso filenames. The syso files can be placed directly into a package, for a better developer experience.
Proposal
This proposal is to extend filename build constraints to include build tags with a
_#tag
pattern in the filename suffix, before any GOOS and/or GOARCH patterns.Multiple tags can be supported with
_#tag1_#tag2_#tag3
, where an AND matching condition (tag1
&&tag2
&&tag3
) is used for selection.Where no tag suffix items are present, the behavior is the same as it is now.
Example
We can use the user-defined build tags
macosx
(macOS),iphoneos
(iOS),iphonesimulator
(iOS-simulator),appletvos
(tvOS),appletvsimulator
(tvOS-simulator),androidos
(Android),linuxos
(Linux) on Macs, iOS/tvOS devices, Linux, Android, and Windows as follows:lib_#macosx_amd64.syso
for macOS on an Intel Maclib_#macosx_arm64.syso
for macOS on an Apple Silicon Maclib_#iphonesimulator_amd64.syso
for iOS-simulator on an Intel Maclib_#iphonesimulator_arm64.syso
for iOS-simulator on an Apple Silicon Maclib_#appletvsimulator_amd64.syso
for tvOS-simulator on an Intel Maclib_#appletvsimulator_arm64.syso
for tvOS-simulator on an Apple Silicon Maclib_#iphoneos.syso
for an iOS device (note: no GOARCH)lib_#appletvos.syso
for an Apple TV device (note: no GOARCH)lib_#linuxos_amd64.syso
for Linux desktop 64-bit Intellib_#linuxos_arm64.syso
for Linux desktop 64-bit ARMlib_#androidos_arm64.syso
for Android 64-bit ARMlib_windows_amd64.syso
for Windows 64-bit IntelNotes
While this proposal could work for Go source files and syso files alike, the current proposal is for syso files only since, unlike Go source files, there is no alternative other than the workaround above.
I have tested the use of the "#" character in filenames on macOS, Windows, and Linux filesystems, and didn't see a problem with using that character for this purpose.
The text was updated successfully, but these errors were encountered: