go: add basic support for cgo #16413

tdyas · 2022-08-04T07:27:36Z

Add support for packages that use cgo, which allows interfacing with C code from Go.

[ci skip-rust]

[ci skip-build-wheels]

tdyas · 2022-08-14T04:19:59Z

This should be complete for basic cgo support. Parts of it should probably be extracted into separate PRs: for example, the inclusion of resource targets into packages regardless of whether resource embedding is in effect.

tdyas · 2022-08-30T03:58:39Z

Resource files in packages extracted to #16688.

Extract the package name returned by the package analyzer and make it available to Go rules. This will be needed for the Cgo support to be added by #16413. [ci skip-rust]

…g) (#16688) Support placing resource files into packages just as `.go` files are put into packages (and not just embedding them in the compiled package via `//go:embed` directives). This is necessary to allow placing `.c` files into Go packages for the Cgo support to be added by #16413. [ci skip-rust]

tdyas · 2022-08-31T22:30:50Z

GoSdkProcess changes extracted to #16722.

Add `replace_sandbox_root_in_args` field to `GoSdkProcess` to teach the Go backend how to replace the string `__PANTS_SANDBOX_ROOT__` in args passed into `GoSdkProcess` with the absolute path to the sandbox root. This will be used by the forthcoming cgo support to be added by #16413. [ci skip-rust] [ci skip-build-wheels]

Fix a comment and a analysis JSON extraction nit. Extracted from cgo changes in #16413. [ci skip-build-wheels]

sureshjoshi · 2022-09-02T12:05:57Z

src/python/pants/backend/go/subsystems/golang.py

+        help="Name of the tool to use to compile fortran code included via CGo in a Go package.",
+    )
+
+    cgo_default_cflags = StrListOption(


Bikeshed: In other Subsystems, we don't tend to use the term "default" in the option name. Is this intended to indicate something specific to cgo?

With go, it does not use the -g -O2 default for cflags if CGO_CFLAGS is set. But for Pants, we won't be looking at CGO_CFLAGS so I agree makes more sense to just call this cgo_cflags (and similar change to the other options).

sureshjoshi · 2022-09-02T12:07:13Z

src/python/pants/backend/go/subsystems/golang.py

+        help="Default options used for FFLAGS when compiling cgo code. Equivalent to CGO_FFLAGS passed to `go`.",
+    )
+
+    cgo_default_ldflags = StrListOption(


Using the term ldflags here is done intentionally to line up with CGO's args, correct?

In lieu of something like link_flags or link_options - this is to be clear that this option directly passes into CGO_LDFLAGS?

Yes, I chose ldflags to match with its use with CGO_LDFLAGS. I assume go developers will be familiar with the go terminology here.

For reference, Bazel rules_go naming for fields on targets related to compile/link options:

copts == CGO_CFLAGS

cppopts == CGO_CPPFLAGS

cxxopts == CGO_CXXFLAGS

clinkopts == CGO_LDFLAGS

Thus there is some precedent in not using cflags / ldflags naming. Although I'm not particularly in favor of clinkopts as any better than ldflags. Going verbose here and having it be similar to the cc backend's naming choices could lead to a more consistent experience for Pants users.

So I might consider renaming the options. Especially since I would want to add fields on the go_binary target in a follow-on PR to allow configuring these options.

Tangentially, I note that the cc backend is using "cpp" to refer to C++ while Cgo uses "cpp" to refer to the C/C++ preprocessor and "cxx" to refer to C++. (Go will prepend the "cpp" flags to both "cflags" and "cxxflags.")

@sureshjoshi : Thoughts on cpp versus cxx naming?

I personally rather dislike how Bazel names options (in the CC backend especially - copts, hdrs, etc). Using word short-forms, no spacing, ugh - feels very 90's or 00's coding.

Anyways, I agree that keeping cgo in line with cc will be useful.

I'm finishing up some unit tests, then I'll push a preliminary version for a biiiiig bikeshed on naming. I'm using CPP, only because what's what I've always used for CPlusPlus.

I'm guessing Bazel is using more of a Makefile convention (https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html).

CMake uses CXX as well - https://cmake.org/cmake/help/latest/prop_tgt/CXX_STANDARD.html

I've got no opposition to changing and aligning with other tooling, I ran with CPP just so I wouldn't waste time on naming when there was a whole backend to write, and the CXX vs CPP decision would be bigger than just my opinion anyways.

In general, verbosely following existing conventions are my preference - plain English reading where possible.

🤦🏽
Completely forgot that gcc uses CPP as C pre-processor in docs.

I almost never touch those options directly, so I don't even think about it anymore. Probably a good reason to use cxx and never use cpp... Even though all my C++ files are named .cpp

sureshjoshi · 2022-09-02T12:11:19Z

src/python/pants/backend/go/subsystems/golang.py

+    )
+
+    cgo_default_ldflags = StrListOption(
+        default=lambda _: list(_DEFAULT_COMPILER_FLAGS),


Is this a correct default?

I believe so given the go source here: https://github.com/golang/go/blob/202b7e7e76128f524142ac7d328fe8458a096dbf/src/cmd/go/internal/work/exec.go#L2724

I would like a second opinion though on the defaults for all of those variables being -g -O2.

Ah, okay, I'm totally fine passing through the cgo defaults following my principle-of-least-surprise methodology.

I was a bit surprised at -g being passed through to ld as I thought ld ignores it by default, but I guess it's harmless enough.

Also, my opinion is predominantly embedded, where I almost exclusively use -Og and -Ofast, but I wouldn't propose those here.

[ci skip-build-wheels]

[ci skip-rust] [ci skip-build-wheels]

benjyw

This is a big deal, really nice! One thing I was unclear on is why we need to replicate so much logic from exec.go rather than invoke that logic via the go tool?

src/python/pants/backend/go/goals/debug_goals.py

src/python/pants/backend/go/util_rules/build_pkg.py

src/python/pants/backend/go/util_rules/cgo.py

tdyas · 2022-09-12T13:11:37Z

One thing I was unclear on is why we need to replicate so much logic from exec.go rather than invoke that logic via the go tool?

Because this logic is internal to go build and is not exposed. Where it is exposed by go tool cgo, this PR already invokes it.

Eric-Arellano

Thank you!

src/python/pants/backend/go/goals/debug_goals.py

src/python/pants/backend/go/subsystems/golang.py

src/python/pants/backend/go/goals/debug_goals.py

src/python/pants/backend/go/util_rules/cgo_test.py

src/python/pants/backend/go/util_rules/cgo.py

Eric-Arellano · 2022-09-12T17:52:37Z

add basic support

As opposed to what? That is, what is still missing for full Cgo support? Should those be tracked in issues?

[ci skip-rust] [ci skip-build-wheels]

tdyas · 2022-09-12T19:29:34Z

As opposed to what? That is, what is still missing for full Cgo support? Should those be tracked in issues?

Some stuff is missing (like the "syso" support) plus I haven't ported the extensive go integration tests for cgo. There is a unit test but it is for very basic C usage, and does not include stuff like testing linking to libraries.

This PR is intended as a base for that future work and allows the future work to consist of much smaller changes, which should be easier to review. Will open issues as needed.

[ci skip-rust] [ci skip-build-wheels]

tdyas added category:new feature backend: Go Go backend-related issues labels Aug 4, 2022

tdyas force-pushed the golang_cgo_support branch 2 times, most recently from bdca274 to 86a08f6 Compare August 8, 2022 06:55

tdyas changed the title ~~[WIP] go: add basic support for cgo~~ go: add basic support for cgo Aug 8, 2022

tdyas requested review from Eric-Arellano and stuhood August 8, 2022 06:56

tdyas marked this pull request as ready for review August 8, 2022 06:56

tdyas force-pushed the golang_cgo_support branch 5 times, most recently from 45bbcab to bf82a68 Compare August 14, 2022 04:18

tdyas requested a review from sureshjoshi August 14, 2022 04:20

tdyas force-pushed the golang_cgo_support branch from bf7dc63 to 5473b39 Compare August 21, 2022 15:44

tdyas mentioned this pull request Aug 21, 2022

Support third-party Go packages with CFiles #14338

Closed

tdyas mentioned this pull request Aug 30, 2022

go: support placing resource files in packages (and not just embedding) #16688

Merged

tdyas mentioned this pull request Aug 30, 2022

go: extract package name from analysis #16689

Merged

tdyas force-pushed the golang_cgo_support branch from 5473b39 to e5a02b3 Compare August 31, 2022 20:39

tdyas mentioned this pull request Aug 31, 2022

go: provide a way to inject sandbox root into args #16722

Merged

tdyas mentioned this pull request Aug 31, 2022

go: minor changes to analysis #16724

Merged

tdyas pushed a commit that referenced this pull request Aug 31, 2022

go: minor changes to analysis (#16724)

107d56b

Fix a comment and a analysis JSON extraction nit. Extracted from cgo changes in #16413. [ci skip-build-wheels]

tdyas force-pushed the golang_cgo_support branch from 3b661c3 to 228b4fc Compare September 1, 2022 00:42

tdyas mentioned this pull request Sep 1, 2022

go: check that main packages are actually called main #16734

Merged

sureshjoshi reviewed Sep 2, 2022

View reviewed changes

tdyas force-pushed the golang_cgo_support branch from 353ff3f to c761762 Compare September 8, 2022 05:58

Tom Dyas added 9 commits September 11, 2022 17:22

[ci skip-rust]

a025a1b

[ci skip-build-wheels]

default compiler options

f8126f2

[ci skip-rust] [ci skip-build-wheels]

debug goal to export Cgo codegen

cc718b4

[ci skip-rust] [ci skip-build-wheels]

SRCDIR replacement

609ce27

[ci skip-rust] [ci skip-build-wheels]

add bash path

8edf881

[ci skip-rust] [ci skip-build-wheels]

cleanup merging this with coverage support

189acd7

[ci skip-rust] [ci skip-build-wheels]

rebase fixes

abb8b3b

[ci skip-rust] [ci skip-build-wheels]

diff fixes to remove some changes

438ad07

[ci skip-rust] [ci skip-build-wheels]

update option names

73e7380

[ci skip-rust] [ci skip-build-wheels]

tdyas force-pushed the golang_cgo_support branch from c761762 to 73e7380 Compare September 11, 2022 21:24

fmt

437a53a

[ci skip-rust] [ci skip-build-wheels]

tdyas requested a review from benjyw September 11, 2022 21:37

benjyw reviewed Sep 12, 2022

View reviewed changes

Eric-Arellano approved these changes Sep 12, 2022

View reviewed changes

add tests for pkgconfig parsing + fix a bug

1453b34

[ci skip-rust] [ci skip-build-wheels]

Tom Dyas added 2 commits September 12, 2022 15:55

some code review comments

2807559

[ci skip-rust] [ci skip-build-wheels]

use MultiGet in debug goal

6a1d24b

[ci skip-rust] [ci skip-build-wheels]

Eric-Arellano approved these changes Sep 12, 2022

View reviewed changes

Tom Dyas added 2 commits September 12, 2022 16:26

additional copyright notices and help update

b74ed3b

[ci skip-rust] [ci skip-build-wheels]

more comments

8f19cdc

[ci skip-rust] [ci skip-build-wheels]

Eric-Arellano approved these changes Sep 12, 2022

View reviewed changes

tdyas merged commit 7d7eb22 into pantsbuild:main Sep 12, 2022

tdyas deleted the golang_cgo_support branch September 12, 2022 22:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

go: add basic support for cgo #16413

go: add basic support for cgo #16413

tdyas commented Aug 4, 2022

tdyas commented Aug 14, 2022

tdyas commented Aug 30, 2022

tdyas commented Aug 31, 2022

sureshjoshi Sep 2, 2022

tdyas Sep 2, 2022

sureshjoshi Sep 2, 2022

sureshjoshi Sep 2, 2022

tdyas Sep 2, 2022

tdyas Sep 5, 2022 •

edited

Loading

tdyas Sep 5, 2022

sureshjoshi Sep 5, 2022

sureshjoshi Sep 5, 2022

sureshjoshi Sep 2, 2022

tdyas Sep 2, 2022 •

edited

Loading

sureshjoshi Sep 2, 2022

sureshjoshi Sep 2, 2022

benjyw left a comment

tdyas commented Sep 12, 2022

Eric-Arellano left a comment

Eric-Arellano commented Sep 12, 2022

tdyas commented Sep 12, 2022

go: add basic support for cgo #16413

go: add basic support for cgo #16413

Conversation

tdyas commented Aug 4, 2022

tdyas commented Aug 14, 2022

tdyas commented Aug 30, 2022

tdyas commented Aug 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdyas Sep 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdyas Sep 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benjyw left a comment

Choose a reason for hiding this comment

tdyas commented Sep 12, 2022

Eric-Arellano left a comment

Choose a reason for hiding this comment

Eric-Arellano commented Sep 12, 2022

tdyas commented Sep 12, 2022

tdyas Sep 5, 2022 •

edited

Loading

tdyas Sep 2, 2022 •

edited

Loading