[internal] go: initial support for embedding resources #13743

tdyas · 2021-11-30T05:29:43Z

Support embedding resources into Go binaries as per the embed package.

Note: The analysis of what to embed is done by our Go-based package analysis code and not in the Python rules.

[ci skip-rust]

tdyas · 2021-11-30T05:30:54Z

@stuhood: Any advice on this dependency cycle?:

E           pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
E
E           Engine traceback:
E             in select
E             in pants.backend.go.util_rules.first_party_pkg.compute_first_party_package_info (//:pkg)
E             in pants.engine.internals.graph.transitive_targets
E             in pants.engine.internals.graph.transitive_dependency_mapping
E             in pants.engine.internals.graph.resolve_targets (//:pkg)
E             in pants.engine.internals.graph.resolve_unexpanded_targets (//:pkg)
E             in pants.engine.internals.graph.resolve_dependencies (//:pkg)
E           Traceback (no traceback):
E             <pants native internals>
E           Exception: The dependency graph contained a cycle:
E
E             @rule(pants.engine.platform.current_platform()) <-
E             @rule(pants.backend.go.util_rules.first_party_pkg.compute_first_party_package_info(//:pkg))
E             @rule(pants.backend.go.util_rules.first_party_pkg.setup_analyzer())
E             @rule(pants.backend.go.util_rules.import_analysis.generate_import_config())
E             @rule(pants.backend.go.util_rules.import_analysis.determine_go_std_lib_imports())
E             @rule(pants.engine.process.fallible_to_exec_result_or_raise())
E             @rule(pants.engine.process.upcast_process()) <-
E
E           If the dependencies in the above path are for your BUILD targets, you may need to use more granular targets or replace BUILD target dependencies with file dependencies. If they are not for your BUILD targets, then please file a Github issue!
E
E           See https://www.pantsbuild.org/v2.9/docs/targets#dependencies-and-dependency-inference for more information.

from running: ./pants test src/python/pants/backend/go/util_rules/first_party_pkg_test.py -- -vv -k test_embeds_supported

stuhood · 2021-11-30T05:34:32Z

Hm, yikes. It looks like maybe the native cycle message is broken... I expect that the actual cycle matches the traceback: compute_first_party_package_info (//:pkg) cycling back on itself.

EDIT: Opened #13744.

tdyas · 2021-11-30T07:37:54Z

Hm, yikes. It looks like maybe the native cycle message is broken... I expect that the actual cycle matches the traceback: compute_first_party_package_info (//:pkg) cycling back on itself.

That turned out to be it. Requesting TransitiveTargets triggered dependency inference which tries to do analysis for go_package targets which loops back to compute_first_party_package_info.

For now, I limited the scope of resource[s] targets to just those provided as explicit dependencies to work around the loop by requesting ExplictlyProvidedDependencies. It would be helpful for DependenciesRequest to be able to resolve only resource-related targets. (TODOs added.) This would allow using resources targets that may be non-direct dependencies.

The integration test works. Will open this for review after some cleanups tomorrow.

[ci skip-rust] [ci skip-build-wheels]

tdyas · 2021-12-02T22:51:25Z

src/python/pants/backend/go/util_rules/analyze_package.go

-	TestEmbedPatterns  []string `json:",omitempty"` // patterns from TestGoFiles
-	XTestEmbedPatterns []string `json:",omitempty"` // patterns from XTestGoFiles
+	EmbedPatterns      []string  `json:",omitempty"` // patterns from GoFiles, CgoFiles
+	EmbedConfig        *EmbedCfg `json:",omitempty"` // files matching the EmbedPatterns


Note: The go list command returns lists of files to embed here and relies on the invoking tool to apply wildcards. That approach won't work in Python since we want to use the glob algorithm in the Go standard library which seems to differ somewhat from the one in Python (around certain syntax).

go list returns EmbedFiles, TestEmbedFiles, and XTestEmbedFiles. Since we analyze the wildcards in Go, this analysis just returns the exact embedcfg to use.

This would probably be good to put in a comment. Wdyt?

As discussed in chat, will do in follow-up PR.

tdyas · 2021-12-02T22:51:53Z

src/python/pants/backend/go/util_rules/analyze_package.go

+	// Obtain a list of files in and under the package's directory. These will be embeddable files.
+	// TODO: Support resource targets elsewhere in the repository.
+
+	fmt.Fprintf(os.Stderr, "computeEmbedConfigs(directory=%s)\n", directory)


This is a debugging print and will need to come out (along with other instances).

tdyas · 2021-12-02T22:52:28Z

src/python/pants/backend/go/util_rules/build_pkg.py

@@ -186,39 +184,6 @@ class BuiltGoPackage:
    import_paths_to_pkg_a_files: FrozenDict[str, str]


-@dataclass(unsafe_hash=True)
-@frozen_after_init
-class EmbedConfig:


Moved to separate file.

tdyas · 2021-12-02T22:55:12Z

src/python/pants/backend/go/util_rules/first_party_pkg.py

+        if tgt.has_field(ResourceSourceField) or tgt.has_field(ResourcesGeneratingSourcesField)
+        # TODO: Currently limited to resources directly under package in source tree. Allow any resource and
+        # have paths be relative to the resources' spec_path.
+        and tgt.address.spec_path.startswith(request.address.spec_path)


The wildcards with Go file embedding are relative to the package directory and .. seems to be prohibited (at least by Bazel rules_go). Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

For now, this code makes a simpler choice and limits resources to be in or under the package directory.

Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

That's generally where source roots are applied. It would be a bit awkward to apply them in some cases, but not in others... perhaps something in here should fail-fast if any resource dependency is not located below a go target in the filesystem?

tdyas · 2021-12-02T22:56:24Z

src/python/pants/backend/go/util_rules/first_party_pkg.py

    )
+    resources_digest = await Get(Digest, AddPrefix(stripped_resources_digest, "__resources__"))


This code puts the resources into a separate directory so that the analysis code doesn't consider source files for the wildcards. The analysis code is passed the __resources__ path so it knows where the resources to consider are.

stuhood

Didn't grok the go portion, but the rest looks good.

It would be helpful for DependenciesRequest to be able to resolve only resource-related targets. (TODOs added.) This would allow using resources targets that may be non-direct dependencies.

Mm... yea, this is related to the edge-label filtering proposal, and to SpecialCasedDependencies.

stuhood · 2021-12-03T00:44:57Z

src/python/pants/backend/go/util_rules/first_party_pkg.py

+        if tgt.has_field(ResourceSourceField) or tgt.has_field(ResourcesGeneratingSourcesField)
+        # TODO: Currently limited to resources directly under package in source tree. Allow any resource and
+        # have paths be relative to the resources' spec_path.
+        and tgt.address.spec_path.startswith(request.address.spec_path)


Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

That's generally where source roots are applied. It would be a bit awkward to apply them in some cases, but not in others... perhaps something in here should fail-fast if any resource dependency is not located below a go target in the filesystem?

Internal changes: * [internal] Remove superfluous f-string specifiers ([#13821](#13821)) * [internal] scala: extract annotations as consumed types ([#13810](#13810)) * [jvm] Split nailgun digest from input file digests ([#13813](#13813)) * [internal] jvm: add jre_major_version and use stderr to properly extract version ([#13812](#13812)) * [internal] Clean up Go `embed` support's handling of dependencies ([#13801](#13801)) * [internal] scala: handle package object syntax in parser ([#13809](#13809)) * [internal] java: fix junit sentinel rule setup ([#13815](#13815)) * [internal] upgrade to rust v1.57.0 ([#13807](#13807)) * [internal] add failing test for FrozenDict equality issue ([#13389](#13389)) * [internal] Use `PyObject` instead of `Value` in more places ([#13802](#13802)) * Remove MultiPlatform Process abstractions ([#12725](#12725)) * [internal] add `JvmToolBase` for lockfile handling for JVM tools ([#13777](#13777)) * [internal] Port `MergeDigests` to Rust ([#13773](#13773)) * [jvm] Spawn nailgun servers outside the pool lock ([#13796](#13796)) * [internal] DRY loading internal Go binaries ([#13800](#13800)) * [internal] Convert unit tests to use pytest ([#13798](#13798)) * [internal] remove dead code - socket util. ([#13797](#13797)) * [internal] Reorganize Go parser scripts ([#13791](#13791)) * Adds Jackson core/datatype to `JVM_ARTIFACT_MAPPINGS` ([#13792](#13792)) * [internal] go: initial support for embedding resources ([#13743](#13743)) * [internal] Refer to `go.mod` path when downloading packages ([#13786](#13786)) * [internal] More robust Go dependency inference tests ([#13785](#13785)) * [internal] `tailor` doesn't add `go_package` for `testdata` folder ([#13783](#13783)) * [internal] Add Scala backend to dryrun for wheel builds. ([#13772](#13772)) * [internal] Unify JVM thirdparty resolves ([#13771](#13771)) * [internal] scala: infer dependencies from consumed symbols and intermediate scopes ([#13758](#13758)) * [internal] java: infer scala encoded symbols ([#13739](#13739)) * [internal] scala: parse and report package scopes ([#13738](#13738)) * [internal] go: configure included env vars for `GoSdkProcess` ([#13734](#13734)) * Fix some `./cargo audit` vulnerabilities. ([#13728](#13728)) * [internal] fix non-empty __init__.py ([#13730](#13730)) * Compute RepositoryPex directly from addresses. ([#13716](#13716)) * Upgrade to cargo-audit 0.16.0. ([#13729](#13729)) * Simplify `NativeHandler`. ([#13727](#13727)) * [internal] Switch to a maintained fork of the `fuse` crate for `brfs`. ([#13679](#13679)) * [internal] Add infrastructure to support deprecating field names ([#13666](#13666)) * [internal] Introduce OptionalPex/OptionalPexRequest. ([#13712](#13712)) * [internal] tailor adds go_package targets ([#13703](#13703)) * [internal] Remove unused testproject for pants-plugin ([#13704](#13704)) * [internal] Rename ambiguous `subpath` variable for Go code ([#13701](#13701)) * [internal] scala: generate the JVM names seen by Java code for Scala code ([#13696](#13696)) * Use RequirementsPexRequest in run_pex_binary.py. ([#13693](#13693)) * [internal] Refactor finding owning `go_mod` for first-party packages ([#13695](#13695)) * [internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest ([#13599](#13599)) * [internal] switch back to official `cargo-ensure-prefix` crate ([#13692](#13692)) * [internal] scala: extract type names from all Type.* AST nodes ([#13685](#13685)) * [internal] Convert unit tests to use pytest ([#13652](#13652)) * Unblock codegen support for java export analysis (#13645) ([#13675](#13675)) * [internal] upgrade to Rust 2021 Edition ([#13644](#13644)) * [internal] Don't store derived values on `go_first_party_package` targets ([#13676](#13676)) * Upgrade to py03 0.15.1. ([#13725](#13725)) * Add PyPDF2 to module mapping ([#13717](#13717)) * Upgrade console and indacatif. ([#13726](#13726))

stuhood mentioned this pull request Nov 30, 2021

Reported cycle for native cycle detection is incorrect #13744

Closed

Tom Dyas added 4 commits December 2, 2021 17:21

embed resources

b3a9cec

[ci skip-rust] [ci skip-build-wheels]

get it working

530edb8

[ci skip-rust] [ci skip-build-wheels]

cleanups

482ab80

[ci skip-rust] [ci skip-build-wheels]

handle empty EmbedConfig's

22a3d05

[ci skip-rust] [ci skip-build-wheels]

tdyas force-pushed the golang_embed_resources branch from 14c4f94 to 22a3d05 Compare December 2, 2021 22:47

tdyas changed the title ~~[WIP] [internal] go: support embedding resources~~ [internal] go: initial support for embedding resources Dec 2, 2021

tdyas marked this pull request as ready for review December 2, 2021 22:49

tdyas commented Dec 2, 2021

View reviewed changes

tdyas requested review from Eric-Arellano and stuhood December 2, 2021 22:56

stuhood approved these changes Dec 3, 2021

View reviewed changes

tdyas merged commit a91d25f into pantsbuild:main Dec 3, 2021

tdyas deleted the golang_embed_resources branch December 3, 2021 18:40

illicitonion mentioned this pull request Dec 6, 2021

Prep for 2.9.0.dev3 #13823

Merged

Eric-Arellano mentioned this pull request Dec 7, 2021

Go: support resource embedding #13193

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[internal] go: initial support for embedding resources #13743

[internal] go: initial support for embedding resources #13743

tdyas commented Nov 30, 2021 •

edited

Loading

tdyas commented Nov 30, 2021

stuhood commented Nov 30, 2021 •

edited

Loading

tdyas commented Nov 30, 2021 •

edited

Loading

tdyas Dec 2, 2021

Eric-Arellano Dec 3, 2021

tdyas Dec 3, 2021

tdyas Dec 2, 2021

tdyas Dec 2, 2021

tdyas Dec 2, 2021

stuhood Dec 3, 2021

tdyas Dec 3, 2021

tdyas Dec 2, 2021

stuhood left a comment

stuhood Dec 3, 2021

		)
		resources_digest = await Get(Digest, AddPrefix(stripped_resources_digest, "__resources__"))

[internal] go: initial support for embedding resources #13743

[internal] go: initial support for embedding resources #13743

Conversation

tdyas commented Nov 30, 2021 • edited Loading

tdyas commented Nov 30, 2021

stuhood commented Nov 30, 2021 • edited Loading

tdyas commented Nov 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdyas commented Nov 30, 2021 •

edited

Loading

stuhood commented Nov 30, 2021 •

edited

Loading

tdyas commented Nov 30, 2021 •

edited

Loading