Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[internal] go: initial support for embedding resources #13743

Merged
merged 4 commits into from
Dec 3, 2021

Conversation

tdyas
Copy link
Contributor

@tdyas tdyas commented Nov 30, 2021

Support embedding resources into Go binaries as per the embed package.

Note: The analysis of what to embed is done by our Go-based package analysis code and not in the Python rules.

[ci skip-rust]

@tdyas
Copy link
Contributor Author

tdyas commented Nov 30, 2021

@stuhood: Any advice on this dependency cycle?:

E           pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
E
E           Engine traceback:
E             in select
E             in pants.backend.go.util_rules.first_party_pkg.compute_first_party_package_info (//:pkg)
E             in pants.engine.internals.graph.transitive_targets
E             in pants.engine.internals.graph.transitive_dependency_mapping
E             in pants.engine.internals.graph.resolve_targets (//:pkg)
E             in pants.engine.internals.graph.resolve_unexpanded_targets (//:pkg)
E             in pants.engine.internals.graph.resolve_dependencies (//:pkg)
E           Traceback (no traceback):
E             <pants native internals>
E           Exception: The dependency graph contained a cycle:
E
E             @rule(pants.engine.platform.current_platform()) <-
E             @rule(pants.backend.go.util_rules.first_party_pkg.compute_first_party_package_info(//:pkg))
E             @rule(pants.backend.go.util_rules.first_party_pkg.setup_analyzer())
E             @rule(pants.backend.go.util_rules.import_analysis.generate_import_config())
E             @rule(pants.backend.go.util_rules.import_analysis.determine_go_std_lib_imports())
E             @rule(pants.engine.process.fallible_to_exec_result_or_raise())
E             @rule(pants.engine.process.upcast_process()) <-
E
E           If the dependencies in the above path are for your BUILD targets, you may need to use more granular targets or replace BUILD target dependencies with file dependencies. If they are not for your BUILD targets, then please file a Github issue!
E
E           See https://www.pantsbuild.org/v2.9/docs/targets#dependencies-and-dependency-inference for more information.

from running: ./pants test src/python/pants/backend/go/util_rules/first_party_pkg_test.py -- -vv -k test_embeds_supported

@stuhood
Copy link
Member

stuhood commented Nov 30, 2021

Hm, yikes. It looks like maybe the native cycle message is broken... I expect that the actual cycle matches the traceback: compute_first_party_package_info (//:pkg) cycling back on itself.

EDIT: Opened #13744.

@tdyas
Copy link
Contributor Author

tdyas commented Nov 30, 2021

Hm, yikes. It looks like maybe the native cycle message is broken... I expect that the actual cycle matches the traceback: compute_first_party_package_info (//:pkg) cycling back on itself.

That turned out to be it. Requesting TransitiveTargets triggered dependency inference which tries to do analysis for go_package targets which loops back to compute_first_party_package_info.

For now, I limited the scope of resource[s] targets to just those provided as explicit dependencies to work around the loop by requesting ExplictlyProvidedDependencies. It would be helpful for DependenciesRequest to be able to resolve only resource-related targets. (TODOs added.) This would allow using resources targets that may be non-direct dependencies.

The integration test works. Will open this for review after some cleanups tomorrow.

Tom Dyas added 4 commits December 2, 2021 17:21
[ci skip-rust]

[ci skip-build-wheels]
[ci skip-rust]

[ci skip-build-wheels]
[ci skip-rust]

[ci skip-build-wheels]
[ci skip-rust]

[ci skip-build-wheels]
@tdyas tdyas changed the title [WIP] [internal] go: support embedding resources [internal] go: initial support for embedding resources Dec 2, 2021
@tdyas tdyas marked this pull request as ready for review December 2, 2021 22:49
TestEmbedPatterns []string `json:",omitempty"` // patterns from TestGoFiles
XTestEmbedPatterns []string `json:",omitempty"` // patterns from XTestGoFiles
EmbedPatterns []string `json:",omitempty"` // patterns from GoFiles, CgoFiles
EmbedConfig *EmbedCfg `json:",omitempty"` // files matching the EmbedPatterns
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: The go list command returns lists of files to embed here and relies on the invoking tool to apply wildcards. That approach won't work in Python since we want to use the glob algorithm in the Go standard library which seems to differ somewhat from the one in Python (around certain syntax).

go list returns EmbedFiles, TestEmbedFiles, and XTestEmbedFiles. Since we analyze the wildcards in Go, this analysis just returns the exact embedcfg to use.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would probably be good to put in a comment. Wdyt?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in chat, will do in follow-up PR.

// Obtain a list of files in and under the package's directory. These will be embeddable files.
// TODO: Support resource targets elsewhere in the repository.

fmt.Fprintf(os.Stderr, "computeEmbedConfigs(directory=%s)\n", directory)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a debugging print and will need to come out (along with other instances).

@@ -186,39 +184,6 @@ class BuiltGoPackage:
import_paths_to_pkg_a_files: FrozenDict[str, str]


@dataclass(unsafe_hash=True)
@frozen_after_init
class EmbedConfig:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to separate file.

if tgt.has_field(ResourceSourceField) or tgt.has_field(ResourcesGeneratingSourcesField)
# TODO: Currently limited to resources directly under package in source tree. Allow any resource and
# have paths be relative to the resources' spec_path.
and tgt.address.spec_path.startswith(request.address.spec_path)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wildcards with Go file embedding are relative to the package directory and .. seems to be prohibited (at least by Bazel rules_go). Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

For now, this code makes a simpler choice and limits resources to be in or under the package directory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

That's generally where source roots are applied. It would be a bit awkward to apply them in some cases, but not in others... perhaps something in here should fail-fast if any resource dependency is not located below a go target in the filesystem?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

)
resources_digest = await Get(Digest, AddPrefix(stripped_resources_digest, "__resources__"))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code puts the resources into a separate directory so that the analysis code doesn't consider source files for the wildcards. The analysis code is passed the __resources__ path so it knows where the resources to consider are.

Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't grok the go portion, but the rest looks good.

It would be helpful for DependenciesRequest to be able to resolve only resource-related targets. (TODOs added.) This would allow using resources targets that may be non-direct dependencies.

Mm... yea, this is related to the edge-label filtering proposal, and to SpecialCasedDependencies.

if tgt.has_field(ResourceSourceField) or tgt.has_field(ResourcesGeneratingSourcesField)
# TODO: Currently limited to resources directly under package in source tree. Allow any resource and
# have paths be relative to the resources' spec_path.
and tgt.address.spec_path.startswith(request.address.spec_path)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs discussion on what we want to support here for resources targets not in or under the package directory. For example, what should the embed be relative to in that case?

That's generally where source roots are applied. It would be a bit awkward to apply them in some cases, but not in others... perhaps something in here should fail-fast if any resource dependency is not located below a go target in the filesystem?

@tdyas tdyas merged commit a91d25f into pantsbuild:main Dec 3, 2021
@tdyas tdyas deleted the golang_embed_resources branch December 3, 2021 18:40
illicitonion added a commit that referenced this pull request Dec 7, 2021
Internal changes:

* [internal] Remove superfluous f-string specifiers ([#13821](#13821))

* [internal] scala: extract annotations as consumed types ([#13810](#13810))

* [jvm] Split nailgun digest from input file digests ([#13813](#13813))

* [internal] jvm: add jre_major_version and use stderr to properly extract version ([#13812](#13812))

* [internal] Clean up Go `embed` support's handling of dependencies ([#13801](#13801))

* [internal] scala: handle package object syntax in parser ([#13809](#13809))

* [internal] java: fix junit sentinel rule setup ([#13815](#13815))

* [internal] upgrade to rust v1.57.0 ([#13807](#13807))

* [internal] add failing test for FrozenDict equality issue ([#13389](#13389))

* [internal] Use `PyObject` instead of `Value` in more places ([#13802](#13802))

* Remove MultiPlatform Process abstractions ([#12725](#12725))

* [internal] add `JvmToolBase` for lockfile handling for JVM tools ([#13777](#13777))

* [internal] Port `MergeDigests` to Rust ([#13773](#13773))

* [jvm] Spawn nailgun servers outside the pool lock ([#13796](#13796))

* [internal] DRY loading internal Go binaries ([#13800](#13800))

* [internal] Convert unit tests to use pytest ([#13798](#13798))

* [internal] remove dead code - socket util. ([#13797](#13797))

* [internal] Reorganize Go parser scripts ([#13791](#13791))

* Adds Jackson core/datatype to `JVM_ARTIFACT_MAPPINGS` ([#13792](#13792))

* [internal] go: initial support for embedding resources ([#13743](#13743))

* [internal] Refer to `go.mod` path when downloading packages ([#13786](#13786))

* [internal] More robust Go dependency inference tests ([#13785](#13785))

* [internal] `tailor` doesn't add `go_package` for `testdata` folder ([#13783](#13783))

* [internal] Add Scala backend to dryrun for wheel builds. ([#13772](#13772))

* [internal] Unify JVM thirdparty resolves ([#13771](#13771))

* [internal] scala: infer dependencies from consumed symbols and intermediate scopes ([#13758](#13758))

* [internal] java: infer scala encoded symbols ([#13739](#13739))

* [internal] scala: parse and report package scopes ([#13738](#13738))

* [internal] go: configure included env vars for `GoSdkProcess` ([#13734](#13734))

* Fix some `./cargo audit` vulnerabilities. ([#13728](#13728))

* [internal] fix non-empty __init__.py ([#13730](#13730))

* Compute RepositoryPex directly from addresses. ([#13716](#13716))

* Upgrade to cargo-audit 0.16.0. ([#13729](#13729))

* Simplify `NativeHandler`. ([#13727](#13727))

* [internal] Switch to a maintained fork of the `fuse` crate for `brfs`. ([#13679](#13679))

* [internal] Add infrastructure to support deprecating field names ([#13666](#13666))

* [internal] Introduce OptionalPex/OptionalPexRequest. ([#13712](#13712))

* [internal] tailor adds go_package targets ([#13703](#13703))

* [internal] Remove unused testproject for pants-plugin ([#13704](#13704))

* [internal] Rename ambiguous `subpath` variable for Go code ([#13701](#13701))

* [internal] scala: generate the JVM names seen by Java code for Scala code ([#13696](#13696))

* Use RequirementsPexRequest in run_pex_binary.py. ([#13693](#13693))

* [internal] Refactor finding owning `go_mod` for first-party packages ([#13695](#13695))

* [internal] repl: add append_only_caches / run_in_workspace attributes to ReplRequest ([#13599](#13599))

* [internal] switch back to official `cargo-ensure-prefix` crate ([#13692](#13692))

* [internal] scala: extract type names from all Type.* AST nodes ([#13685](#13685))

* [internal] Convert unit tests to use pytest ([#13652](#13652))

* Unblock codegen support for java export analysis (#13645) ([#13675](#13675))

* [internal] upgrade to Rust 2021 Edition ([#13644](#13644))

* [internal] Don't store derived values on `go_first_party_package` targets ([#13676](#13676))

* Upgrade to py03 0.15.1. ([#13725](#13725))

* Add PyPDF2 to module mapping ([#13717](#13717))

* Upgrade console and indacatif. ([#13726](#13726))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants