Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

./pants --v2 export-v2 [HACKY PROTOTYPE -- NOT FOR MERGING] #8760

Conversation

cosmicexplorer
Copy link
Contributor

@cosmicexplorer cosmicexplorer commented Dec 4, 2019

The v2 engine has the potential to make running and especially re-running tasks mostly instantaneous. This is a series of somewhat-orthogonal commits to implement resolve, gen, and export in v2 to:

  1. determine how much effort these would be to productionize,
  2. determine how much more effort is needed to realize the benefits of the v2 engine.

This draft PR currently takes a very hacky approach to the v1/v2 target API in #4535 by literally instantiating a v1 target for every v2 target, and accessing every single property on the v1 target, including memoized properties, copying them onto the v2 target. As a result (HYPOTHESIS -- PROFILING NEEDED), it is slower to run than v1 export. We expect that we can overcome these difficulties now that we have a working prototype to kick around.

Problem

(explain the context of the problem and why you're making this change. include
references to all relevant github issues.
)

Solution

(describe the modifications you've made.)

Result

(describe how your changes affect the end-user behavior of the system. this section is
optional, and should generally be summarized in the title of the pull request.
)

@cosmicexplorer cosmicexplorer changed the title ./pants --v2 export-v2 [HACKY PROTOTYPE] ./pants --v2 export-v2 [HACKY PROTOTYPE -- NOT FOR MERGING] Dec 4, 2019
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really, really neat as a proof of concept: lots of substantial work in here. Thanks Danny!

coursier: CoursierSubsystem,
build_root: BuildRoot,
) -> SnapshottedResolveResult:
# TODO: figure out whether the arbitrary_root kwarg is the right way to snapshot things outside of
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no safe way to do this with the engine currently, because we can't watch arbitrary paths. Perhaps after #4999

@cosmicexplorer cosmicexplorer force-pushed the export-classpath-v2-hacky-prototype branch from 72d1238 to a601b83 Compare December 14, 2019 03:31
panic at 'nested block_on: EnterError { reason: "attempted to run an executor while another executor
is already running" }', src/libcore/result.rs:1165

it's likely we'll have to make this an intrinsic!
use the new `test-thrift-gen-classpath` goal!!!
the command i'm currently using is:

rm -rfv wowzer && mkdir -vp ./wowzer && PEX_IGNORE_RCFILES=true ./pants --no-process-execution-{use-local-cache,cleanup-local-dirs} --no-enable-pantsd --glob-expansion-failure=ignore --v2 export-v2 --output-dir=./wowzer --output-file=out.json :: --tag='-impossible_to_import' && wc -c out.json
@cosmicexplorer
Copy link
Contributor Author

Always sad.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants