Add support for cross-building Python applications to test, package inside-of/for a Docker image
#13682
Comments
|
I'm sketching an idea, see if that could be in the right direction.. Example BUILD file: python_tests(name="tests")
docker_image(name="base-tools")
docker_test_image(name="test-image", dependencies=[":tests"], instructions=["FROM :base-tools"])This new Seen here is also a new unimplemented feature, to reference another image by it's target address in the The purpose of the Does this make sense? |
|
My original though was to avoid adding a new goal type and invert the logic a bit. Something like– docker_image(name="base-tools")
python_tests(
name="tests",
run_in_container = ":base-tools",
) |
|
I do like this living in |
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
|
+1 I have the same use case... my current alternative is to build the target docker image and then run having a supported target for this would be great and I think very useful! |
|
@meganvw Thanks for letting us know. It helps knowing what features are valuable to implement. |
I've been successfully running For multiple concurrently running containers I also had to add the |
stuhood
changed the title
test, run, package inside-of/for a Docker image
|
I think this would also address this issue #15557. Is this feature on the roadmap and to be expected shortly? |
|
Hi @ptrhck, It is not to be expected shortly. I've done some preliminary proof of concept work to try stuff out, but it is not actively being worked on by me, and I am unaware of anyone else working on this either. |
|
@ptrhck that being said, it's an open source project and we LOVE contributions from non-maintainers. We would be very happy to help you or anyone else if you're interested in implementing this! |
thejcannon
added
backend: Python
|
A few folks from Toolchain will be starting on this in the next few days. Here is a preliminary design doc: Native Container Support, but there will be independent design for some of the components. We'd love to collaborate with anyone else who is interested in seeing this feature come to fruition. |
|
Very much looking forward to this! Maybe https://github.com/dagger/dagger can give some inspiration for design choices. Overall, it adresses similiar issues. As you have stated in the motivation of the design doc, users might have applications they want to test with pants that rely on a lot of system-dependencies or third-party libraries. This feature will greatly improve portability. That is the idea of dagger as well: It runs on my machine, I can be sure it runs on CI. It only needs access to Docker, or BuildKit to be more specific. |
) In order to support differentiating `@rule` subgraphs for different "environments" as described in the design doc for #13682, `@rules` need to be able inject more than one parameter into any particular subgraph. As a concrete example: the `test` `@rules` need to be able to request "a `TestResult` for this `FieldSet` in this `Environment`". This change adds support for multiple parameters to a `Get`, such that that use case would be spelled out as: ```python test_result = await Get(TestResult, {field_set: FieldSet, environment: Environment}) ``` Fixes #7490. [ci skip-build-wheels]
Part of #13682. This target serves two purposes: 1. Define what image to use, which we will pass to the new Docker `CommandRunner` when setting up a `Process`. 2. Configure the env vars and search paths to use. A follow up will allow individual targets to specify which `environment` to use. If set to a `docker_environment` target, then we will propagate the image name to `Process`. [ci skip-rust] [ci skip-build-wheels]
…lementers (#16717) As described in #12934, each `@union` `Get` callsite currently implicitly represents an API where exactly the parameters provided to the `Get` are available, and no others. In order for other types to be made available to plugins (most immediately, a new `EnvironmentName` type to differentiate subgraphs for different environments in #13682), we need a new syntax to indicate that `@union`s provide additional types as part of their API. To that end, this change allows a `@union` declaration to specify additional types that will be in scope for callees (and consequently must be available to callers): ```python # Declares that callsites using a `Vehicle` must have `Fuel` in scope, and that `Fuel` will be # in scope for callees/implementers of the `@union`. @union(in_scope_types=[Fuel]) class Vehicle(ABC): ... ``` Fixes #12934. [ci skip-build-wheels]
In order to differentiate subgraphs of the `@rule` graph for different platforms and environments for #7735 (to support #13682), the environment that is in use needs to become a very widespread parameter in the `@rule` graph. To do that, this change: 1. Adjusts all production `QueryRule`s to declare that an `EnvironmentName` must be provided. 2. Adjusts most `@union`s to use #16717 to declare that an `EnvironmentName` will be available. 3. Computes the `Platform` from the `EnvironmentName` (currently as a noop, since we don't actually have multiple instances of the `EnvironmentName`). A caveat for point 1 is that to avoid churn, we only update _production_ `QueryRule`s in this change. For continuity in tests, `RuleRunner` will install a singleton `EnvironmentName` by default, and uses a (probably temporary) facility to filter the `EnvironmentName` back out of all `QueryRule`s. Tests that actually want to test with multiple environments should disable that facility.
|
A status update on this! As suggested by the linked issues/pulls above:
Our next steps will be to:
|
stuhood
changed the title
test, run, package inside-of/for a Docker imagetest, package inside-of/for a Docker image
|
Note: I've removed |
Part of #7735 and #13682. This allows individual test targets to choose which environment is used during their run, including by using a `docker_environment`. The entire subgraph will be updated appropriately, e.g. we will rediscover `BashBinary` and rebuild the PEXes needed, if relevant. [ci skip-rust]
This solves several problems we had with remote execution: 1. You can now specify different env vars than localhost, #7735 2. You can now mix remote execution with Docker execution from #13682 3. You can have multiple configurations for the remote execution server, which is particularly useful so you can specify different Docker images to run in remote execution. If you are not yet using the environments mechanism, then remote execution continues to be toggled via the `--remote-execution` option. Otherwise, remote execution is only used when a remote execution environment target is used. Follow up work will re-evaluate this by leveraging "environment matchers". See #16907 for more info. A follow up will add the `extra_platform_properties` field (#16908).
|
Another update here. This week:
Next week:
Stretch for next week is beginning to tackle additional use cases from the design doc, including adding environment awareness to the Thanks to @Eric-Arellano, @tdyas, and @chrisjrn for their work on this! |
Adds initial documentation for the Environments feature designed and implemented in #7735 and #13682. This initial version is meant to capture what will be possible in the `2.15.x` release, but it will/should contain some references to future work (some of which should block stabilizing environments). Fixes #13682, and fixes #16393. [ci skip-rust] [ci skip-build-wheels]
|
We just got And it looks like |
Is your feature request related to a problem? Please describe.
My problem is that we have a good chunk of python code which depends on a lot of system libraries and command line tools being on the PATH. This is bioinformatics code for which this pattern is quite common.
This code will run in a container, whose image is built by pants, in CI and production. When running tests in development environments I would like to be able to use the same environment.
Describe the solution you'd like
I would like to be able to tell pants– "when running this target, run it in a container from this image". Ideally that image can be built by pants automatically when needed.
Describe alternatives you've considered
One alternative we are considering is to move all our development work to linux machines and/or docker containers where we have everything installed. (We may do this regardless of this change).
Additional context
slack discussion - https://pantsbuild.slack.com/archives/C046T6T9U/p1637272879490300
Notes from #14145:
Cross-building Python dists (wheels) to target a different platform could be accomplished internally to Pants using Docker, to allow both cross-platform PEXes and Docker images to be built without pre-built dists.
Cross-building inside of a docker image is very similar to remote execution to a different plaform (as described in #11148): at a fundamental level, it's a bit like executing all logic remotely inside of the container. Similarly, it involves isolating the subgraph in which the cross-build is occuring using a different
Platform/Environment.The text was updated successfully, but these errors were encountered: