Refactor .travis.yml to deduplicate uses of Docker and use Mustache comments #7351
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Now that we have the parameterized mustache template, we never end up even using this script entry. Every single use of this base anchor overrides script, so it simply adds noise and makes things more confusing.
Build is a more appropriate name than launch, because it mirrors docker. Also starting with `docker` will make `docker_run_image` more symmetrical.
We want to explain some of our quirks, but its annoying for the comment to print to .travis.yml everytime it gets used. Instead, only put the comment once!
THe bootstrap shards were failing to read their args (`-b` vs. `-2b`). This is because Docker expects a string with its exact arguments, and will not do any env var resolution from the external environment. Instead, all env vars must be valuated by the parent bash process and then the result passed to Docker. This leads to slightly more duplication, but in some ways is more explicit and is an acceptable tradeoff.
cosmicexplorer
approved these changes
Mar 10, 2019
Danny pointed out it can be a bit confusing that we have two levels of templating: the mustache template and the env vars that it populates. Add comments to the mustache templates stressing that it is expecting env vars and explaining what it expects these to be. This is kinda like docstring for mustache. Also add comments in .travis.yml where we have a base image define a script, but then expect the callers of the anchor to provide env vars. Make this expectation clear.
It was a bit unclear why we need to set the env var and made it sound like there's an upcoming issue to resolve. Clarify this is intended and clarify how this fits in with everything else.
illicitonion
approved these changes
Mar 11, 2019
|
Thanks all! Merging in order to land #7197. Happy to address any concerns Benjy or Stu that you may have in a followup PR. |
Eric-Arellano
added a commit
to Eric-Arellano/pants
that referenced
this pull request
Mar 11, 2019
pantsbuild#7351 should have deleted $BOOTSTRAP_ARGS for the Linux Build engine shards. This does its cleanup.
Eric-Arellano
added a commit
that referenced
this pull request
Mar 12, 2019
### Problem Every time we run a Docker shard, we spend about 3 minutes installing Python 3.6 and its dependencies, even if that shard never once uses Python 3. In particular, this slows down our `Linux Build Wheels UCS4` shard and our `Linux Build Engine Py2` for a collective cost of 6 minutes every CI run (i.e. total CI cost, not wall time). Ideally, we would want Python 3 pre-installed on the base Centos6 image so that we never for any shard have to waste time installing Python 3 (see #7064). Until that lands, though, this is an additional unneeded burden. ### Solution Create a new image `travis_ci_py36`. Thanks to how we can now parametrize the docker image name in `travis.yml.mustache`, this brings no additional cognitive overload to our `.travis.yml`, which was the original reason we did not create a duplicate Dockerfile. Also does a little cleanup from #7351 in removing a bad env var `$BOOTSTRAP_ARGS` for the linux build engine shards.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
#7235 introduced an initial deduplication of launching docker images, notably allowing the image name to be parametrized. While working on #7197, it became evident how frequently the same code was being repeated for calling
docker run. The only thing that changes between these invocations is what goes insidesh -c "my bash command".This duplication makes our Travis code harder to understand, along with the usual arguments for DRY.
Solution
Introduce a new
docker_run_imagemustache template that takes the parameters$docker_image_nameand$docker_run_command.Also changed:
launch_docker_image->docker_build_image&travis_docker_image, as it gets overridden every time by its caller so is unnecessary and confusing..travis.yml, which is good for reducing noise, while still allowing us to explain the concept in the template.Result
CI should perform the same with the benefit of a shorter and better abstracted
travis.yml.mustache.