Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(containers): container builds use GCS for cache storage #14

Merged
merged 3 commits into from
Feb 25, 2020

Conversation

plumpy
Copy link
Member

@plumpy plumpy commented Feb 18, 2020

This makes a few changes:

  1. Before the compilation, we pull down a $repo-$branch.tgz file and
    uncompress it (e.g. igor-master.tgz or rosco-release-1.17.x.tgz) to
    the /workspace volume
  2. The Dockerfile.compile image is now expected to be a build step. This
    means we can run the container and it will compile files on the
    /workspace volume. Currently, we copy all the files into the
    container, compile them there, and then have to copy them back out.
    This new method is more along the lines with how GCB is supposed to
    work, but it does mean every Dockerfile.compile has to be
    changed to use CMD instead of RUN.
  3. After the compile, we tar up .gradle/caches and .gradle/wrapper to
    $repo-$branch.tgz and send it to gs://spinnaker-build-cache. This
    storage bucket is set to auto-delete files that haven't been
    modified in 14 days.

Removing our dependence on the gradle_cache image means that the
various microservices can define their own compile image with different
build tool configurations. I'd really like to have this for the Java 11
migration.

It also means we could move the Debian builds into Cloud Build, since
we aren't depending on them to produce that gradle_cache artifact.

This makes a few changes:
1. Before the compilation, we pull down a $repo-$branch.tgz file and
   uncompress it (e.g. igor-master.tgz or rosco-release-1.17.x.tgz) to
   the /workspace volume
2. The Dockerfile.compile image is now expected to be a build step. This
   means we can run the container and it will compile files on the
   /workspace volume. Currently, we copy all the files into the
   container, compile them there, and then have to copy them back out.
   This new method is more along the lines with how GCB is supposed to
   work, but it does mean every Dockerfile.compile has to be
   changed to use CMD instead of RUN.
3. After the compile, we tar up .gradle/caches and .gradle/wrapper to
   $repo-$branch.tgz and send it to gs://spinnaker-build-cache. This
   storage bucket is set to auto-delete files that haven't been
   modified in 14 days.

Removing our dependence on the `gradle_cache` image means that the
various microservices can define their own compile image with different
build tool configurations. I'd really like to have this for the Java 11
migration.

It also means we could move the Debian builds into Cloud Build, since
we aren't depending on them to produce that `gradle_cache` artifact.
@plumpy plumpy requested review from ezimanyi and ttomsu February 18, 2020 18:41
Copy link
Contributor

@ezimanyi ezimanyi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📦 ⚒️ 💰 🎉

plumpy added a commit to plumpy/echo that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/deck that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/clouddriver that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/fiat that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/fiat that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/front50 that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/gate that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/igor that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/igor that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/kayenta that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/rosco that referenced this pull request Feb 18, 2020
plumpy added a commit to plumpy/orca that referenced this pull request Feb 18, 2020
@ttomsu
Copy link
Member

ttomsu commented Feb 20, 2020

Thinking a bit here from helping a user compile a docker image - I wonder if there is a way we can compose the Docker images in each repo in such a way that they can docker build a slim/production container by just concatenating each Dockerfile into a multistage build?

Just an idea, haven't explored it at all.

@plumpy plumpy merged commit e252e0e into spinnaker:master Feb 25, 2020
plumpy added a commit to spinnaker/clouddriver that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/echo that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/fiat that referenced this pull request Feb 25, 2020
plumpy added a commit to spinnaker/front50 that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/gate that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/igor that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/kayenta that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/orca that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
plumpy added a commit to spinnaker/rosco that referenced this pull request Feb 25, 2020
* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
@plumpy plumpy deleted the container_builds branch February 25, 2020 02:31
plumpy added a commit to plumpy/spinnaker-monitoring that referenced this pull request Feb 25, 2020
plumpy added a commit to spinnaker/spinnaker-monitoring that referenced this pull request Feb 25, 2020
opsmxdemo pushed a commit to OpsMx/rosco that referenced this pull request Sep 15, 2020
…r#533)

* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
opsmxdemo pushed a commit to OpsMx/rosco that referenced this pull request Sep 16, 2020
…r#533)

* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
KathrynLewis pushed a commit to KathrynLewis/orca that referenced this pull request Jan 31, 2021
…r#3450)

* chore(build): Update Dockerfiles for new cloudbuild configs

See spinnaker/buildtool#14

* chore(dockerfile): use openjdk:8 instead of rolling our own
yunzhangit pushed a commit to yunzhangit/deck that referenced this pull request Mar 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants