Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Build docker compose images nightly and use them in CI #2579

Closed
wants to merge 20 commits into from

Conversation

MOZGIII
Copy link
Contributor

@MOZGIII MOZGIII commented May 12, 2020

This is an attempt to finally streamline the CI and ensure we have quick builds in CI and a unified environment configs across local and CI tasks.

To make things work, I had to introduce the following changes:

  • configure images for all the services at docker-compose.yml
  • refresh the CI images build script
  • set up automated nightly builds for CI images (the auto builds at the docker hub end are to be disabled)
  • made the per-push CI flow use pre-built docker images - this allows us to properly control build environment at CI, but eliminates the image build step, dramatically speeding up the flow
  • solve the issue with tmp files being on overlayfs (cause of issues with tests failing in docker)

While working on this, I uncovered a few issues that I'm not planning on solving in this PR:

  • build image Dockerfiles are not cache-friendly, we should correct this
  • docker-compose.yml, Makefile and .github/workflow/code.yml each contains configuration for the same "unit"s, but it's all coded separately; there's way too much attention required to implement keep things in order and consistent, and it's hard to maintain; we should revisit the approach, and root all those places in a single source of truth

@MOZGIII MOZGIII changed the title chore: Build docker compose images nightly and used them in CI chore: Build docker compose images nightly and use them in CI May 12, 2020
@MOZGIII MOZGIII requested a review from binarylogic May 12, 2020 09:47
@MOZGIII MOZGIII marked this pull request as ready for review May 12, 2020 09:47
@MOZGIII MOZGIII requested a review from a user May 12, 2020 09:47
@MOZGIII
Copy link
Contributor Author

MOZGIII commented May 12, 2020

The rest of the CI failures are actual errors that went undetected with the current CI setup. Let's merge this and fix the errors in other PRs.

@fanatid fanatid mentioned this pull request May 12, 2020
@MOZGIII MOZGIII removed the request for review from a user May 12, 2020 18:24
@MOZGIII MOZGIII force-pushed the nightly-images branch 4 times, most recently from 0f655c6 to eccffa6 Compare May 12, 2020 22:24
@MOZGIII
Copy link
Contributor Author

MOZGIII commented May 13, 2020

This PR still has one failing job - the test-unit. The interesting part is that it only consistently fails if we’re running the job within docker.
@LucioFranco you mentioned you were working on tests troubleshooting - is this among the things you’re already looking into?

@MOZGIII MOZGIII force-pushed the nightly-images branch 5 times, most recently from 49c16ee to 6a36ff4 Compare May 14, 2020 12:48
MOZGIII added 10 commits May 14, 2020 16:36
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
MOZGIII added 10 commits May 14, 2020 16:36
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
…script

Signed-off-by: MOZGIII <mike-n@narod.ru>
…thout docker

Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
19.04 is deprecated and is not longer updated at Docker Hub.
20.04, in turn, in the fresh latest LTS release.

Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
@MOZGIII
Copy link
Contributor Author

MOZGIII commented May 18, 2020

This is blocked by #2618

@Hoverbear
Copy link
Contributor

Is this still relevant @MOZGIII ?

@MOZGIII
Copy link
Contributor Author

MOZGIII commented Jun 12, 2020

The PR - definitely not. We might still want to build an environment image every night though.

@MOZGIII MOZGIII closed this Jun 12, 2020
@MOZGIII MOZGIII deleted the nightly-images branch June 12, 2020 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants