Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline Creation for Building and Pushing Connectors Docker Images #2152

Merged
merged 10 commits into from
Feb 15, 2024
Merged

Pipeline Creation for Building and Pushing Connectors Docker Images #2152

merged 10 commits into from
Feb 15, 2024

Conversation

markjhoy
Copy link
Contributor

@markjhoy markjhoy commented Feb 13, 2024 •
edited
Loading

Creation of a pipeline for scheduled releases of Docker images for Elastic Connectors. This builds, tests, and pushes Docker images for arm64, amd64, and a multi-architecture image containing both.

This adds a pipeline and scripts that:

  • Builds docker images for each platform (and runs a quick test on each to verify they look correct)
  • Pushes the images to the registry
  • Builds a multi-arch image from the individual pushed images

This will create three new artifacts in the elastic-connectors Docker registry:

  • docker.elastic.co/enterprise-search/elastic-connectors-amd64:<VERSION>: The amd64 platform version of the image
  • docker.elastic.co/enterprise-search/elastic-connectors-arm64:<VERSION>: The arm64 platform version of the image
  • docker.elastic.co/enterprise-search/elastic-connectors:<VERSION>: A multi-architecture image that includes both the amd64 and arm64 versions

Checklists

Pre-Review Checklist

  • this PR has a meaningful title
  • this PR has a thorough description
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)

@markjhoy markjhoy marked this pull request as ready for review February 15, 2024 16:47
@markjhoy markjhoy requested review from a team, seanstory and artem-shelkovnikov February 15, 2024 16:47
seanstory
seanstory reviewed Feb 15, 2024
View reviewed changes
Copy link
Member

@seanstory seanstory left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is awesome.

All looks good - have a few nits, and two quick security concern comments. But once those are addressed I think we should go ahead and merge this so that we can try actually running it to publish something.

.buildkite/publish/build-multiarch-docker.sh
buildah login --username="${DOCKER_USER}" --password="${DOCKER_PASSWORD}" docker.elastic.co

echo "Creating manifest..."
buildah manifest create $TAG_NAME \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh this is cool. I didn't know about buildah manifest. Do you know what the tradeoffs are between this and using drivah with their toml syntax? Should we switch the data-extraction-service to use buildah manifest instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That I do not... I will look into it though!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... and I don't know if drivah can create multi-arch images... :/.

@markjhoy markjhoy requested a review from seanstory February 15, 2024 21:54
@markjhoy markjhoy requested review from seanstory and a team February 15, 2024 21:54
@markjhoy markjhoy enabled auto-merge (squash) February 15, 2024 21:55
@markjhoy markjhoy merged commit 31fd45a into main Feb 15, 2024
2 checks passed
@markjhoy markjhoy deleted the markjhoy/add_docker_release_pipeline branch February 15, 2024 22:04
github-actions bot pushed a commit that referenced this pull request Feb 15, 2024
@markjhoy
Pipeline Creation for Building and Pushing Connectors Docker Images (#…
c6a5334 
…2152)
@github-actions github-actions bot mentioned this pull request Feb 15, 2024
Merged
@github-actions GitHub Actions
Copy link

💚 Backport PR(s) successfully created

Status Branch Result
✅ 8.13 #2178

This backport PR will be merged automatically after passing CI.

markjhoy added a commit that referenced this pull request Feb 15, 2024
@github-actions @markjhoy
[8.13] Pipeline Creation for Building and Pushing Connectors Docker I…
0480077 
…mages (#2152) (#2178)

Co-authored-by: Mark J. Hoy <mark.hoy@elastic.co>
This was referenced Feb 15, 2024
Closed
Merged
seanstory pushed a commit that referenced this pull request Feb 29, 2024
@markjhoy
Pipeline Creation for Building and Pushing Connectors Docker Images (#…
f0bfc53 
…2152)

(cherry picked from commit 31fd45a)

# Conflicts:
#	catalog-info.yaml
@seanstory seanstory mentioned this pull request Feb 29, 2024
Merged
@seanstory
Copy link
Member

💚 All backports created successfully

Status Branch Result
✅ 8.12

Questions ?

Please refer to the Backport tool documentation

seanstory added a commit that referenced this pull request Feb 29, 2024
@seanstory @markjhoy
[8.12] Pipeline Creation for Building and Pushing Connectors Docker I…
3567509 
…mages (#2152) (#2219)

Co-authored-by: Mark J. Hoy <mark.hoy@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@artem-shelkovnikov artem-shelkovnikov Awaiting requested review from artem-shelkovnikov

@seanstory seanstory Awaiting requested review from seanstory

Assignees
No one assigned
Labels
auto-backport v8.12.2.0 v8.13.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@markjhoy @seanstory