[refactor][ci] Build the docker image with docker-maven-plugin #17148

tisonkun · 2022-08-17T15:30:41Z

This closes Build the docker image without dockerfile-maven-plugin #16652.
This closes Changed Docker Maven plugin from Spotify to fabric8 #16099.
This closes [WIP][ci] Change packaging plugin from dockerfile-maven-plugin to docker-maven-plugin #16677.

doc-not-needed

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2022-08-18T02:57:19Z

Verify build successfully on macOS with M1 chip. Although a weird issue doesn't exist in the CI environment: https://github.com/apache/pulsar/pull/17129/files#r948602287. It should not be included in this patch, though.

tisonkun · 2022-08-18T03:09:31Z

cc @skyleaworlder you're welcome to review this patch. It's based on your work.

cc @merlimat @nicoloboschi @eolivelli @Shoothzj @michaeljmarshall @mattisonchao

Build docker image succeed. I may retrigger other flaky tests when all steps finish.

tisonkun · 2022-08-18T03:50:01Z

/pulsarbot run-failure-checks

tisonkun · 2022-08-18T05:43:02Z

/pulsarbot run-failure-checks

tisonkun · 2022-08-18T06:12:00Z

All required tests pass now.

skyleaworlder · 2022-08-18T07:01:38Z

I think packaging part is pretty good now! Thank you very much for fixing my previous pending errors!

tests/docker-images/latest-version-image/pom.xml

tests/docker-images/java-test-image/pom.xml

docker/pulsar/pom.xml

docker/pulsar-all/pom.xml

nodece · 2022-08-18T15:13:05Z

docker/pulsar/pom.xml

                </configuration>
              </execution>
              <execution>
-                <id>add-no-repo-and-latest</id>
+                <id>push-latest</id>


By the way, I notice that there are redundant configurations here, and I think we can simplify them.

<executions> <execution> <id>default</id> <phase>package</phase> <goals> <goal>build</goal> <goal>push</goal> </goals> <configuration> <images> <image> <name>${docker.organization}/pulsar</name> <build> <contextDir>${project.basedir}</contextDir> <tags> <tag>latest</tag> <tag>${project.version}</tag> </tags> </build> </image> </images> </configuration> </execution> </executions>

Then maybe we also need to update the https://github.com/apache/pulsar/blob/master/docker/publish.sh#L65

So like using ${docker.organization}/pulsar:latest instead of pulsar:latest.

There needs the PMC confirmation, I don't know how to publish this image when they release the Pulsar.

This seems reasonable. I'm glad to integrate with this change if @sijie @merlimat @codelipenghui can confirm.

I found the original logic was introduced at #4705

We don't need to change the image publish process

Please see: https://github.com/apache/pulsar/blob/master/wiki/release/release-process.md#10-publish-docker-images

@codelipenghui I checked this process. Using the above config, we need to change the docker tag in publish.sh.

@nodece I think the point here is that it's possible that the release manager doesn't have the permission to push images to apachepulasr org. Thus we leave the ability to push to another org and ask the maintainer to sync. In this case, we generate the image with no repository for adding org when publish.sh.

Based on this information, I tend to keep the logic as is and if you have further improvement idea, we can start a new thread to discuss it.

Ok, we can make a new PR to improve that.

docker/pulsar-all/pom.xml

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2022-08-19T01:10:49Z

Thanks for your detailed review @nodece! Comments addressed or replied.

nodece · 2022-08-19T03:23:53Z

Thanks for your detailed review @nodece! Comments addressed or replied.

LGTM, I will approve this PR after add-no-repo is solved.

tisonkun · 2022-08-19T15:27:54Z

/pulsarbot run-failure-checks

Signed-off-by: tison <wander4096@gmail.com>

tisonkun · 2022-08-22T04:58:07Z

/pulsarbot run-failure-checks

nodece

LGTM

- #17148 changes aren't in branch-2.11 - sync pulsar_ci_tool.sh with #17770

…e#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com>

* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3) * PIP-155: Removed Python 2 support (apache#15376) * Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <mmarshall@apache.org> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][docker] Switch to Temurin JDK (apache#17129) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 4378856) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Support ARM64-based docker images (apache#17733) (cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * PIP-209: Removed C++/Python clients from main repo (apache#17881) * PIP-209: Removed C++/Python clients from main repo * Removed python directory from Docekrfile * Fixed python client version argument scoping * Fixed handling of pulsar.functions.serde (cherry picked from commit f3c547b) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][build] Avoid building image multiple times (apache#17208) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 79a97a9) * [improve] Allow to build and push multi-arch Docker images (apache#19432) Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zixuan Liu <nodeces@gmail.com> Co-authored-by: tison <wander4096@gmail.com> (cherry picked from commit 4190e40) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix publish image script (apache#20305) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 94c7bf3) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435) Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit d7f3558) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Adapt to Python client to be compatible with ARM arch Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550) (cherry picked from commit 05f7e62) * [improve][misc] Include native epoll library for Netty for arm64 From apache#22319 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][misc] Rename all shaded Netty native libraries From apache#22415 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [cleanup][build] Cleanup -Ddocker.nocache=true Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix ubuntu mirror Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix license Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Downgrade docker-maven to 0.43.3 Signed-off-by: Zixuan Liu <nodeces@gmail.com> --------- Signed-off-by: Zixuan Liu <nodeces@gmail.com> Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Matteo Merli <mmerli@apache.org> Co-authored-by: Michael Marshall <mmarshall@apache.org> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org>

tisonkun added 7 commits August 17, 2022 22:39

remove docker-info classifier

ea7f778

Signed-off-by: tison <wander4096@gmail.com>

migrate pulsar-docker-image to docker-maven-plugin

306f2e2

Signed-off-by: tison <wander4096@gmail.com>

migrate pulsar-all-docker-image to docker-maven-plugin

8855363

Signed-off-by: tison <wander4096@gmail.com>

dependency pom type

02898e0

Signed-off-by: tison <wander4096@gmail.com>

only retain build order

85e3aaf

Signed-off-by: tison <wander4096@gmail.com>

migrate test images to docker-maven-plugin

17e33aa

Signed-off-by: tison <wander4096@gmail.com>

cleanup property

f6141c7

Signed-off-by: tison <wander4096@gmail.com>

tisonkun marked this pull request as ready for review August 18, 2022 02:19

github-actions bot added the doc-not-needed Your PR changes do not impact docs label Aug 18, 2022

update squash property name

dc7392c

Signed-off-by: tison <wander4096@gmail.com>

tisonkun mentioned this pull request Aug 18, 2022

[improve][docker] Switch to Temurin JDK #17129

Merged

4 tasks

Merge branch 'master' into docker-maven-plugin

5037bb1

hezhangjian approved these changes Aug 18, 2022

View reviewed changes

nodece requested changes Aug 18, 2022

View reviewed changes

nodece requested review from lhotari, nicoloboschi, michaeljmarshall, codelipenghui and merlimat August 18, 2022 15:22

retain imagePullPolicy and noCache

25b640d

Signed-off-by: tison <wander4096@gmail.com>

tisonkun removed request for merlimat, lhotari and codelipenghui August 19, 2022 01:17

tisonkun requested review from nodece and removed request for nicoloboschi and michaeljmarshall August 19, 2022 01:17

Merge branch 'master' into docker-maven-plugin

8c067e4

nodece requested review from lhotari, nicoloboschi, michaeljmarshall, codelipenghui, Technoboy- and eolivelli August 19, 2022 10:04

Merge branch 'master' into docker-maven-plugin

9dd032c

tisonkun added 3 commits August 22, 2022 08:48

Merge branch 'master' into docker-maven-plugin

833ce56

must pull if not present

77f3fd8

Signed-off-by: tison <wander4096@gmail.com>

Merge branch 'master' into docker-maven-plugin

a2c0e1a

nodece approved these changes Aug 22, 2022

View reviewed changes

codelipenghui added this to the 2.12.0 milestone Aug 22, 2022

codelipenghui assigned tisonkun Aug 22, 2022

codelipenghui approved these changes Aug 22, 2022

View reviewed changes

codelipenghui merged commit a68b58d into apache:master Aug 22, 2022

tisonkun deleted the docker-maven-plugin branch August 22, 2022 08:15

tisonkun mentioned this pull request Aug 23, 2022

[improve][build] Avoid building image multiple times #17208

Merged

4 tasks

lhotari added a commit that referenced this pull request Sep 21, 2022

[fix][CI] Fix issue in cherry-picking CI changes

9221575

- #17148 changes aren't in branch-2.11 - sync pulsar_ci_tool.sh with #17770

nodece pushed a commit to nodece/pulsar that referenced this pull request Apr 28, 2024

[refactor][ci] Build the docker image with docker-maven-plugin (apach…

90e1e70

…e#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[refactor][ci] Build the docker image with docker-maven-plugin #17148

[refactor][ci] Build the docker image with docker-maven-plugin #17148

tisonkun commented Aug 17, 2022 •

edited

Loading

tisonkun commented Aug 18, 2022 •

edited

Loading

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

skyleaworlder commented Aug 18, 2022

nodece Aug 18, 2022

nodece Aug 18, 2022 •

edited

Loading

tisonkun Aug 19, 2022

codelipenghui Aug 22, 2022

nodece Aug 22, 2022

tisonkun Aug 22, 2022

nodece Aug 22, 2022

tisonkun commented Aug 19, 2022

nodece commented Aug 19, 2022

tisonkun commented Aug 19, 2022

tisonkun commented Aug 22, 2022

nodece left a comment

[refactor][ci] Build the docker image with docker-maven-plugin #17148

[refactor][ci] Build the docker image with docker-maven-plugin #17148

Conversation

tisonkun commented Aug 17, 2022 • edited Loading

tisonkun commented Aug 18, 2022 • edited Loading

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

tisonkun commented Aug 18, 2022

skyleaworlder commented Aug 18, 2022

nodece Aug 18, 2022

Choose a reason for hiding this comment

nodece Aug 18, 2022 • edited Loading

Choose a reason for hiding this comment

tisonkun Aug 19, 2022

Choose a reason for hiding this comment

codelipenghui Aug 22, 2022

Choose a reason for hiding this comment

nodece Aug 22, 2022

Choose a reason for hiding this comment

tisonkun Aug 22, 2022

Choose a reason for hiding this comment

nodece Aug 22, 2022

Choose a reason for hiding this comment

tisonkun commented Aug 19, 2022

nodece commented Aug 19, 2022

tisonkun commented Aug 19, 2022

tisonkun commented Aug 22, 2022

nodece left a comment

Choose a reason for hiding this comment

tisonkun commented Aug 17, 2022 •

edited

Loading

tisonkun commented Aug 18, 2022 •

edited

Loading

nodece Aug 18, 2022 •

edited

Loading