Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[refactor][ci] Build the docker image with docker-maven-plugin #17148

Merged
merged 15 commits into from
Aug 22, 2022
Merged

[refactor][ci] Build the docker image with docker-maven-plugin #17148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
ea7f778
remove docker-info classifier
tisonkun Aug 17, 2022
306f2e2
migrate pulsar-docker-image to docker-maven-plugin
tisonkun Aug 17, 2022
8855363
migrate pulsar-all-docker-image to docker-maven-plugin
tisonkun Aug 17, 2022
02898e0
dependency pom type
tisonkun Aug 18, 2022
85e3aaf
only retain build order
tisonkun Aug 18, 2022
17e33aa
migrate test images to docker-maven-plugin
tisonkun Aug 18, 2022
f6141c7
cleanup property
tisonkun Aug 18, 2022
dc7392c
update squash property name
tisonkun Aug 18, 2022
5037bb1
Merge branch 'master' into docker-maven-plugin
tisonkun Aug 18, 2022
25b640d
retain imagePullPolicy and noCache
tisonkun Aug 19, 2022
8c067e4
Merge branch 'master' into docker-maven-plugin
tisonkun Aug 19, 2022
9dd032c
Merge branch 'master' into docker-maven-plugin
tisonkun Aug 19, 2022
833ce56
Merge branch 'master' into docker-maven-plugin
tisonkun Aug 22, 2022
77f3fd8
must pull if not present
tisonkun Aug 22, 2022
a2c0e1a
Merge branch 'master' into docker-maven-plugin
tisonkun Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/tune-runner-vm/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ runs:
sudo systemctl stop walinuxagent.service || true

# enable docker experimental mode which is
# required for using "docker build --squash" / "-Ddockerfile.build.squash=true"
# required for using "docker build --squash" / "-Ddocker.squash=true"
daemon_json="$(sudo cat /etc/docker/daemon.json | jq '.experimental = true')"
echo "$daemon_json" | sudo tee /etc/docker/daemon.json
# restart docker daemon
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pulsar-ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ jobs:
run: |
# build docker image
mvn -B -am -pl tests/docker-images/java-test-image install -Pcore-modules,-main,integrationTests,docker \
-Dmaven.test.skip=true -Ddockerfile.build.squash=true -DskipSourceReleaseAssembly=true \
-Dmaven.test.skip=true -Ddocker.squash=true -DskipSourceReleaseAssembly=true \
-Dspotbugs.skip=true -Dlicense.skip=true -Dcheckstyle.skip=true -Drat.skip=true

- name: save docker image apachepulsar/java-test-image:latest to Github artifact cache
Expand Down Expand Up @@ -576,7 +576,7 @@ jobs:
# build docker image
# include building of Pulsar SQL, Connectors, Offloaders and server distros
mvn -B -am -pl pulsar-sql/presto-distribution,distribution/io,distribution/offloaders,distribution/server,tests/docker-images/latest-version-image install \
-Pmain,docker -Dmaven.test.skip=true -Ddockerfile.build.squash=true \
-Pmain,docker -Dmaven.test.skip=true -Ddocker.squash=true \
-Dspotbugs.skip=true -Dlicense.skip=true -Dcheckstyle.skip=true -Drat.skip=true

- name: save docker image apachepulsar/pulsar-test-latest-version:latest to Github artifact cache
Expand Down
2 changes: 1 addition & 1 deletion build/build_java_test_image.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ SQUASH_PARAM=""
# check if docker experimental mode is enabled which is required for
# using "docker build --squash" for squashing all intermediate layers of the build to a single layer
if [[ "$(docker version -f '{{.Server.Experimental}}' 2>/dev/null)" == "true" ]]; then
SQUASH_PARAM="-Ddockerfile.build.squash=true"
SQUASH_PARAM="-Ddocker.squash=true"
fi
mvn -am -pl tests/docker-images/java-test-image -Pcore-modules,-main,integrationTests,docker \
-Dmaven.test.skip=true -DskipSourceReleaseAssembly=true -Dspotbugs.skip=true -Dlicense.skip=true $SQUASH_PARAM \
Expand Down
85 changes: 55 additions & 30 deletions docker/pulsar-all/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,13 +68,24 @@
<profiles>
<profile>
<id>docker</id>
<properties>
<docker.buildArg.PULSAR_IO_DIR>target/apache-pulsar-io-connectors-${project.version}-bin</docker.buildArg.PULSAR_IO_DIR>
<docker.buildArg.PULSAR_OFFLOADER_TARBALL>target/pulsar-offloader-distribution-${project.version}-bin.tar.gz</docker.buildArg.PULSAR_OFFLOADER_TARBALL>
</properties>
<!-- include the docker image only when docker profile is active -->
<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-docker-image</artifactId>
<version>${project.version}</version>
<classifier>docker-info</classifier>
<type>pom</type>
<scope>provided</scope>
<exclusions>
<exclusion>
<groupId>*</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
<build>
Expand Down Expand Up @@ -119,57 +130,71 @@
</executions>
</plugin>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>${dockerfile-maven.version}</version>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
<executions>
<execution>
<id>default</id>
<phase>package</phase>
<goals>
<goal>build</goal>
</goals>
</execution>
<execution>
<id>add-no-repo-and-version</id>
<goals>
<goal>tag</goal>
</goals>
<configuration>
<repository>pulsar-all</repository>
<tag>${project.version}</tag>
<images>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
<image>
<name>${docker.organization}/pulsar-all</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
<tag>${project.version}</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
<execution>
<id>add-no-repo-and-latest</id>
<id>push-latest</id>
<goals>
<goal>tag</goal>
<goal>push</goal>
</goals>
<configuration>
<repository>pulsar-all</repository>
<tag>latest</tag>
<images>
<image>
<name>${docker.organization}/pulsar-all</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
<execution>
<id>tag-and-push-latest</id>
<id>add-no-repo</id>
<phase>package</phase>
<goals>
<goal>tag</goal>
<goal>push</goal>
<goal>build</goal>
</goals>
<configuration>
<repository>${docker.organization}/pulsar-all</repository>
<tag>latest</tag>
<images>
<image>
<name>pulsar-all</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
<tag>${project.version}</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
</executions>
<configuration>
<repository>${docker.organization}/pulsar-all</repository>
<pullNewerImage>false</pullNewerImage>
<tag>${project.version}</tag>
<buildArgs>
<PULSAR_IO_DIR>target/apache-pulsar-io-connectors-${project.version}-bin</PULSAR_IO_DIR>
<PULSAR_OFFLOADER_TARBALL>target/pulsar-offloader-distribution-${project.version}-bin.tar.gz</PULSAR_OFFLOADER_TARBALL>
</buildArgs>
</configuration>
</plugin>
</plugins>
</build>
Expand Down
76 changes: 47 additions & 29 deletions docker/pulsar/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@
<profiles>
<profile>
<id>docker</id>
<properties>
<docker.buildArg.PULSAR_TARBALL>target/pulsar-server-distribution-${project.version}-bin.tar.gz</docker.buildArg.PULSAR_TARBALL>
<docker.buildArg.UBUNTU_MIRROR>${env.UBUNTU_MIRROR}</docker.buildArg.UBUNTU_MIRROR>
</properties>
<build>
<plugins>
<!-- build Python client, copy the wheel file and then build docker image -->
Expand Down Expand Up @@ -103,57 +107,71 @@
</executions>
</plugin>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>${dockerfile-maven.version}</version>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
<executions>
<execution>
<id>default</id>
<phase>package</phase>
<goals>
<goal>build</goal>
</goals>
</execution>
<execution>
<id>add-no-repo-and-version</id>
<goals>
<goal>tag</goal>
</goals>
<configuration>
<repository>pulsar</repository>
<tag>${project.version}</tag>
<images>
<image>
<name>${docker.organization}/pulsar</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
<tag>${project.version}</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
<execution>
<id>add-no-repo-and-latest</id>
<id>push-latest</id>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, I notice that there are redundant configurations here, and I think we can simplify them.

<executions>
      <execution>
        <id>default</id>
        <phase>package</phase>
        <goals>
          <goal>build</goal>
          <goal>push</goal>
        </goals>
        <configuration>
          <images>
            <image>
              <name>${docker.organization}/pulsar</name>
              <build>
                <contextDir>${project.basedir}</contextDir>
                <tags>
                  <tag>latest</tag>
                  <tag>${project.version}</tag>
                </tags>
              </build>
            </image>
          </images>
        </configuration>
      </execution>
</executions>

Then maybe we also need to update the https://github.com/apache/pulsar/blob/master/docker/publish.sh#L65

So like using ${docker.organization}/pulsar:latest instead of pulsar:latest.

Copy link
Member

@nodece nodece Aug 18, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There needs the PMC confirmation, I don't know how to publish this image when they release the Pulsar.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems reasonable. I'm glad to integrate with this change if @sijie @merlimat @codelipenghui can confirm.

I found the original logic was introduced at #4705

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to change the image publish process

Please see: https://github.com/apache/pulsar/blob/master/wiki/release/release-process.md#10-publish-docker-images

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codelipenghui I checked this process. Using the above config, we need to change the docker tag in publish.sh.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nodece I think the point here is that it's possible that the release manager doesn't have the permission to push images to apachepulasr org. Thus we leave the ability to push to another org and ask the maintainer to sync. In this case, we generate the image with no repository for adding org when publish.sh.

Based on this information, I tend to keep the logic as is and if you have further improvement idea, we can start a new thread to discuss it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we can make a new PR to improve that.

<goals>
<goal>tag</goal>
<goal>push</goal>
</goals>
<configuration>
<repository>pulsar</repository>
<tag>latest</tag>
<images>
<image>
<name>${docker.organization}/pulsar</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
<execution>
<id>tag-and-push-latest</id>
<id>add-no-repo</id>
<phase>package</phase>
<goals>
<goal>tag</goal>
<goal>push</goal>
<goal>build</goal>
</goals>
<configuration>
<repository>${docker.organization}/pulsar</repository>
<tag>latest</tag>
<images>
<image>
<name>pulsar</name>
<build>
<contextDir>${project.basedir}</contextDir>
<tags>
<tag>latest</tag>
<tag>${project.version}</tag>
</tags>
</build>
</image>
</images>
</configuration>
</execution>
</executions>
<configuration>
<repository>${docker.organization}/pulsar</repository>
<pullNewerImage>false</pullNewerImage>
<tag>${project.version}</tag>
<buildArgs>
<PULSAR_TARBALL>target/pulsar-server-distribution-${project.version}-bin.tar.gz</PULSAR_TARBALL>
<UBUNTU_MIRROR>${env.UBUNTU_MIRROR}</UBUNTU_MIRROR>
</buildArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
Expand Down
8 changes: 7 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,8 @@ flexible messaging model and an intuitive client API.</description>
<reflections.version>0.9.11</reflections.version>
<swagger.version>1.6.2</swagger.version>
<puppycrawl.checkstyle.version>8.37</puppycrawl.checkstyle.version>
<dockerfile-maven.version>1.4.13</dockerfile-maven.version>
<docker-maven.version>0.40.2</docker-maven.version>
<docker.verbose>true</docker.verbose>
<typetools.version>0.5.0</typetools.version>
<protobuf3.version>3.19.2</protobuf3.version>
<protoc3.version>${protobuf3.version}</protoc3.version>
Expand Down Expand Up @@ -1892,6 +1893,11 @@ flexible messaging model and an intuitive client API.</description>
<artifactId>properties-maven-plugin</artifactId>
<version>${properties-maven-plugin.version}</version>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<version>${docker-maven.version}</version>
</plugin>
</plugins>
</pluginManagement>
<extensions>
Expand Down
41 changes: 19 additions & 22 deletions tests/docker-images/java-test-image/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,10 @@
<profiles>
<profile>
<id>docker</id>
<properties>
<docker.buildArg.PULSAR_TARBALL>target/pulsar-server-distribution-bin.tar.gz</docker.buildArg.PULSAR_TARBALL>
<docker.buildArg.UBUNTU_MIRROR>${env.UBUNTU_MIRROR}</docker.buildArg.UBUNTU_MIRROR>
</properties>
<activation>
<property>
<name>integrationTests</name>
Expand Down Expand Up @@ -135,39 +139,32 @@
</executions>
</plugin>
<plugin>
<groupId>com.spotify</groupId>
<artifactId>dockerfile-maven-plugin</artifactId>
<version>${dockerfile-maven.version}</version>
<groupId>io.fabric8</groupId>
<artifactId>docker-maven-plugin</artifactId>
<executions>
<execution>
<id>default</id>
<phase>package</phase>
<goals>
<goal>build</goal>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
</goals>
</execution>
<execution>
<id>add-latest-tag</id>
<phase>package</phase>
<goals>
<goal>tag</goal>
</goals>
<configuration>
<repository>${docker.organization}/java-test-image</repository>
<tag>latest</tag>
<images>
<image>
<name>${docker.organization}/java-test-image</name>
<build>
<contextDir>${project.basedir}</contextDir>
tisonkun marked this conversation as resolved.
Show resolved Hide resolved
<tags>
<tag>latest</tag>
<tag>${project.version}</tag>
</tags>
<noCache>true</noCache>
</build>
</image>
</images>
</configuration>
</execution>
</executions>
<configuration>
<repository>${docker.organization}/java-test-image</repository>
<tag>${project.version}</tag>
<pullNewerImage>false</pullNewerImage>
<noCache>true</noCache>
<buildArgs>
<PULSAR_TARBALL>target/pulsar-server-distribution-bin.tar.gz</PULSAR_TARBALL>
<UBUNTU_MIRROR>${env.UBUNTU_MIRROR}</UBUNTU_MIRROR>
</buildArgs>
</configuration>
</plugin>
</plugins>
</build>
Expand Down
Loading