WIP

merlimat · May 7, 2024 · eef763c · eef763c
1 parent 09364a9
commit eef763c
Show file tree

Hide file tree

Showing 12 changed files with 158 additions and 159 deletions.
diff --git a/README.md b/README.md
@@ -21,7 +21,7 @@
 
 ![logo](https://pulsar.apache.org/img/pulsar.svg)
 
-[![docker pull](https://img.shields.io/docker/pulls/apachepulsar/pulsar-all.svg)](https://hub.docker.com/r/apachepulsar/pulsar)
+[![docker pull](https://img.shields.io/docker/pulls/apachepulsar/pulsar.svg)](https://hub.docker.com/r/apachepulsar/pulsar)
 [![contributors](https://img.shields.io/github/contributors-anon/apache/pulsar)](https://github.com/apache/pulsar/graphs/contributors)
 [![last commit](https://img.shields.io/github/last-commit/apache/pulsar)](https://github.com/apache/pulsar/commits/master)
 [![release](https://img.shields.io/github/v/release/apache/pulsar?sort=semver)](https://pulsar.apache.org/download/)
@@ -200,13 +200,13 @@ Here are some general instructions for building custom docker images:
 * Java 11 is the recommended JDK version in `branch-2.8`, `branch-2.9` and `branch-2.10`.
 * Java 17 is the recommended JDK version in `master`.
 
-The following command builds the docker images `apachepulsar/pulsar-all:latest` and `apachepulsar/pulsar:latest`:
+The following command builds the docker image `apachepulsar/pulsar:latest`:
 
 ```bash
 mvn clean install -DskipTests
 # setting DOCKER_CLI_EXPERIMENTAL=enabled is required in some environments with older docker versions
 export DOCKER_CLI_EXPERIMENTAL=enabled
-mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests
+mvn package -Pdocker,-main -am -pl docker/pulsar -DskipTests
 ```
 
 After the images are built, they can be tagged and pushed to your custom repository. Here's an example of a bash script that tags the docker images with the current version and git revision and pushes them to `localhost:32000/apachepulsar`.
@@ -217,8 +217,6 @@ pulsar_version=$(mvn initialize help:evaluate -Dexpression=project.version -pl .
 gitrev=$(git rev-parse HEAD | colrm 10)
 tag="${pulsar_version}-${gitrev}"
 echo "Using tag $tag"
-docker tag apachepulsar/pulsar-all:latest ${image_repo_and_project}/pulsar-all:$tag
-docker push ${image_repo_and_project}/pulsar-all:$tag
 docker tag apachepulsar/pulsar:latest ${image_repo_and_project}/pulsar:$tag
 docker push ${image_repo_and_project}/pulsar:$tag
 ```

diff --git a/build/run_unit_group.sh b/build/run_unit_group.sh
@@ -167,7 +167,7 @@ function test_group_proxy() {
 
 function test_group_other() {
   mvn_test --clean --install \
-           -pl '!org.apache.pulsar:distribution,!org.apache.pulsar:pulsar-offloader-distribution,!org.apache.pulsar:pulsar-server-distribution,!org.apache.pulsar:pulsar-io-distribution,!org.apache.pulsar:pulsar-all-docker-image' \
+           -pl '!org.apache.pulsar:distribution,!org.apache.pulsar:pulsar-offloader-distribution,!org.apache.pulsar:pulsar-server-distribution,!org.apache.pulsar:pulsar-io-distribution,!org.apache.pulsar:pulsar-functions-python-docker-image' \
            -PskipTestsForUnitGroupOther -DdisableIoMainProfile=true -DskipIntegrationTests \
            -Dexclude='**/ManagedLedgerTest.java,
                    **/OffloadersCacheTest.java

diff --git a/docker/README.md b/docker/README.md
@@ -24,7 +24,8 @@
 The Apache Pulsar community produces 2 docker images with each official release.
 
 * `apachepulsar/pulsar` - contains the necessary components for a working Pulsar cluster
-* `apachepulsar/pulsar-all` - extends the `apachepulsar/pulsar` image by adding many Pulsar connectors and offloaders
+* `apachepulsar/pulsar-functions-python` - extends the `apachepulsar/pulsar` image by adding the Python
+   dependencies required to run Pulsar Functions with Python runtime. 
 
 Since the 2.10.0 release, these docker images run as an unnamed, non-root user that is also part of the root group, by
 default. This was done to increase container security. The user is part of the root group to ensure that the container
@@ -37,44 +38,27 @@ Note that you first must build the project in order to have the right dependenci
 
 ## Building Derivative Custom Images
 
-If you find the `apachepulsar/pulsar-all` docker image too large, but you want to use a connector or an offloader,
-you can easily build an image with a curated list of connectors or offloaders based on the official Apache Pulsar
+You can easily build an image with a curated list of connectors or offloaders based on the official Apache Pulsar
 images. You can use the following sample docker image as a guide:
 
 ```Dockerfile
 ARG VERSION
 
-# Load the pulsar-all image as a builder image
-FROM apachepulsar/pulsar-all:${VERSION} as pulsar-all
-
 FROM apachepulsar/pulsar:${VERSION}
 
 # Add the cassandra connector
-COPY --from=pulsar-all /pulsar/connectors/pulsar-io-cassandra-*.nar /pulsar/connectors
-
-# Add the jcloud offloader
-COPY --from=pulsar-all /pulsar/connectors/tiered-storage-jcloud-*.nar /pulsar/offloaders
+RUN mkdir -p connectors && \
+    cd connectors && \
+    wget https://downloads.apache.org/pulsar/pulsar-${VERSION}/connectors/pulsar-io-cassandra-${VERSION}.nar 
 ```
 
-NOTE: the above example uses a wildcard in the `COPY` commands because argument expansion does not work for `COPY`.
-
 Assuming that you have the above `Dockerfile` in your local directory and are running docker on your local host, you can
-run the following command to build a custom image with the cassandra connector and the jcloud offloader.
+run the following command to build a custom image with the cassandra connector.
 
 ```shell
 docker build --build-arg VERSION=2.9.1 -t pulsar-custom:2.9.1 .
 ```
 
-For reference, here are the sizes of the official 2.9.1 docker images and the custom image built from the above
-`Dockerfile`:
-
-| REPOSITORY              | TAG   | SIZE   |
-| :---------------------- | :---- | :----- |
-| apachepulsar/pulsar     | 2.9.1 | 1.59GB |
-| apachepulsar/pulsar-all | 2.9.1 | 3.44GB |
-| pulsar-custom           | 2.9.1 | 1.6GB  |
-
-
 ## Troubleshooting non-root containers
 
 Troubleshooting is harder because the docker image runs as a non-root user. For example, a non-root user won't be able

diff --git a/docker/pom.xml b/docker/pom.xml
@@ -58,7 +58,7 @@
       </activation>
       <modules>
         <module>pulsar</module>
-        <module>pulsar-all</module>
+        <module>pulsar-functions-python</module>
       </modules>
       <build>
         <plugins>

diff --git a/docker/publish.sh b/docker/publish.sh
@@ -67,19 +67,19 @@ set -e
 
 if [[ "$GIT_BRANCH" == "master" ]]; then
   docker tag apachepulsar/pulsar:${IMAGE_TAG} ${docker_registry_org}/pulsar:latest
-  docker tag apachepulsar/pulsar-all:${IMAGE_TAG} ${docker_registry_org}/pulsar-all:latest
+  docker tag apachepulsar/pulsar-functions-python:${IMAGE_TAG} ${docker_registry_org}/pulsar-functions-python:latest
 fi
 
 docker tag apachepulsar/pulsar:${IMAGE_TAG} ${docker_registry_org}/pulsar:$MVN_VERSION
-docker tag apachepulsar/pulsar-all:${IMAGE_TAG} ${docker_registry_org}/pulsar-all:$MVN_VERSION
+docker tag apachepulsar/pulsar-functions-python:${IMAGE_TAG} ${docker_registry_org}/pulsar-functions-python:$MVN_VERSION
 
 # Push all images and tags
 if [[ "$GIT_BRANCH" == "master" ]]; then
   docker push ${docker_registry_org}/pulsar:latest
-  docker push ${docker_registry_org}/pulsar-all:latest
+  docker push ${docker_registry_org}/pulsar-functions-python:latest
 fi
 
 docker push ${docker_registry_org}/pulsar:$MVN_VERSION
-docker push ${docker_registry_org}/pulsar-all:$MVN_VERSION
+docker push ${docker_registry_org}/pulsar-functions-python:$MVN_VERSION
 
 echo "Finished pushing images to ${docker_registry_org}"
diff --git a/docker/pulsar-all/Dockerfile → docker/pulsar-functions-python/Dockerfile b/docker/pulsar-all/Dockerfile → docker/pulsar-functions-python/Dockerfile
@@ -17,16 +17,38 @@
 # under the License.
 #
 
+ARG PULSAR_CLIENT_PYTHON_VERSION
 ARG PULSAR_IMAGE
-FROM busybox as pulsar-all
 
-ARG PULSAR_IO_DIR
-ARG PULSAR_OFFLOADER_TARBALL
+## Create stage to build the Python dependencies
+## Since it needs to have GCC available, we're doing it in a different layer
+FROM alpine:3.19 AS python-deps
 
-ADD ${PULSAR_IO_DIR} /connectors
-ADD ${PULSAR_OFFLOADER_TARBALL} /
-RUN mv /apache-pulsar-offloaders-*/offloaders /offloaders
+RUN apk add --no-cache \
+       bash \
+       python3-dev \
+       g++ \
+       musl-dev \
+       libffi-dev \
+       py3-pip \
+       py3-grpcio \
+       py3-yaml
 
+RUN pip3 install --break-system-packages \
+        kazoo
+
+ARG PULSAR_CLIENT_PYTHON_VERSION
+RUN pip3 install --break-system-packages \
+    pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION}
+
+# Main image stage
+ARG PULSAR_IMAGE
 FROM $PULSAR_IMAGE
-COPY --from=pulsar-all /connectors /pulsar/connectors
-COPY --from=pulsar-all /offloaders /pulsar/offloaders
+
+# Switch to root user to install PIP package
+USER root
+RUN apk add --no-cache py3-pip
+USER 10000
+
+# Copy Python depedencies from the other stage
+COPY --from=python-deps /usr/lib/python3.11/site-packages /usr/lib/python3.11/site-packages
diff --git a/docker/pulsar-all/pom.xml → docker/pulsar-functions-python/pom.xml b/docker/pulsar-all/pom.xml → docker/pulsar-functions-python/pom.xml
@@ -26,42 +26,16 @@
     <version>3.3.0-SNAPSHOT</version>
   </parent>
   <modelVersion>4.0.0</modelVersion>
-  <artifactId>pulsar-all-docker-image</artifactId>
-  <name>Apache Pulsar :: Docker Images :: Pulsar Latest Version (Include All Components)</name>
+  <artifactId>pulsar-functions-python-docker-image</artifactId>
+  <name>Apache Pulsar :: Docker Images :: Pulsar Functions Python</name>
   <packaging>pom</packaging>
 
   <dependencies>
     <dependency>
       <groupId>${project.groupId}</groupId>
-      <artifactId>pulsar-io-distribution</artifactId>
-      <version>${project.parent.version}</version>
+      <artifactId>pulsar-docker-image</artifactId>
+      <version>${project.version}</version>
       <type>pom</type>
-      <scope>provided</scope>
-      <exclusions>
-        <exclusion>
-          <groupId>*</groupId>
-          <artifactId>*</artifactId>
-        </exclusion>
-      </exclusions>
-    </dependency>
-    <dependency>
-      <groupId>io.perfmark</groupId>
-      <artifactId>perfmark-api</artifactId>
-      <scope>runtime</scope>
-    </dependency>
-    <dependency>
-      <groupId>${project.groupId}</groupId>
-      <artifactId>pulsar-offloader-distribution</artifactId>
-      <version>${project.parent.version}</version>
-      <classifier>bin</classifier>
-      <type>tar.gz</type>
-      <scope>provided</scope>
-      <exclusions>
-        <exclusion>
-          <groupId>*</groupId>
-          <artifactId>*</artifactId>
-        </exclusion>
-      </exclusions>
     </dependency>
   </dependencies>
 
@@ -97,45 +71,6 @@
       </dependencies>
       <build>
         <plugins>
-          <plugin>
-            <artifactId>maven-resources-plugin</artifactId>
-            <executions>
-              <execution>
-                <id>copy-resources</id>
-                <phase>generate-resources</phase>
-                <goals>
-                  <goal>copy-resources</goal>
-                </goals>
-                <configuration>
-                  <outputDirectory>${basedir}/target/apache-pulsar-io-connectors-${project.version}-bin</outputDirectory>
-                  <resources>
-                    <resource>
-                      <directory>${basedir}/../../distribution/io/target/apache-pulsar-io-connectors-${project.version}-bin</directory>
-                      <filtering>false</filtering>
-                    </resource>
-                  </resources>
-                </configuration>
-              </execution>
-            </executions>
-          </plugin>
-          <plugin>
-            <groupId>org.apache.maven.plugins</groupId>
-            <artifactId>maven-dependency-plugin</artifactId>
-            <executions>
-              <execution>
-                <id>copy-offloader-tarball</id>
-                <goals>
-                  <goal>copy-dependencies</goal>
-                </goals>
-                <phase>generate-resources</phase>
-                <configuration>
-                  <outputDirectory>${project.build.directory}/</outputDirectory>
-                  <includeArtifactIds>pulsar-offloader-distribution</includeArtifactIds>
-                  <excludeTransitive>true</excludeTransitive>
-                </configuration>
-              </execution>
-            </executions>
-          </plugin>
           <plugin>
             <groupId>io.fabric8</groupId>
             <artifactId>docker-maven-plugin</artifactId>
@@ -150,17 +85,16 @@
                 <configuration>
                   <images>
                     <image>
-                      <name>${docker.organization}/pulsar-all</name>
+                      <name>${docker.organization}/pulsar-functions-python</name>
                       <build>
                         <contextDir>${project.basedir}</contextDir>
                         <tags>
                           <tag>latest</tag>
                           <tag>${project.version}-${git.commit.id.abbrev}</tag>
                         </tags>
                         <args>
-                          <PULSAR_IO_DIR>target/apache-pulsar-io-connectors-${project.version}-bin</PULSAR_IO_DIR>
-                          <PULSAR_OFFLOADER_TARBALL>target/pulsar-offloader-distribution-${project.version}-bin.tar.gz</PULSAR_OFFLOADER_TARBALL>
                           <PULSAR_IMAGE>${docker.organization}/pulsar:${project.version}-${git.commit.id.abbrev}</PULSAR_IMAGE>
+                          <PULSAR_CLIENT_PYTHON_VERSION>${pulsar.client.python.version}</PULSAR_CLIENT_PYTHON_VERSION>
                         </args>
                         <buildx>
                           <platforms>

diff --git a/docker/pulsar/Dockerfile b/docker/pulsar/Dockerfile
@@ -48,27 +48,6 @@ RUN for SUBDIRECTORY in conf data download logs instances/deps; do \
 RUN chmod -R g+rx /pulsar/bin
 RUN chmod -R o+rx /pulsar
 
-## Create 2nd stage to build the Python dependencies
-## Since it needs to have GCC available, we're doing it in a different layer
-FROM alpine:3.19 AS python-deps
-
-RUN apk add --no-cache \
-       bash \
-       python3-dev \
-       g++ \
-       musl-dev \
-       libffi-dev \
-       py3-pip \
-       py3-grpcio \
-       py3-yaml
-
-RUN pip3 install --break-system-packages \
-        kazoo
-
-ARG PULSAR_CLIENT_PYTHON_VERSION
-RUN pip3 install --break-system-packages \
-    pulsar-client[all]==${PULSAR_CLIENT_PYTHON_VERSION}
-
 
 ###  Create one stage to include JVM distribution
 FROM alpine AS jvm
@@ -87,15 +66,13 @@ RUN echo networkaddress.cache.negative.ttl=1 >> /opt/jvm/conf/security/java.secu
 FROM apachepulsar/glibc-base:2.38 as glibc
 
 ## Create final stage from Alpine image
-## and add OpenJDK and Python dependencies (for Pulsar functions)
 FROM alpine:3.19.1
 ENV LANG C.UTF-8
 
 # Install some utilities
 RUN apk add --no-cache \
             bash \
             python3 \
-            py3-pip \
             gcompat \
             ca-certificates \
             procps
@@ -107,12 +84,9 @@ RUN apk add --allow-untrusted --force-overwrite /root/packages/glibc-*.apk
 COPY --from=jvm /opt/jvm /opt/jvm
 ENV JAVA_HOME=/opt/jvm
 
-# The default is /pulsat/bin and cannot be written.
+# The default is /pulsar/bin and cannot be written.
 ENV PULSAR_PID_DIR=/pulsar/logs
 
-# Copy Python depedencies from the other stage
-COPY --from=python-deps /usr/lib/python3.11/site-packages /usr/lib/python3.11/site-packages
-
 ENV PULSAR_ROOT_LOGGER=INFO,CONSOLE
 
 COPY --from=pulsar /pulsar /pulsar

diff --git a/docker/pulsar/pom.xml b/docker/pulsar/pom.xml
@@ -81,7 +81,6 @@
                       <build>
                         <args>
                           <PULSAR_TARBALL>target/pulsar-server-distribution-${project.version}-bin.tar.gz</PULSAR_TARBALL>
-                          <PULSAR_CLIENT_PYTHON_VERSION>${pulsar.client.python.version}</PULSAR_CLIENT_PYTHON_VERSION>
                         </args>
                         <contextDir>${project.basedir}</contextDir>
                         <tags>

diff --git a/pulsar-bom/pom.xml b/pulsar-bom/pom.xml
@@ -182,7 +182,7 @@
       </dependency>
       <dependency>
         <groupId>org.apache.pulsar</groupId>
-        <artifactId>pulsar-all-docker-image</artifactId>
+        <artifactId>pulsar-functions-python-docker-image</artifactId>
         <version>${project.version}</version>
       </dependency>
       <dependency>
-Original file line number
+Diff line change
@@ Expand Up / @@ -81,7 +81,6 @@ @@
                           <build>
                             <args>
                               <PULSAR_TARBALL>target/pulsar-server-distribution-${project.version}-bin.tar.gz</PULSAR_TARBALL>
-                              <PULSAR_CLIENT_PYTHON_VERSION>${pulsar.client.python.version}</PULSAR_CLIENT_PYTHON_VERSION>
                             </args>
                             <contextDir>${project.basedir}</contextDir>
                             <tags>
@@ Expand Down @@