Pulsar branch-2.9 build failure in auditwheel running build-wheel-file-within-docker.sh

[samzph]

Description
Docker build fails in pulsar-client-cpp/docker/build-wheel-file-within-docker.sh with error on + auditwheel repair dist/pulsar_client-2.8.2-cp38-cp38-linux_x86_64.whl dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl

To Reproduce
Steps to reproduce the behavior:
git clone https://github.com/apache/pulsar.git
git checkout branch-2.9
Follow steps here:

mvn clean install -DskipTests
mvn package -Pdocker,-main -am -pl docker/pulsar-all -DskipTests

Expected behavior
Pulsar successfully builds docker images on latest version with Log4J 2.17+ to be tagged and pushed to a custom or local docker registry.

Trace

[100%] Built target _pulsar
make[2]: Leaving directory `/pulsar/pulsar-client-cpp'
/usr/local/bin/cmake -E cmake_progress_start /pulsar/pulsar-client-cpp/CMakeFiles 0
make[1]: Leaving directory `/pulsar/pulsar-client-cpp'
+ cd python
+ python setup.py bdist_wheel
2.9.1
pulsar-client
running bdist_wheel
running build
running build_py
copying pulsar/schema/definition.py -> build/lib.linux-x86_64-3.8/pulsar/schema
copying pulsar/schema/schema.py -> build/lib.linux-x86_64-3.8/pulsar/schema
running build_ext
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
copying build/lib.linux-x86_64-3.8/_pulsar.cpython-38-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/pulsar
copying build/lib.linux-x86_64-3.8/pulsar/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar
copying build/lib.linux-x86_64-3.8/pulsar/exceptions.py -> build/bdist.linux-x86_64/wheel/pulsar
creating build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/context.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/function.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
copying build/lib.linux-x86_64-3.8/pulsar/functions/serde.py -> build/bdist.linux-x86_64/wheel/pulsar/functions
creating build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/__init__.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/definition.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/schema.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
copying build/lib.linux-x86_64-3.8/pulsar/schema/schema_avro.py -> build/bdist.linux-x86_64/wheel/pulsar/schema
running install_egg_info
running egg_info
writing pulsar_client.egg-info/PKG-INFO
writing dependency_links to pulsar_client.egg-info/dependency_links.txt
writing requirements to pulsar_client.egg-info/requires.txt
writing top-level names to pulsar_client.egg-info/top_level.txt
reading manifest file 'pulsar_client.egg-info/SOURCES.txt'
writing manifest file 'pulsar_client.egg-info/SOURCES.txt'
Copying pulsar_client.egg-info to build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1.dist-info/WHEEL
creating 'dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding '_pulsar.cpython-38-x86_64-linux-gnu.so'
adding 'pulsar/__init__.py'
adding 'pulsar/exceptions.py'
adding 'pulsar/functions/__init__.py'
adding 'pulsar/functions/context.py'
adding 'pulsar/functions/function.py'
adding 'pulsar/functions/serde.py'
adding 'pulsar/schema/__init__.py'
adding 'pulsar/schema/definition.py'
adding 'pulsar/schema/schema.py'
adding 'pulsar/schema/schema_avro.py'
adding 'pulsar_client-2.9.1.dist-info/METADATA'
adding 'pulsar_client-2.9.1.dist-info/WHEEL'
adding 'pulsar_client-2.9.1.dist-info/top_level.txt'
adding 'pulsar_client-2.9.1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ auditwheel repair dist/pulsar_client-2.8.2-cp38-cp38-linux_x86_64.whl dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: unrecognized arguments: dist/pulsar_client-2.9.1-cp38-cp38-linux_x86_64.whl
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Pulsar 2.9.1:
[INFO]
[INFO] Pulsar ............................................. SUCCESS [ 24.345 s]
[INFO] Apache Pulsar :: Docker Images ..................... SUCCESS [  0.630 s]
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version FAILURE [27:17 min]
[INFO] Apache Pulsar :: Docker Images :: Pulsar Latest Version (Include All Components) SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  27:43 min
[INFO] Finished at: 2021-12-22T09:50:54-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:exec (build-pulsar-clients-python-35) on project pulsar-docker-image: Command execution failed.: Process exited with an error: 2 (Exit value: 2) -> [Help 1]

Desktop:

• OS: MacOS 12.0 Monterey Darwin <hostname removed>.local 21.0.1 Darwin Kernel Version 21.0.1: Tue Sep 14 20:56:24 PDT 2021; root:xnu-8019.30.61~4/RELEASE_ARM64_T6000 arm64
  However, the command is failing inside a container it seems.

Additional context
This error occurs on branch 2.9. I've also tried to build master and branch-2.8 with no success. I've also tried building following the steps to reproduce on a clean install of Linux Mint with openjdk-11 and maven 3.8.4, but that gives a different error about Text file busy: 'build/bdist.linux-x86_64/wheel/pulsar_client-2.9.1-py3.8.egg-info' - so should be a separate issue.

Maven:

➜ mvn --version
Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537)
Maven home: /opt/homebrew/Cellar/maven/3.8.4/libexec
Java version: 11, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Java:

➜  java --version
openjdk 11 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

Python (although it seems to be using a bundled version so this probably doesn't matter):
Python 3.9.7

Side note: Is there any timeline for when the Log4J patched (2.17) images will be released to the Docker Registry? I feel like there are a lot of people struggling with issues like this and I'm surprised the patch has been completed in here but hasn't been released. There's still a lot of vulnerable Pulsar instances because of this. It's blocking development for us as we also can't seem to build pulsar images across different machines.

[eolivelli]

Can you try to 'git clean -xdf' your repo?
IIRC that errors occurs if you have multiple builds locally

I have built the official images for 2.9.1 and 2.9.0 and it worked well

[samzph]

Can you try to 'git clean -xdf' your repo? IIRC that errors occurs if you have multiple builds locally

I have built the official images for 2.9.1 and 2.9.0 and it worked well

That seemed to get rid of the auditwheel error, but now it's failing on the java.security file. This appears to be an issue in the docker container as it's using the amd64 version and I'm compiling on arm.

[ERROR] Failed to execute goal com.spotify:dockerfile-maven-plugin:1.4.13:build (default) on project pulsar-docker-image: Could not build image: The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-11-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2 -> [Help 1]

More trace:

[INFO] dockerfile: null
[INFO] contextDirectory: /Users/samwork/Code/pulsar/docker/pulsar
[INFO] Building Docker context /Users/samwork/Code/pulsar/docker/pulsar
[INFO] Path(dockerfile): null
[INFO] Path(contextDirectory): /Users/samwork/Code/pulsar/docker/pulsar
[INFO]
[INFO] Image will be built as apachepulsar/pulsar:2.9.2-SNAPSHOT
[INFO]
[INFO] Step 1/26 : FROM busybox as pulsar

[...]

[INFO]  ---> Using cache
[INFO]  ---> 19d52019bcaa
[INFO] Step 17/26 : RUN python3 get-pip.py
[INFO]
[INFO]  ---> Using cache
[INFO]  ---> 19e943035cac
[INFO] Step 18/26 : RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 10
[INFO]
[INFO]  ---> Using cache
[INFO]  ---> 0ac369505cc6
[INFO] Step 19/26 : ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64
[INFO]
[INFO]  ---> Using cache
[INFO]  ---> 7d06df455d73
[INFO] Step 20/26 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-11-openjdk-amd64/conf/security/java.security
[INFO]
[INFO]  ---> Running in 977017eabc3e
[INFO] /bin/sh: 1: cannot create /usr/lib/jvm/java-11-openjdk-amd64/conf/security/java.security: Directory nonexistent

[sarimak]

I encountered the same issue when self-building manylinux2014 wheel for Pulsar Client 2.8.1 and Python 3.9 using backported Docker-based build procedure from https://github.com/apache/pulsar/pull/10954/files to v2.8.1 tag. The wheel gets built and works fine, just the auditwheel step failed.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

Closed as stale. The development of the Python client has been permanently moved to http://github.com/apache/pulsar-client-python. Please open an issue there if it's still relevant.