PIP-155: Removed Python 2 support #15376

merlimat · 2022-04-28T23:31:40Z

Motivation

Remove Python 2 from build scripts

Modifications

Use Python3 in build image
Update build image to ubuntu:20.04 (There is no Python 3.7 support in the old Ubuntu)
Use python3 instead of python in the executable scripts

BewareMyPower · 2022-04-29T05:54:35Z

CMake Error at CMakeLists.txt:272 (string):
  string sub-command REPLACE requires at least four arguments.


-- PYTHON:

It looks like find_package(PythonLibs REQUIRED) didn't find the PYTHONLIBS_VERSION_STRING correctly.

BewareMyPower

It looks like now we use higher clang-format to format C++ source code. Could you update it in pulsar-client-cpp/README.md? See https://github.com/apache/pulsar/tree/master/pulsar-client-cpp#requirements-for-contributors.

Because the rule of latest clang-format is a little different with clang-format-5.0.

### Motivation For branch-2.11 and master branches, the rpm and deb packaging is broken. > /usr/bin/env: 'python3': No such file or directory It's caused by [PIP-155](apache#15376), which removes the Python2 support. In addition, the Dockerfiles of `rpm` and `deb` subdirectories of `pulsar-client-cpp/pkg` don't install Boost for Python3 well. But the rpm build disables the `BUILD_PYTHON_WRAPPER` option while the deb build doesn't. ### Modifications - Fix Dockerfiles under `pkg` and `rpm` subdirectories and speed up the `docker build` process: - Install python3 to fix the bug - Download the CMake 3.24.0 binaries directly instead of building from source - Upgrade Boost to 1.79 so that only headers are required (the boost::regex module is required for CentOS 7 because the default GCC is 4.8 that there is something wrong with std::regex) - Add `-j8` option when compiling some large dependencies that uses 8 threads to speed the compilation - Remove the `-v` option of `tar` to avoid showing too many info - Check the `BUILD_IMAGE` environment variable in `docker-build-xxx.sh` under `rpm` and `deb` subdirectories. If it's defined, build the image rather than pull the image from DockerHub before executing the build script. Because currently only a few committers have the permission to push the Docker images. - Add two workflows to build RPM and DEB packages if the files under `pulsar-client-cpp` changed. They build the docker images and run the build script for RPM and DEB packaging.

### Motivation For branch-2.11 and master branches, the rpm and deb packaging is broken. > /usr/bin/env: 'python3': No such file or directory It's caused by [PIP-155](apache#15376), which removes the Python2 support. In addition, the Dockerfiles of `rpm` and `deb` subdirectories of `pulsar-client-cpp/pkg` don't install Boost for Python3 well. But the rpm build disables the `BUILD_PYTHON_WRAPPER` option while the deb build doesn't. ### Modifications - Fix Dockerfiles under `pkg` and `rpm` subdirectories and speed up the `docker build` process: - Install python3 to fix the bug - Download the CMake 3.24.0 binaries directly instead of building from source - Upgrade Boost to 1.79 so that only headers are required (the boost::regex module is required for CentOS 7 because the default GCC is 4.8 that there is something wrong with std::regex) - Add `-j8` option when compiling some large dependencies that uses 8 threads to speed the compilation - Remove the `-v` option of `tar` to avoid showing too many info - Check the `BUILD_IMAGE` environment variable in `docker-build-xxx.sh` under `rpm` and `deb` subdirectories. If it's defined, build the image rather than pull the image from DockerHub before executing the build script. Because currently only a few committers have the permission to push the Docker images. - Add two workflows to build RPM and DEB packages if the files under `pulsar-client-cpp` changed. They build the docker images and run the build script for RPM and DEB packaging. - Specify the platform to `linux/amd64` so that the scripts can be used in ARM64 machines like Mac M1.

* Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <mmarshall@apache.org> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <nodeces@gmail.com>

* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3) * PIP-155: Removed Python 2 support (apache#15376) * Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <mmarshall@apache.org> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][docker] Switch to Temurin JDK (apache#17129) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 4378856) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Support ARM64-based docker images (apache#17733) (cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * PIP-209: Removed C++/Python clients from main repo (apache#17881) * PIP-209: Removed C++/Python clients from main repo * Removed python directory from Docekrfile * Fixed python client version argument scoping * Fixed handling of pulsar.functions.serde (cherry picked from commit f3c547b) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][build] Avoid building image multiple times (apache#17208) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 79a97a9) * [improve] Allow to build and push multi-arch Docker images (apache#19432) Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zixuan Liu <nodeces@gmail.com> Co-authored-by: tison <wander4096@gmail.com> (cherry picked from commit 4190e40) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix publish image script (apache#20305) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 94c7bf3) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435) Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit d7f3558) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Adapt to Python client to be compatible with ARM arch Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550) (cherry picked from commit 05f7e62) * [improve][misc] Include native epoll library for Netty for arm64 From apache#22319 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][misc] Rename all shaded Netty native libraries From apache#22415 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [cleanup][build] Cleanup -Ddocker.nocache=true Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix ubuntu mirror Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix license Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Downgrade docker-maven to 0.43.3 Signed-off-by: Zixuan Liu <nodeces@gmail.com> --------- Signed-off-by: Zixuan Liu <nodeces@gmail.com> Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Matteo Merli <mmerli@apache.org> Co-authored-by: Michael Marshall <mmarshall@apache.org> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org>

michaeljmarshall and others added 5 commits April 28, 2022 14:55

Remove Pulsar Client Build for Python 2.7

a64d484

Remove outdated homebrew files (source of truth is upstream homebrew)

5c86fab

Remove Python 2.7 build references; print error in some cases

0cd4342

Update python client tests to run with python client for python 3.5m

b94cc73

PIP-155: Removed Python 2 support

6742301

merlimat added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. release/note-required labels Apr 28, 2022

merlimat added this to the 2.11.0 milestone Apr 28, 2022

merlimat requested a review from BewareMyPower April 28, 2022 23:31

github-actions bot assigned merlimat Apr 28, 2022

merlimat requested review from codelipenghui and hangc0276 April 28, 2022 23:32

Fixed invocation in pulsar-build image

c425ac2

merlimat added 16 commits April 29, 2022 06:49

Fixed clang-format-10 indent differences

eddf94d

Fixed script invocation with wrong python

d0f3cfd

We don't need to rebuild the manylinux image each time

66ce84e

Fixed image name

11286e8

Reverted back to use newer protobuf

c5792fd

Fixed image name

035fde7

Fixed missing python3 in centos:7 image

b38ea88

Use python3 for gtest-parallel

8f31a2b

Show bash commands in docker-tests.sh

f0817b8

Fixed gh action issue with git directory permissions

947d3c9

Fixed python to 3

535ea84

Fixed custom_logger_test.py

044bd64

Fixed path in run_python_instance_tests.sh

6849e9b

Function runtime should use python3

907468e

Fixed function runtime test python expectation

2bfd0df

Fixed presto worker launcher

b692710

hangc0276 approved these changes May 5, 2022

View reviewed changes

BewareMyPower requested changes May 5, 2022

View reviewed changes

Fixed notes on how to format C++ code

988c03d

BewareMyPower approved these changes May 5, 2022

View reviewed changes

merlimat merged commit 2b2e0c5 into apache:master May 5, 2022

merlimat deleted the remove-python-2.7-build branch May 5, 2022 06:01

lhotari mentioned this pull request May 7, 2022

[improve][build] PIP-156 Build Pulsar Server on Java 17 #15264

Merged

5 tasks

zymap mentioned this pull request May 30, 2022

[fix][python-function] Fix the failed CPP test CI caused by python function #15827

Closed

BewareMyPower mentioned this pull request Jul 4, 2022

Branch issue 8622: Remove python 2.7 #9268

Closed

1 task

yaalsn mentioned this pull request Aug 8, 2022

[fix][cpp-client] Add python3 to rpm build Dockerfile #16992

Closed

5 tasks

BewareMyPower mentioned this pull request Aug 11, 2022

[fix][C++ client] Fix rpm and deb packaging #17064

Merged

4 tasks

tisonkun mentioned this pull request Mar 19, 2023

python functions cannot find python3 interpreter #5518

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PIP-155: Removed Python 2 support #15376

PIP-155: Removed Python 2 support #15376

merlimat commented Apr 28, 2022

BewareMyPower commented Apr 29, 2022

BewareMyPower left a comment

PIP-155: Removed Python 2 support #15376

PIP-155: Removed Python 2 support #15376

Conversation

merlimat commented Apr 28, 2022

Motivation

Modifications

BewareMyPower commented Apr 29, 2022

BewareMyPower left a comment

Choose a reason for hiding this comment