[BEAM-12395] Add support to enable google cloud profiler for python S…

[y1chi]

…DK harness

Please add a meaningful description for your change here

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Choose reviewer(s) and mention them in a comment (R: @username).
• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---			---		---
Java
Python	---			---		---
XLang				---		---

Examples testing status on various runners

Lang	ULR	Dataflow	Flink	Samza	Spark	Twister2
Go	---	---	---	---	---	---	---
Java	---		---	---	---	---	---
Python	---	---	---	---	---	---	---
XLang	---	---	---	---	---	---	---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go	Java	Python

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---			---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

[y1chi]

R: @tvalentyn

[tvalentyn]

do you want to install some version in this range in

beam/sdks/python/container/base_image_requirements.txt

Line 43 in e52e9e4

# GCP extra features

?

[y1chi]

good point.

[codecov]

Codecov Report

Merging #14877 (de0815a) into master (8732012) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #14877      +/-   ##
==========================================
- Coverage   83.78%   83.77%   -0.02%     
==========================================
  Files         434      434              
  Lines       58280    58293      +13     
==========================================
+ Hits        48829    48833       +4     
- Misses       9451     9460       +9     

Impacted Files	Coverage Δ
...y38/build/srcs/sdks/python/apache_beam/io/kafka.py
...s/python/apache_beam/runners/worker/log_handler.py
...m/portability/api/beam_interactive_api_pb2_grpc.py
...ython/apache_beam/io/gcp/experimental/spannerio.py
...pache_beam/dataframe/pandas_top_level_functions.py
...examples/snippets/transforms/aggregation/sample.py
...examples/snippets/transforms/elementwise/kvswap.py
...sdks/python/apache_beam/runners/worker/__init__.py
...apache_beam/examples/complete/game/leader_board.py
...38/build/srcs/sdks/python/apache_beam/io/avroio.py
... and 858 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c18a06c...de0815a. Read the comment docs.

[jli]

Could you describe the benefit of adding Cloud Profiler? Does this enable automatic instrumentation of Beam pipelines running on GCP? My team would actually benefit from having that feature :)

FWIW, adding google-cloud-profiler makes using apache-beam within Docker more difficult.

google-cloud-profiler doesn't provide wheels, so upgrading apache-beam from 2.30.0 to 2.31.0 requires adding a compiler toolchain to the Docker build. To avoid bloating the final Docker image, that means setting up multi-stage builds in the Dockerfile. Multi-stage builds to build a Python library is a little obscure. The Cloud Profiler docs does have good documentation for it, but it's a new step all Beam Docker users will need to deal with, and it's a bit annoying to jump through that hoop for a transitive dependency.

It would have been nice to have this change flagged in the release notes (https://github.com/apache/beam/releases/tag/v2.31.0).

I guess the ideal solution here is for google-cloud-profiler to distribute wheels... I filed GoogleCloudPlatform/cloud-profiler-python#98

[y1chi]

Could you describe the benefit of adding Cloud Profiler? Does this enable automatic instrumentation of Beam pipelines running on GCP? My team would actually benefit from having that feature :)

FWIW, adding google-cloud-profiler makes using apache-beam within Docker more difficult.

google-cloud-profiler doesn't provide wheels, so upgrading apache-beam from 2.30.0 to 2.31.0 requires adding a compiler toolchain to the Docker build. To avoid bloating the final Docker image, that means setting up multi-stage builds in the Dockerfile. Multi-stage builds to build a Python library is a little obscure. The Cloud Profiler docs does have good documentation for it, but it's a new step all Beam Docker users will need to deal with, and it's a bit annoying to jump through that hoop for a transitive dependency.

It would have been nice to have this change flagged in the release notes (https://github.com/apache/beam/releases/tag/v2.31.0).

I guess the ideal solution here is for google-cloud-profiler to distribute wheels... I filed GoogleCloudPlatform/cloud-profiler-python#98

Yes, this will enable automatic instrumentation of beam pipeline on GCP when the pipeline options has '--experiments=enable_google_cloud_profiler', I'll add that to release note. The package should have been pre-installed if running GCP. Didn't notice that this will introduce complexity to docker user, thanks for creating the issue for google-cloud-profiler to distribute the wheel. We can remove the dependency from base requirement and maybe have user to provide that dependency through requirements if they indeed need this functionality.