[SPARK-48152][BUILD] Make `spark-profiler` as a part of release and publish to maven central repo #46402

panbingkun · 2024-05-06T14:22:19Z

What changes were proposed in this pull request?

The pr aims to

make the module spark-profiler as a part of Spark release
publish the module spark-profiler to maven central repository
add instructions on how to compile supports spark-profiler in the doc docs/building-spark.md

Why are the changes needed?

1.The modules released in the current daily spark-4.0.0 do not include spark-profiler. I believe that according to the current logic, the spark-profiler will not appear in the future official version of spark.

2.Align the compilation description of other modules in doc docs/building-spark.md, eg:

Does this PR introduce any user-facing change?

Yes, make it easy for users to use spark-profiler in the future version of Spark, instead of manually compiling spark-profiler based on source code.

How was this patch tested?

Pass GA.
It is necessary to observe whether the daily snapshots spark-profilter_2.13 generate
https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-profiler_2.13/4.0.0-SNAPSHOT/

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun

Thank you, @panbingkun .

dongjoon-hyun

Ur, I agree with publishing this module. However, I don't think we need to include this as Apache Spark Binary or dev/deps. This module is designed to be like kafka module, @panbingkun .

[Only test] make spark-profiler publish snapshot [SPARK-48152][BUILD] Make the module spark-profiler as a part of Spark release

dongjoon-hyun · 2024-05-07T01:34:48Z

dev/test-dependencies.sh

 #   dev/create-release/release-build.sh
 HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pkubernetes -Pyarn -Phive \
-    -Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud"
+    -Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud -Pjvm-profiler"


~~Like Kafka module, we should not include this here.~~

Do you know how we skip Kafka module's dependency here?

$ grep kafka dev/deps/spark-deps-hadoop-3-hive-2.3 | wc -l 0

Okay, I will remove it.
But I have a question. If the end user wants to use spark-profiler to analyze the executor, does he download 'me.bechberger:ap-loader-all:xxx' from the maven central repository and use it together with module spark-profiler? If that's the way it used to be, I am fine to it.
Maybe we need to add detailed guide in some document?

~~Like Kafka module, we should not include this here.~~

Do you know how we skip Kafka module's dependency here?

Let me investigate it.

Yes, it's identical with Apache Spark's Kafka module and Apache Spark Hadoop Cloud module

https://mvnrepository.com/artifact/org.apache.spark/spark-sql-kafka-0-10_2.13/3.5.1

https://mvnrepository.com/artifact/org.apache.spark/spark-hadoop-cloud_2.13/3.5.1

Yeah, let me add detailed guide in some document.

~~Like Kafka module, we should not include this here.~~

Do you know how we skip Kafka module's dependency here?

I already know about it. It is implemented through maven scope overloading. Let me try.

dongjoon-hyun · 2024-05-07T02:26:02Z

connector/profiler/pom.xml

      <groupId>me.bechberger</groupId>
      <artifactId>ap-loader-all</artifactId>
-      <version>3.0-8</version>
+      <version>3.0-9</version>


Could you proceed this dependency update separately?

A separate PR:
#46427

panbingkun · 2024-05-07T06:25:22Z

connector/profiler/README.md

 *   Linux (musl, x64)
 *   MacOS

 To get maximum profiling information set the following jvm options for the executor :


I'm not sure if we need to rename the file connector/profiler/README.md to "jvm-profiler-integration.md" and move it to the directory docs/jvm-profiler-integration.md", while linking it in the file docs/building-spark.md? Do we need to do this? @dongjoon-hyun

cc @parthchandra

panbingkun · 2024-05-07T06:26:31Z

docs/building-spark.md

+
+**Note:** The `jvm-profiler` profile builds the assembly without including the dependency `ap-loader`,
+you can download it manually from maven central repo and use it together with `spark-profiler_{{site.SCALA_BINARY_VERSION}}`.
+


The actual effect is like this:

dongjoon-hyun · 2024-05-07T15:18:57Z

I merged the following. Could you rebase this PR?

[SPARK-48165][BUILD] Update ap-loader to 3.0-9 #46427

dongjoon-hyun · 2024-05-07T15:19:32Z

connector/profiler/pom.xml

      <groupId>me.bechberger</groupId>
      <artifactId>ap-loader-all</artifactId>
      <version>3.0-8</version>
+      <scope>provided</scope>


cc @parthchandra , too

It's great to include this feature in the spark release. I feel though, that if we are making it available in the release the ap-loader dependency should be included as well. (Currently, if we build with the jvm-profiler profile, the dependency is included in the build).
Surprisingly, the jar file is not very large (I have version 2.9.7 and it is only 830K).
Either way, including this in the release is already a big step so I'm fine with asking users to download the jar.

dongjoon-hyun · 2024-05-07T22:22:41Z

Could you revise the PR title, @panbingkun ?

panbingkun · 2024-05-07T23:13:10Z

I merged the following. Could you rebase this PR?

[SPARK-48165][BUILD] Update ap-loader to 3.0-9 #46427

Done.

panbingkun · 2024-05-07T23:13:18Z

Could you revise the PR title, @panbingkun ?

Done.

dongjoon-hyun

+1, LGTM for Apache Spark 4.0.0-preview.

Thank you for completing area, @panbingkun .
Thank you for review and understanding the current usage, @parthchandra .

Let's bring this into 4.0.0-preview and get more feedback. We can revise later before 4.0.0.

dongjoon-hyun · 2024-05-08T00:23:58Z

Merged to master.

panbingkun · 2024-05-08T01:09:27Z

Let's bring this into 4.0.0-preview and get more feedback. We can revise later before 4.0.0.

Thanks. ❤️

panbingkun · 2024-05-09T06:10:29Z

It's so happy to observe that maven's snapshot repo has spark-profiler:
https://repository.apache.org/content/repositories/snapshots/org/apache/spark/spark-profiler_2.13/4.0.0-SNAPSHOT/

### What changes were proposed in this pull request? Bump ap-loader version from 3.0-8 to 3.0-9. ### Why are the changes needed? ap-loader has already released v3.0-9, which should bump version from 3.0-8 for `JVMProfiler`. Backport: 1. apache/spark#46402 2. apache/spark#49440 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI. Closes #3072 from SteNicholas/CELEBORN-1842. Authored-by: SteNicholas <programgeek@163.com> Signed-off-by: SteNicholas <programgeek@163.com>

make spark-profiler publish snapshot

042cdd1

github-actions bot added BUILD DOCS INFRA labels May 6, 2024

dongjoon-hyun reviewed May 6, 2024

View reviewed changes

panbingkun changed the title ~~[Only test] make spark-profiler publish snapshot~~ [SPARK-48152][BUILD] Make the module spark-profiler as a part of Spark release May 7, 2024

dongjoon-hyun requested changes May 7, 2024

View reviewed changes

dongjoon-hyun reviewed May 7, 2024

View reviewed changes

panbingkun changed the title ~~[SPARK-48152][BUILD] Make the module spark-profiler as a part of Spark release~~ [SPARK-48152][BUILD] Publish the module spark-profiler to maven central repository May 7, 2024

dongjoon-hyun reviewed May 7, 2024

View reviewed changes

update

2a0589b

panbingkun commented May 7, 2024

View reviewed changes

panbingkun marked this pull request as ready for review May 7, 2024 11:55

dongjoon-hyun reviewed May 7, 2024

View reviewed changes

resolve conflict

b5991bb

panbingkun changed the title ~~[SPARK-48152][BUILD] Publish the module spark-profiler to maven central repository~~ [SPARK-48152][BUILD] Make spark-profiler as a part of release and publish to maven central repo May 7, 2024

dongjoon-hyun approved these changes May 8, 2024

View reviewed changes

dongjoon-hyun closed this in 553e1b8 May 8, 2024

SteNicholas mentioned this pull request Jan 20, 2025

[CELEBORN-1842] Bump ap-loader version from 3.0-8 to 3.0-9 apache/celeborn#3072

Closed


		Note: The `jvm-profiler` profile builds the assembly without including the dependency `ap-loader`,
		you can download it manually from maven central repo and use it together with `spark-profiler_{{site.SCALA_BINARY_VERSION}}`.

[SPARK-48152][BUILD] Make spark-profiler as a part of release and publish to maven central repo #46402

[SPARK-48152][BUILD] Make spark-profiler as a part of release and publish to maven central repo #46402

Uh oh!

Conversation

panbingkun commented May 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun May 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 7, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 7, 2024

Uh oh!

panbingkun commented May 7, 2024

Uh oh!

panbingkun commented May 7, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 8, 2024

Uh oh!

panbingkun commented May 8, 2024

Uh oh!

panbingkun commented May 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-48152][BUILD] Make `spark-profiler` as a part of release and publish to maven central repo #46402

[SPARK-48152][BUILD] Make `spark-profiler` as a part of release and publish to maven central repo #46402

panbingkun commented May 6, 2024 •

edited

Loading

dongjoon-hyun May 7, 2024 •

edited

Loading