[SPARK-41450][BUILD] Fix shading in `core` module #38999

pan3793 · 2022-12-09T08:36:43Z

What changes were proposed in this pull request?

Fix shading in core module.

Why are the changes needed?

Fixed issue mentioned in #38779 (comment)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

➜  dist git:(SPARK-41244-followup) bin/spark-shell
22/12/09 16:35:43 WARN Utils: Your hostname, Chengs-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 10.221.96.92 instead (on interface en0)
22/12/09 16:35:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/12/09 16:35:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context Web UI available at http://10.221.96.92:4040
Spark context available as 'sc' (master = local[*], app id = local-1670574946548).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.4.0-SNAPSHOT
      /_/

Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 1.8.0_332)
Type in expressions to have them evaluated.
Type :help for more information.

scala> spark.sql("select 1").show
+---+
|  1|
+---+
|  1|
+---+

dongjoon-hyun · 2022-12-09T08:40:29Z

cc @gengliangwang , @LuciferYang @cloud-fan @mridulm @amaliujia @tgravescs , @ulysses-you

dongjoon-hyun · 2022-12-09T08:41:29Z

cc @parthchandra , too.

LuciferYang · 2022-12-09T08:43:58Z

pom.xml

          <artifactSet>
            <includes>
              <include>org.spark-project.spark:unused</include>
-              <include>org.eclipse.jetty:jetty-io</include>


if we move this part to core/pom.xml, also need remove the relocation of jetty from parent pom.xml?

Not exactly, jetty classes should be included into core only, but relocation is required for all modules, otherwise, other modules like sql reference the vanilla jetty classes but core ships relocated jetty classes.

LuciferYang · 2022-12-09T08:45:50Z

core/pom.xml

          <shadeTestJar>true</shadeTestJar>
          <artifactSet>
            <includes>
+              <include>org.spark-project.spark:unused</include>


Have you compared the compiled logs of Maven shade? If it is not use append mode, we need make sure there is no omission.

Yes, the result is expected.

This PR

[INFO] --- maven-shade-plugin:3.2.4:shade (default) @ spark-core_2.12 --- [INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including com.google.protobuf:protobuf-java:jar:3.21.9 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.

Before #38779

[INFO] --- maven-shade-plugin:3.2.4:shade (default) @ spark-core_2.12 --- [INFO] Including org.eclipse.jetty:jetty-plus:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-security:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-util:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-server:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-io:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-http:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-continuation:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlet:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-proxy:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-client:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.eclipse.jetty:jetty-servlets:jar:9.4.49.v20220914 in the shaded jar. [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.

LuciferYang

+1, LGTM

dongjoon-hyun

+1, LGTM.

HyukjinKwon · 2022-12-09T11:46:05Z

let me change the JIRA in the title since there's already a JIRA open.

HyukjinKwon

LGTM, thanks for fixing this.

dongjoon-hyun · 2022-12-09T15:42:10Z

Merged to master. Thank you, @pan3793 and all!

gengliangwang · 2022-12-09T17:50:42Z

@pan3793 Thanks for fixing it!

### What changes were proposed in this pull request? Fix shading in core module. ### Why are the changes needed? Fixed issue mentioned in apache#38779 (comment) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` ➜ dist git:(SPARK-41244-followup) bin/spark-shell 22/12/09 16:35:43 WARN Utils: Your hostname, Chengs-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 10.221.96.92 instead (on interface en0) 22/12/09 16:35:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/12/09 16:35:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Spark context Web UI available at http://10.221.96.92:4040 Spark context available as 'sc' (master = local[*], app id = local-1670574946548). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.4.0-SNAPSHOT /_/ Using Scala version 2.12.17 (OpenJDK 64-Bit Server VM, Java 1.8.0_332) Type in expressions to have them evaluated. Type :help for more information. scala> spark.sql("select 1").show +---+ | 1| +---+ | 1| +---+ ``` <img width="1604" alt="image" src="https://user-images.githubusercontent.com/26535726/206659907-95d95d7f-dce2-4a4f-9aef-e54fb4612aba.png"> Closes apache#38999 from pan3793/SPARK-41244-followup. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-41244] Fix shading in core module

32366e8

github-actions bot added BUILD CORE labels Dec 9, 2022

pan3793 mentioned this pull request Dec 9, 2022

[SPARK-41244][UI] Introducing a Protobuf serializer for UI data on KV store #38779

Closed

nit

661d269

dongjoon-hyun changed the title ~~[SPARK-41244] Fix shading in core module~~ [SPARK-41244][BUILD] Fix shading in core module Dec 9, 2022

nit

07bee24

dongjoon-hyun changed the title ~~[SPARK-41244][BUILD] Fix shading in core module~~ [SPARK-41244][BUILD][FOLLOWUP] Fix shading in core module Dec 9, 2022

LuciferYang reviewed Dec 9, 2022

View reviewed changes

LuciferYang approved these changes Dec 9, 2022

View reviewed changes

dongjoon-hyun approved these changes Dec 9, 2022

View reviewed changes

HyukjinKwon changed the title ~~[SPARK-41244][BUILD][FOLLOWUP] Fix shading in core module~~ [SPARK-41450][BUILD] Fix shading in core module Dec 9, 2022

HyukjinKwon approved these changes Dec 9, 2022

View reviewed changes

dongjoon-hyun closed this in fc3c0f1 Dec 9, 2022

WeichenXu123 mentioned this pull request Dec 10, 2022

Fix spark dev build in cross test mlflow/mlflow#7471

Closed

33 tasks

[SPARK-41450][BUILD] Fix shading in core module #38999

[SPARK-41450][BUILD] Fix shading in core module #38999

Uh oh!

Conversation

pan3793 commented Dec 9, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Dec 9, 2022

Uh oh!

dongjoon-hyun commented Dec 9, 2022

Uh oh!

LuciferYang Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

pan3793 Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

pan3793 Dec 9, 2022

Choose a reason for hiding this comment

Uh oh!

LuciferYang left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Dec 9, 2022

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Dec 9, 2022

Uh oh!

gengliangwang commented Dec 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[SPARK-41450][BUILD] Fix shading in `core` module #38999

[SPARK-41450][BUILD] Fix shading in `core` module #38999