[SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt` #49550

LuciferYang · 2025-01-17T09:02:43Z

What changes were proposed in this pull request?

This pr aims to remove the explicit dependency on Guava from plugins.sbt to minimize the number of modifications needed when upgrading the Guava version.

Why are the changes needed?

Reduce the configuration items for defining the Guava version.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions
Manual check dev/sbt-checkstyle, It can work properly.

For example:

spark/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java

Lines 217 to 220 in 1c4cfcb

    
           if (newArray.size() < array.size()) { 
        
             // checkstyle.off: RegexpSinglelineJava 
        
             throw new SparkOutOfMemoryError("_LEGACY_ERROR_TEMP_3301", new HashMap<>()); 
        
             // checkstyle.on: RegexpSinglelineJava

Remove the line 218 from UnsafeInMemorySorter.java( // checkstyle.off: RegexpSinglelineJava )and then execute dev/sbt-checkstyle.

Before

Checkstyle failed at following occurrences:
[error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86)
[error] 	at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57)
[error] 	at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] (core / checkstyle) Severity of checkstyle errors exceeds project limit

After

Checkstyle failed at following occurrences:
[error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit
[error] 	at scala.sys.package$.error(package.scala:30)
[error] 	at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86)
[error] 	at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57)
[error] 	at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49)
[error] 	at scala.Function1.$anonfun$compose$1(Function1.scala:49)
[error] 	at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63)
[error] 	at sbt.std.Transform$$anon$4.work(Transform.scala:69)
[error] 	at sbt.Execute.$anonfun$submit$2(Execute.scala:283)
[error] 	at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24)
[error] 	at sbt.Execute.work(Execute.scala:292)
[error] 	at sbt.Execute.$anonfun$submit$1(Execute.scala:283)
[error] 	at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265)
[error] 	at sbt.CompletionService$$anon$2.call(CompletionService.scala:65)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[error] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error] 	at java.base/java.lang.Thread.run(Thread.java:840)
[error] (core / checkstyle) Severity of checkstyle errors exceeds project limit

The displayed check results show no difference.

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2025-01-17T09:06:43Z

project/plugins.sbt

 libraryDependencies += "com.puppycrawl.tools" % "checkstyle" % "10.20.2"

-// checkstyle uses guava 33.3.1-jre.
-libraryDependencies += "com.google.guava" % "guava" % "33.3.1-jre"


@HyukjinKwon , do you still remember the reason for adding an explicit dependency on Guava in #21399? I found that even without configuring it, the execution of dev/sbt-checkstyle meets the expectations.

However, I found that before this pr, the format of the check results had already changed to something similar to

Checkstyle failed at following occurrences: [error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit [error] at scala.sys.package$.error(package.scala:30) [error] at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [error] at java.base/java.lang.Thread.run(Thread.java:840) [error] (core / checkstyle) Severity of checkstyle errors exceeds project limit

, rather than

I do remember. It was done at 4a14dc0, and scalas scheck style failed because of Guava version mismatch (SBT specifically). Since the Guava version in plugins.sbt here doesn't affect the main code, we can just set whatever version we want.

It doesn't affect anything in the main code, so if it works better, we can remove, or upgrade, etc. It's like if it works, then LGTM

Okay, let me confirm again. If we can remove it, we'll have one less place to modify when upgrading Guava.

HyukjinKwon · 2025-01-20T07:39:57Z

Merged to master and branch-4.0.

…plugins.sbt` This pr aims to remove the explicit dependency on `Guava` from `plugins.sbt` to minimize the number of modifications needed when upgrading the Guava version. Reduce the configuration items for defining the Guava version. No - Pass GitHub Actions - Manual check `dev/sbt-checkstyle`, It can work properly. For example: https://github.com/apache/spark/blob/1c4cfcb0277d673a3f53b49e74ab74452118da91/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L217-L220 Remove the line 218 from `UnsafeInMemorySorter.java`( `// checkstyle.off: RegexpSinglelineJava` )and then execute `dev/sbt-checkstyle`. **Before** ``` Checkstyle failed at following occurrences: [error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit [error] at scala.sys.package$.error(package.scala:30) [error] at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [error] at java.base/java.lang.Thread.run(Thread.java:840) [error] (core / checkstyle) Severity of checkstyle errors exceeds project limit ``` **After** ``` Checkstyle failed at following occurrences: [error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit [error] at scala.sys.package$.error(package.scala:30) [error] at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [error] at java.base/java.lang.Thread.run(Thread.java:840) [error] (core / checkstyle) Severity of checkstyle errors exceeds project limit ``` The displayed check results show no difference. No Closes #49550 from LuciferYang/remove-guava-from-plugins. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit 4874b6e) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

LuciferYang · 2025-01-20T07:42:24Z

Thanks @HyukjinKwon

…plugins.sbt` This pr aims to remove the explicit dependency on `Guava` from `plugins.sbt` to minimize the number of modifications needed when upgrading the Guava version. Reduce the configuration items for defining the Guava version. No - Pass GitHub Actions - Manual check `dev/sbt-checkstyle`, It can work properly. For example: https://github.com/apache/spark/blob/de8f581d9a62f12f927ce43c4c498212b7892ad6/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeInMemorySorter.java#L217-L220 Remove the line 218 from `UnsafeInMemorySorter.java`( `// checkstyle.off: RegexpSinglelineJava` )and then execute `dev/sbt-checkstyle`. **Before** ``` Checkstyle failed at following occurrences: [error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit [error] at scala.sys.package$.error(package.scala:30) [error] at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [error] at java.base/java.lang.Thread.run(Thread.java:840) [error] (core / checkstyle) Severity of checkstyle errors exceeds project limit ``` **After** ``` Checkstyle failed at following occurrences: [error] java.lang.RuntimeException: Severity of checkstyle errors exceeds project limit [error] at scala.sys.package$.error(package.scala:30) [error] at com.etsy.sbt.checkstyle.Checkstyle$.checkstyle(Checkstyle.scala:86) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1(CheckstylePlugin.scala:57) [error] at com.etsy.sbt.checkstyle.CheckstylePlugin$autoImport$.$anonfun$checkstyleTask$1$adapted(CheckstylePlugin.scala:49) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:63) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:69) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:283) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:24) [error] at sbt.Execute.work(Execute.scala:292) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:283) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:65) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [error] at java.base/java.lang.Thread.run(Thread.java:840) [error] (core / checkstyle) Severity of checkstyle errors exceeds project limit ``` The displayed check results show no difference. No Closes apache#49550 from LuciferYang/remove-guava-from-plugins. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org> (cherry picked from commit b981aac) Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>

init

81be81a

github-actions bot added the BUILD label Jan 17, 2025

LuciferYang marked this pull request as draft January 17, 2025 09:02

LuciferYang commented Jan 17, 2025

View reviewed changes

HyukjinKwon approved these changes Jan 20, 2025

View reviewed changes

Merge branch 'upmaster' into remove-guava-from-plugins

d374288

LuciferYang changed the title ~~Remove the explicit dependency on Guava from plugins.sbt~~ [SPARK-50891][BUILD] Remove the explicit dependency on Guava from plugins.sbt Jan 20, 2025

LuciferYang marked this pull request as ready for review January 20, 2025 05:33

HyukjinKwon closed this in 4874b6e Jan 20, 2025

LuciferYang deleted the remove-guava-from-plugins branch May 2, 2025 05:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt` #49550

[SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt` #49550

Uh oh!

LuciferYang commented Jan 17, 2025 •

edited

Loading

Uh oh!

LuciferYang Jan 17, 2025

Uh oh!

LuciferYang Jan 17, 2025

Uh oh!

HyukjinKwon Jan 20, 2025 •

edited

Loading

Uh oh!

HyukjinKwon Jan 20, 2025

Uh oh!

LuciferYang Jan 20, 2025

Uh oh!

HyukjinKwon commented Jan 20, 2025

Uh oh!

LuciferYang commented Jan 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	if (newArray.size() < array.size()) {
	// checkstyle.off: RegexpSinglelineJava
	throw new SparkOutOfMemoryError("_LEGACY_ERROR_TEMP_3301", new HashMap<>());
	// checkstyle.on: RegexpSinglelineJava

[SPARK-50891][BUILD] Remove the explicit dependency on Guava from plugins.sbt #49550

[SPARK-50891][BUILD] Remove the explicit dependency on Guava from plugins.sbt #49550

Uh oh!

Conversation

LuciferYang commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

LuciferYang Jan 17, 2025

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

LuciferYang Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jan 20, 2025

Uh oh!

LuciferYang commented Jan 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt` #49550

[SPARK-50891][BUILD] Remove the explicit dependency on `Guava` from `plugins.sbt` #49550

LuciferYang commented Jan 17, 2025 •

edited

Loading

HyukjinKwon Jan 20, 2025 •

edited

Loading