[ZEPPELIN-5946][FOLLOWUP] Use Spark 3.4.1 in default

[pan3793]

What is this PR for?

Followup ZEPPELIN-5946, Use Spark 3.4.1 in default

What type of PR is it?

Improvement
Documentation

Todos

• - Task

What is the Jira issue?

ZEPPELIN-5946

How should this be tested?

Pass GA

Screenshots (if appropriate)

Questions:

• Does the license files need to update?
• Is there breaking changes for older versions?
• Does this needs documentation?

[Reamer]

Thx for the follow-up.

[pan3793]

The failed Kotlin test seems to be https://youtrack.jetbrains.com/issue/KT-38325

I suppose we must upgrade Kotlin first to address it. Unfortunately, the Kotlin interpreter uses a very old version (1.3.x, the latest is 1.9.x), and couple with lots of unstable APIs.

https://endoflife.date/kotlin

[Reamer]

@pan3793
Is this PR ready for review?

[pan3793]

@Reamer not yet, I need to revert some changes.

Sorry, I'm quite busy these days, will find the time to finish it ASAP.

[pan3793]

The profiles scala-2.11 and scala-2.12 defined at root pom.xml seems only affects the livy module's test, if so, do we really need to maintain such profiles? Why not hardcode in livy/pom.xml(seems livy is not much active, the latest version only support Scala 2.11)

[pan3793]

cc @Reamer

[Reamer]

The profiles scala-2.11 and scala-2.12 defined at root pom.xml seems only affects the livy module's test, if so, do we really need to maintain such profiles? Why not hardcode in livy/pom.xml(seems livy is not much active, the latest version only support Scala 2.11)

The profiles scala-2.11 and scala-2.12 set the properties scala.version and scala.binary.version.

I see accesses to these properties in the Zeppelin Display Module and Zeppelin Distribution Module. Unfortunately, I don't currently know why Scala dependencies are needed in the Zeppelin Distribution Module.

zeppelin/zeppelin-display/pom.xml

Lines 34 to 55 in 85bb3b4

	<dependencies>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-library</artifactId>
	<version>${scala.version}</version>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-compiler</artifactId>
	<version>${scala.version}</version>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scalap</artifactId>
	<version>${scala.version}</version>
	<scope>provided</scope>
	</dependency>
	</dependencies>

zeppelin/zeppelin-distribution/pom.xml

Lines 49 to 73 in 85bb3b4

	<dependencies>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-library</artifactId>
	<version>${scala.version}</version>
	</dependency>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-compiler</artifactId>
	<version>${scala.version}</version>
	</dependency>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-reflect</artifactId>
	<version>${scala.version}</version>
	</dependency>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scalap</artifactId>
	<version>${scala.version}</version>
	</dependency>
	</dependencies>

[pan3793]

Oh, I missed the Zeppelin Display Module(it does not appear when I do global search using IDEA), it leverages the scala-xml to build AngularElem, that's why scala is required

[pan3793]

Emm, is "Zeppelin Display Module" only used by "Zeppelin: Spark Interpreter"? If so, I suppose the Zeppelin core becomes scala free if we can merge this module into "Zeppelin: Spark Interpreter"

[Reamer]

As far as I know, the Zeppelin display module is only used by the Spark interpreter.

If it is possible we can merge the Display Module to the Spark Interpreter Module.

[pan3793]

As far as I know, the Zeppelin display module is only used by the Spark interpreter.

If it is possible we can merge the Display Module to the Spark Interpreter Module.

Thanks for information, would like a try in another PR.

I limit this PR's scope to the spark related stuffs, CI already passed, and the last commit only touch the docs

[Reamer]

LGTM

[Reamer]

I will merge this pull request on Friday as long as no other comment is received.

[jongyoul]

I missed this PR. LGTM