[SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1 by LuciferYang · Pull Request #44174 · apache/spark

LuciferYang · 2023-12-05T03:31:17Z

What changes were proposed in this pull request?

This PR aims to upgrade Derby from 10.14.2.0 to 10.16.1.1：

The version in pom.xml and related comments were updated. After the update, Spark added a dependency on the org.apache.derby:derbyshared module because it is a cascading dependency of derby.
Since org.apache.derby.jdbc.EmbeddedDriver has been moved to the org.apache.derby:derbytools module, the corresponding dependency was added to the project.
Since org.apache.derby.jdbc.AutoloadedDriver has been moved to org.apache.derby.iapi.jdbc.AutoloadedDriver, the corresponding test configuration in o.a.spark.sql.hive.thriftserver.CliSuite was modified.

Why are the changes needed?

Derby 10.16 release family supports Java 17 and higher with JDBC 4.2, and this version no longer supports the Java SecurityManager.

The full release notes as follows:

https://db.apache.org/derby/releases/release-10_16_1_1.cgi

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

LuciferYang · 2023-12-05T03:34:47Z

Test first. We can't upgrade to use 10.17.1.0 because its minimum supported Java 19.

[INFO] Restricted to JDK 17 yet org.apache.derby:derby:jar:10.17.1.0:test contains module-info.class targeted to JDK 19
[INFO] Restricted to JDK 17 yet org.apache.derby:derbyshared:jar:10.17.1.0:test contains module-info.class targeted to JDK 19
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.323 s
[INFO] Finished at: 2023-12-05T11:29:54+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.3.0:enforce (enforce-versions) on project spark-core_2.13: 
[ERROR] Rule 3: org.codehaus.mojo.extraenforcer.dependencies.EnforceBytecodeVersion failed with message:
[ERROR] Found Banned Dependency: org.apache.derby:derby:jar:10.17.1.0
[ERROR] Found Banned Dependency: org.apache.derby:derbyshared:jar:10.17.1.0
[ERROR] Use 'mvn dependency:tree' to locate the source of the banned dependencies.

dongjoon-hyun

Thank you for making an effort for this, @LuciferYang .

Although I filed SPARK-44977 Upgrade Derby to 10.15.1.3+, I didn't start.

LuciferYang · 2023-12-05T06:36:33Z

Many test cases have failed. Let me investigate

LuciferYang · 2023-12-05T07:23:48Z

core/pom.xml

    </dependency>
+    <dependency>
+      <groupId>org.apache.derby</groupId>
+      <artifactId>derbytools</artifactId>


org.apache.derby.jdbc.EmbeddedDriver has been moved to derbytools in derby 10.16.11

LuciferYang · 2023-12-05T13:02:35Z

sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala

      s"spark.sql.catalog.$catalogName.url=jdbc:derby:memory:$catalogName;create=true"
    val catalogDriver =
-      s"spark.sql.catalog.$catalogName.driver=org.apache.derby.jdbc.AutoloadedDriver"
+      s"spark.sql.catalog.$catalogName.driver=org.apache.derby.iapi.jdbc.AutoloadedDriver"


AutoloadedDriver has been moved to a new package : org.apache.derby.iapi.jdbc.

LuciferYang · 2023-12-05T13:04:45Z

dev/deps/spark-deps-hadoop-3-hive-2.3

 datasketches-memory/2.1.0//datasketches-memory-2.1.0.jar
-derby/10.14.2.0//derby-10.14.2.0.jar
+derby/10.16.1.1//derby-10.16.1.1.jar
+derbyshared/10.16.1.1//derbyshared-10.16.1.1.jar


derbyshared is a transitive dependency of derby.

LuciferYang · 2023-12-05T16:23:01Z

All test passed, I will update the PR description later.

dongjoon-hyun

It's really nice to see the success. Thank you so much, @LuciferYang !

LuciferYang · 2023-12-06T03:15:24Z

Merged into master for Spark 4.0. Thanks @dongjoon-hyun ~

dongjoon-hyun · 2023-12-06T19:19:40Z

Thank you so much, @LuciferYang . This was a headache for me really.

bjornjorgensen · 2023-12-09T18:27:14Z

Great work.. I hope that there soon will be a 10.16.2 that have a backport of https://issues.apache.org/jira/browse/DERBY-7147 This is a fix for CVE-2022-46337 witch have a score of 9.8

dongjoon-hyun · 2023-12-09T22:00:36Z

Go ahead, @bjornjorgensen . :)

### What changes were proposed in this pull request? This PR aims to upgrade Derby from 10.14.2.0 to 10.16.1.1： 1. The version in `pom.xml` and related comments were updated. After the update, Spark added a dependency on the `org.apache.derby:derbyshared` module because it is a cascading dependency of `derby`. 2. Since `org.apache.derby.jdbc.EmbeddedDriver` has been moved to the `org.apache.derby:derbytools` module, the corresponding dependency was added to the project. 3. Since `org.apache.derby.jdbc.AutoloadedDriver` has been moved to `org.apache.derby.iapi.jdbc.AutoloadedDriver`, the corresponding test configuration in `o.a.spark.sql.hive.thriftserver.CliSuite` was modified. ### Why are the changes needed? Derby 10.16 release family [supports Java 17](https://issues.apache.org/jira/browse/DERBY-7137) and higher with JDBC 4.2, and this version [no longer supports the Java SecurityManager](https://issues.apache.org/jira/browse/DERBY-7138). The full release notes as follows: - https://db.apache.org/derby/releases/release-10_16_1_1.cgi ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#44174 from LuciferYang/SPARK-46257. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

…park-3.5 Updated the advisory note to explain why Spark-3.5 cannot upgrade Derby from version 10.14.2.0 to 10.17.1.0 due to Java bytecode version incompatibility. The upstream project has updated to version 10.16.1.1, which does not resolve the vulnerability. The fix is planned with Derby version 10.16.2.x if it gets backported in Spark-4 or later. For more details, see: apache/spark#44174 Signed-off-by: debasishbsws <debasishbsws.dev@gmail.com>

…48hv (#8202) * Feat(adv): Fix not planed for GHSA-rcjc-c4pj-xxrp Derby component inspark-3.5 Updated the advisory note to explain why Spark-3.5 cannot upgrade Derby from version 10.14.2.0 to 10.17.1.0 due to Java bytecode version incompatibility. The upstream project has updated to version 10.16.1.1, which does not resolve the vulnerability. The fix is planned with Derby version 10.16.2.x if it gets backported in Spark-4 or later. For more details, see: apache/spark#44174 Signed-off-by: debasishbsws <debasishbsws.dev@gmail.com> * Feat(adv): Fix not planed for GHSA-c27h-mcmw-48hv because of dependency on hive 2.3 Spark has a transitive dependency on the unmaintained codehaus jackson-mapper-asl library due to Hive 2.3, which is required for initializing the FunctionRegistry. Hive 3.x, planned for Spark 4.x, should remove this dependency. However, any fix in Spark 4.x cannot be backported to Spark 3.5.x due to its reliance on Hive 2.3. Signed-off-by: debasishbsws <debasishbsws.dev@gmail.com> --------- Signed-off-by: debasishbsws <debasishbsws.dev@gmail.com>

…o 10.16.1.1 ### What changes were proposed in this pull request? This PR aims to upgrade Derby from 10.14.2.0 to 10.16.1.1： 1. The version in `pom.xml` and related comments were updated. After the update, Spark added a dependency on the `org.apache.derby:derbyshared` module because it is a cascading dependency of `derby`. 2. Since `org.apache.derby.jdbc.EmbeddedDriver` has been moved to the `org.apache.derby:derbytools` module, the corresponding dependency was added to the project. 3. Since `org.apache.derby.jdbc.AutoloadedDriver` has been moved to `org.apache.derby.iapi.jdbc.AutoloadedDriver`, the corresponding test configuration in `o.a.spark.sql.hive.thriftserver.CliSuite` was modified. ### Why are the changes needed? Derby 10.16 release family [supports Java 17](https://issues.apache.org/jira/browse/DERBY-7137) and higher with JDBC 4.2, and this version [no longer supports the Java SecurityManager](https://issues.apache.org/jira/browse/DERBY-7138). The full release notes as follows: - https://db.apache.org/derby/releases/release-10_16_1_1.cgi ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GitHub Actions ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#44174 from LuciferYang/SPARK-46257. Lead-authored-by: yangjie01 <yangjie01@baidu.com> Co-authored-by: YangJie <yangjie01@baidu.com> Signed-off-by: yangjie01 <yangjie01@baidu.com>

init

4663343

LuciferYang marked this pull request as draft December 5, 2023 03:31

github-actions bot added the BUILD label Dec 5, 2023

Merge branch 'apache:master' into SPARK-46257

c33480c

dongjoon-hyun reviewed Dec 5, 2023

View reviewed changes

add derbytools

9d66027

github-actions bot added SQL CORE labels Dec 5, 2023

LuciferYang commented Dec 5, 2023

View reviewed changes

LuciferYang changed the title ~~[SPARK-46257][BUILD] Upgrade Derby to 10.16.1.1~~ [SPARK-44977][SPARK-46257][BUILD] Upgrade Derby to 10.16.1.1 Dec 5, 2023

rename driver name

5541636

LuciferYang commented Dec 5, 2023

View reviewed changes

LuciferYang changed the title ~~[SPARK-44977][SPARK-46257][BUILD] Upgrade Derby to 10.16.1.1~~ [SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1 Dec 5, 2023

LuciferYang commented Dec 5, 2023

View reviewed changes

dongjoon-hyun reviewed Dec 5, 2023

View reviewed changes

dongjoon-hyun approved these changes Dec 5, 2023

View reviewed changes

LuciferYang marked this pull request as ready for review December 6, 2023 03:10

LuciferYang closed this in 6f79faa Dec 6, 2023

debasishbsws mentioned this pull request Sep 13, 2024

Feat(adv): Fix not planed for GHSA-rcjc-c4pj-xxrp and GHSA-c27h-mcmw-48hv wolfi-dev/advisories#8202

Merged

jamie-albert mentioned this pull request Jan 17, 2025

spark-3.5-scala-2.13: derby and jackson-mapper-asl advisory updates wolfi-dev/advisories#11298

Merged

jamie-albert mentioned this pull request Jan 24, 2025

spark-3.5-scala-2.12/GHSA-rcjc-c4pj-xxrp advisory update wolfi-dev/advisories#11367

Merged

senthh mentioned this pull request Mar 28, 2025

ODP-3835|[SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1 acceldata-io/spark3#77

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1#44174

[SPARK-44977][SPARK-46257][BUILD][CORE][SQL] Upgrade Derby to 10.16.1.1#44174
LuciferYang wants to merge 4 commits intoapache:masterfrom
LuciferYang:SPARK-46257

LuciferYang commented Dec 5, 2023 •

edited

Loading

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

dongjoon-hyun left a comment •

edited

Loading

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

LuciferYang Dec 5, 2023 •

edited

Loading

Uh oh!

LuciferYang Dec 5, 2023

Uh oh!

LuciferYang Dec 5, 2023 •

edited

Loading

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

dongjoon-hyun left a comment

Uh oh!

LuciferYang commented Dec 6, 2023

Uh oh!

dongjoon-hyun commented Dec 6, 2023

Uh oh!

bjornjorgensen commented Dec 9, 2023

Uh oh!

dongjoon-hyun commented Dec 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

LuciferYang commented Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

LuciferYang Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 5, 2023

Choose a reason for hiding this comment

Uh oh!

LuciferYang Dec 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 5, 2023

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Dec 6, 2023

Uh oh!

dongjoon-hyun commented Dec 6, 2023

Uh oh!

bjornjorgensen commented Dec 9, 2023

Uh oh!

dongjoon-hyun commented Dec 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LuciferYang commented Dec 5, 2023 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading

LuciferYang Dec 5, 2023 •

edited

Loading

LuciferYang Dec 5, 2023 •

edited

Loading