KAFKA-12728: version upgrades: gradle (6.8.3 -->> 7.0.2) and gradle shadow plugin (6.1.0 -->> 7.0.0) by dejan2609 · Pull Request #10606 · apache/kafka

dejan2609 · 2021-04-28T19:16:14Z

JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-12728

Related pull requests:

KAFKA-12415: Prepare for Gradle 7.0 and restrict transitive scope for non api dependencies #10203
KAFKA-12417: streams copyDependentLibs should not copy testRuntime configuration jars #10466

Note: we will wait for a Gradle 7.0.1 patch ◀️

dejan2609 · 2021-05-08T20:10:18Z

@ijuma Gradle 7.0.1 will be released in a few days: https://github.com/gradle/gradle/milestone/173 and after that I will open this PR for a review.

Note: Gradle-7-compatible shadow plugin version is also released: https://github.com/johnrengelman/shadow/releases/tag/7.0.0 but I would pass on that one (at least for this PR).

ijuma · 2021-05-08T20:13:54Z

Any reason not to include the shadow upgrade? If it's reasonably low risk, we can include it here.

dejan2609 · 2021-05-08T20:46:24Z

Any reason not to include the shadow upgrade? If it's reasonably low risk, we can include it here.

I would argue that risk is low/acceptable: potential issues with shadow plugin new version should not pass unnoticed that easily (it is a shadow plugin after all; in case that shadowing goes south some tests will surely fail).

So, we have a deal then: for a next few days I will look for some shadow-plugin blocker issues and in case I don't find any I will include this upgrade as well.

ijuma · 2021-05-08T21:05:13Z

Sounds good.

ijuma · 2021-05-11T00:32:13Z

Looks like there is a change in behavior in Gradle 7 related to resource files that's causing a bunch of tests to fail

dejan2609 · 2021-05-11T16:20:52Z

Looks like there is a change in behavior in Gradle 7 related to resource files that's causing a bunch of tests to fail

Indeed... on my local machine I received around 1% of broken tests (and if I recall correctly they were all related to SSL/TLS, so I was hoping for a better test results here).

Back to drawing board, I guess (I will execute tests again on my side with or without gradle shadow upgrade and then compare with test results here).

Changing PR status to draft (again).

ijuma · 2021-05-12T15:13:48Z

I don't think the issues are related to the shadow plugin. They're related to the gradle upgrade.

dejan2609 · 2021-05-12T17:14:49Z

Probably they are... I will try to isolate issue(s) and compare it with Gradle 7 release notes.

dejan2609 · 2021-05-12T20:39:03Z

Note: I am trying to exact JDK version for Github generated Jenkins build: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-10606/3/pipeline/16 becouse it seems that I have more recent Java 8 version installed.

Test that fail here on Jenkins: https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-10606/1/tests

Around 50 test fail only on my local box:

Gradle command: ./gradlew -PscalaVersion=2.12 test

Environment:

dejan@dejan-HP-ProBook-450-G7:~$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
dejan@dejan-HP-ProBook-450-G7:~$ date
сре, 12. мај 2021.  22:20:54 CEST
dejan@dejan-HP-ProBook-450-G7:~$

All failed test stacktrace starts like this:

org.apache.kafka.common.errors.InvalidConfigurationException: Invalid PEM keystore configs
Caused by: java.security.KeyStoreException: Key protection  algorithm not found: java.security.UnrecoverableKeyException: Encrypt Private Key failed: unrecognized algorithm name: PBEWithSHA1AndDESede
	at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:677)
	at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:577)
	at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)

It seems that these other people have similar issues:

Also, Oracle opened few JIRA related tickets:

https://bugs.openjdk.java.net/browse/JDK-8266261 After 8u282 -> 8u292 update: "unrecognized algorithm name: PBEWithSHA1AndDESede"
https://bugs.openjdk.java.net/browse/JDK-8266279 8u292 NoSuchAlgorithmException unrecognized algorithm name: PBEWithSHA1AndDESede

dejan2609 · 2021-05-12T20:54:28Z

Also... both here on Jenkins and on my laptop there is another type of problems with some test failing like this:

org.apache.directory.api.ldap.schema.extractor.UniqueResourceException: Problem locating LDIF file in schema repository
Multiple copies of resource named 'schema/ou=schema/cn=apachedns/ou=objectclasses/m-oid=1.3.6.1.4.1.18060.0.4.2.3.9.ldif' located on classpath at urls
    jarːfileː/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.directory.api/api-all/1.0.2/5a0f957dfe453ecaa6e7cbb19d2698083116e123/api-all-1.0.2.jar!/schema/ou%3dschema/cn%3dapachedns/ou%3dobjectclasses/m-oid%3d1.3.6.1.4.1.18060.0.4.2.3.9.ldif
    jarːfileː/home/jenkins/.gradle/caches/modules-2/files-2.1/org.apache.directory.api/api-ldap-schema-data/1.0.0/eca194f6baca671c466a4eb3ebc669856d997baa/api-ldap-schema-data-1.0.0.jar!/schema/ou%3dschema/cn%3dapachedns/ou%3dobjectclasses/m-oid%3d1.3.6.1.4.1.18060.0.4.2.3.9.ldif

There are some similar problems recorded (although with Maven... but luckily I'm a Maven aficionado, too 😃):

I will investigate this also (could be some regression in Gradle 7).

dejan2609 · 2021-05-13T00:44:01Z

Short update:

first issue occurs only for Java "1.8.0_292" (not only here but worldwide and it is recognized by Oracle)
second issue... I did report it to a Gradle, but maybe it should be addressed to Apache Directory Server instead (note: this project is using apacheds version 2.0.0-M24 and it seems that this particular issue is solved in apacheds 2.0.0-M26 (good news is that upgrade is not required: good old exclusion should do the trick).

dejan2609 · 2021-05-13T15:41:08Z

Ok, closing the gap here:

Kafka Jenkins could skip JDK update to a version "1.8.0_292" (see previous comments). Note: there is a workaround in Gradle for that version, but I guess it is lot easier just to skip that JDK. FYI @ijuma
as for second issue (that affects all enviroments/all JDK versions): it seems that this dependency org.apache.directory.api:api-ldap-schema-data should be excluded out of configuration responsible for a tests executions (or mayve from all configurations); thing is that same dependency is already excluded out of testCompile configuration here: https://github.com/apache/kafka/blob/2.8.0/build.gradle#L824

ijuma · 2021-05-13T16:38:48Z

@dejan2609 did the exclusion stop working with the upgrade to Gradle 7.0? That seems like the most plausible explanation I've seen so far.

dejan2609 · 2021-05-13T17:01:44Z

@dejan2609 did the exclusion stop working with the upgrade to Gradle 7.0? That seems like the most plausible explanation I've seen so far.

Thing is that exclusion covers testCompile configuration only (in trunk and with Gradle 6.8.3) while with Gradle 7.0.1 (this PR) test executions fail (in more details: they don't even start for a set of tests, that is).

I think I managed to get around this (just finished testing on my end).
Change is tiny (I will not squash commits until we get complete solution).

dejan2609 · 2021-05-13T17:27:19Z

So, just to sum it up:

similar UniqueResourceException issues with api-ldap-schema-data module are solved for a testCompile level here: https://github.com/apache/kafka/blob/2.8.0/build.gradle#L824
Apache DS project solved this in a version 2.0.0.AM26 https://issues.apache.org/jira/browse/DIRSERVER-2066
Kafka project at the moment is using apache ds version 2.0.0-M24 (and upgrade is not trivial)
after Gradle upgrade form 6 to 7 UniqueResourceException errors are thrown again (but this time for a tests execution)
obviously, Gradle 7 changed dependency resolution behavior (and many other things) but I would argue that this is not a Gradle 7 issue per-se

build.gradle

dejan2609 · 2021-05-14T07:00:53Z

PR rebased, commits are squashed into one; tests on Jenkins are looking good: it seems that only one unrelated test failed.

ijuma

LGTM. The JDK 15 job failed due to a timeout, I restarted the build.

dejan2609 · 2021-05-14T14:48:55Z

FYI/note: Gradle 7.0.2 hotfix is released few hours ago (it solves one corner case on Alpine Linux): https://github.com/gradle/gradle/releases/tag/v7.0.2

Let me know if you want me to bump Gradle to 7.0.2 (but it also makes sense to skip this, because issue affects only one Linux distro).
I would bump it although... please have your say @ijuma

ijuma · 2021-05-14T14:54:16Z

Let's bump it.

…hadow plugin (6.1.0 -->> 7.0.0) details: * gradle upgrade: 6.8.3 -->> 7.0.2 ** https://github.com/gradle/gradle/releases/tag/v7.0.0 ** https://github.com/gradle/gradle/releases/tag/v7.0.1 ** https://github.com/gradle/gradle/releases/tag/v7.0.2 * 'distributionSha256Sum' gradle property is included into 'gradle-wrapper.properties' file * gradle shadow plugin upgrade: 6.1.0 -->> 7.0.0 ** https://github.com/johnrengelman/shadow/releases/tag/7.0.0 * remaining configurations obsoleted in Gradle 6 (and removed in Gradle 7) are replaced: ** `compile` -->> `implementation` ** `testCompile` -->> `testImplementation`

dejan2609 · 2021-05-14T15:08:35Z

Bumped/force-pushed (I took the liberty to skipped local testing on my end just to save some time).

dejan2609 · 2021-05-14T18:22:36Z

Test result on Jenkins shows no failures.

ijuma · 2021-05-15T13:10:14Z

Thanks for the PR!

gradle/wrapper/gradle-wrapper.properties

…e-allocations-lz4 * apache-github/trunk: (155 commits) KAFKA-12728: Upgrade gradle to 7.0.2 and shadow to 7.0.0 (apache#10606) KAFKA-12778: Fix QuorumController request timeouts and electLeaders (apache#10688) KAFKA-12754: Improve endOffsets for TaskMetadata (apache#10634) Rework on KAFKA-3968: fsync the parent directory of a segment file when the file is created (apache#10680) MINOR: set replication.factor to 1 to make StreamsBrokerCompatibilityService work with old broker (apache#10673) MINOR: prevent cleanup() from being called while Streams is still shutting down (apache#10666) KAFKA-8326: Introduce List Serde (apache#6592) KAFKA-12697: Add Global Topic and Partition count metrics to the Quorum Controller (apache#10679) KAFKA-12648: MINOR - Add TopologyMetadata.Subtopology class for subtopology metadata (apache#10676) MINOR: Update jacoco to 0.8.7 for JDK 16 support (apache#10654) MINOR: exclude all `src/generated` and `src/generated-test` (apache#10671) KAFKA-12772: Move all transaction state transition rules into their states (apache#10667) KAFKA-12758 Added `server-common` module to have server side common classes. (apache#10638) MINOR Removed copying storage libraries specifically as they are already copied. (apache#10647) KAFKA-5876: KIP-216 Part 4, Apply InvalidStateStorePartitionException for Interactive Queries (apache#10657) KAFKA-12747: Fix flakiness in shouldReturnUUIDsWithStringPrefix (apache#10643) MINOR: remove unnecessary placeholder from WorkerSourceTask#recordSent (apache#10659) MINOR: Remove unused `scalatest` definition from `dependencies.gradle` (apache#10655) MINOR: checkstyle version upgrade: 8.20 -> 8.36.2 (apache#10656) KAFKA-12464: minor code cleanup and additional logging in constrained sticky assignment (apache#10645) ...

dejan2609 changed the title ~~KAFKA-12728: [Work In Progress] gradle version upgrade: 6.8 -->> 7.0~~ KAFKA-12728: [Work In Progress] gradle version upgrade: 6.8.3 -->> 7.0.1 May 8, 2021

dejan2609 force-pushed the trunk branch from 114d072 to a7bd81c Compare May 10, 2021 21:50

dejan2609 changed the title ~~KAFKA-12728: [Work In Progress] gradle version upgrade: 6.8.3 -->> 7.0.1~~ KAFKA-12728: version upgrades: gradle (6.8.3 -->> 7.0.1) and gradle shadow plugin (6.1.0 -->> 7.0.0) May 10, 2021

dejan2609 marked this pull request as ready for review May 10, 2021 21:51

dejan2609 marked this pull request as draft May 11, 2021 16:21

dejan2609 mentioned this pull request May 12, 2021

Regression after Gradle upgrade from 6.8.3 to 7.0.1: test execution fails (due to "Problem locating LDIF file in schema repository" issues) gradle/gradle#17180

Closed

ijuma reviewed May 13, 2021

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

dejan2609 mentioned this pull request May 13, 2021

Gradle 7.0.1 regression: module exclusions from deprecated configurations (compile and testCompile) are skipped in silence (no error has been thrown) gradle/gradle#17187

Closed

dejan2609 force-pushed the trunk branch from ff40d28 to 31de0e5 Compare May 14, 2021 00:06

dejan2609 marked this pull request as ready for review May 14, 2021 00:07

ijuma approved these changes May 14, 2021

View reviewed changes

dejan2609 force-pushed the trunk branch from 31de0e5 to 9033401 Compare May 14, 2021 15:06

dejan2609 changed the title ~~KAFKA-12728: version upgrades: gradle (6.8.3 -->> 7.0.1) and gradle shadow plugin (6.1.0 -->> 7.0.0)~~ KAFKA-12728: version upgrades: gradle (6.8.3 -->> 7.0.2) and gradle shadow plugin (6.1.0 -->> 7.0.0) May 14, 2021

ijuma merged commit ad91c5e into apache:trunk May 15, 2021

lbradstreet reviewed May 15, 2021

View reviewed changes

gradle/wrapper/gradle-wrapper.properties Show resolved Hide resolved

Conversation

dejan2609 commented Apr 28, 2021

Uh oh!

dejan2609 commented May 8, 2021

Uh oh!

ijuma commented May 8, 2021

Uh oh!

dejan2609 commented May 8, 2021

Uh oh!

ijuma commented May 8, 2021

Uh oh!

ijuma commented May 11, 2021

Uh oh!

dejan2609 commented May 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented May 12, 2021

Uh oh!

dejan2609 commented May 12, 2021

Uh oh!

dejan2609 commented May 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented May 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented May 13, 2021

Uh oh!

dejan2609 commented May 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented May 13, 2021

Uh oh!

dejan2609 commented May 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented May 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dejan2609 commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

dejan2609 commented May 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented May 14, 2021

Uh oh!

dejan2609 commented May 14, 2021

Uh oh!

dejan2609 commented May 14, 2021

Uh oh!

ijuma commented May 15, 2021

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dejan2609 commented May 11, 2021 •

edited

Loading

dejan2609 commented May 12, 2021 •

edited

Loading

dejan2609 commented May 12, 2021 •

edited

Loading

dejan2609 commented May 13, 2021 •

edited

Loading

dejan2609 commented May 13, 2021 •

edited

Loading

dejan2609 commented May 13, 2021 •

edited

Loading

dejan2609 commented May 14, 2021 •

edited

Loading

dejan2609 commented May 14, 2021 •

edited

Loading