MINOR: Capture build scans on ge.apache.org to benefit from deep build insights #13676

clayburn · 2023-05-04T20:46:27Z

This PR publishes a build scan for every CI build on Jenkins and GitHub Actions and for every local build from an authenticated Apache committer. The build will not fail if publishing fails.

The build scans of the Apache Kafka project are published to the Gradle Enterprise instance at ge.apache.org, hosted by the Apache Software Foundation and run in partnership between the ASF and Gradle. This Gradle Enterprise instance has all features and extensions enabled and is freely available for use by the Apache Kafka project and all other Apache projects.

This pull request enhances the functionality of publishing build scans to the publicly available scans.gradle.com by instead publishing build scans to ge.apache.org. On this Gradle Enterprise instance, Apache Kafka will have access not only to all of the published build scans but other aggregate data features such as:

Dashboards to view all historical build scans, along with performance trends over time
Build failure analytics for enhanced investigation and diagnosis of build failures
Test failure analytics to better understand trends and causes around slow, failing, and flaky tests

If interested in exploring a fully populated Gradle Enterprise instance, please explore the builds already connected to ge.apache.org, the Spring project’s instance, or any number of other OSS projects for which we sponsor instances of Gradle Enterprise.

Please let me know if there are any questions about the value of Gradle Enterprise or the changes in this pull request and I’d be happy to address them.

Committer Checklist (excluded from commit message)

Verify design and implementation
Verify test coverage and CI build status
Verify documentation (including upgrade notes)

clayburn · 2023-05-04T20:46:50Z

build.gradle

@@ -39,7 +39,6 @@ plugins {
  id "io.swagger.core.v3.swagger-gradle-plugin" version "2.2.8"

  id "com.github.spotbugs" version '5.0.13' apply false
-  id 'org.gradle.test-retry' version '1.5.2' apply false


The test-retry plugin is bundled with the Gradle Enterprise plugin.

@clayburn is apache Kafka using Gradle entreprise ?

That's what this PR proposes, publishing build scans to the Gradle Enterprise instance at https://ge.apache.org. You can see the configuration in the settings.gradle portion of the PR.

Because the Gradle Enterprise Gradle Plugin bundles in the test-retry plugin, it is necessary to remove these lines. Otherwise test-retry will be loaded twice and fail. The retry functionality remains the same.

What's the license for the gradle enterprise plugin?

Good question, usage of the Gradle Enterprise Gradle Plugin is covered by a license agreement between Gradle and the ASF.

divijvaidya

Thank you for the contribution.

It would be super helpful (perhaps outside of this PR) if you could add a section in our documentation [1] on how to monitor the result of the scans. I am asking this because we had a similar question from a new contributor recently in the email list [2] and you sound like an expert in the CI build(s) at Apache :)

[2] https://lists.apache.org/thread/hsclzdpk623c0t473prtnnh5y1ps8p79
[1] https://kafka.apache.org/contributing

divijvaidya · 2023-05-05T09:02:34Z

settings.gradle

@@ -13,6 +13,38 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.

+plugins {
+    id 'com.gradle.enterprise' version '3.13.1'


Would these plugins be loaded for local builds as well (I understand that they won't be used based on code below)?

Yes, they would be loaded and applied, but they will essentially be a no-op if a build scan will not be published. The Gradle Enterprise Gradle Plugin only monitors the build without affecting the build at all, unless you opt-in to certain features (such as test-retry, for example).

settings.gradle

divijvaidya

Thank you for addressing the comment. The documentation can be taken up outside this PR.

I like this improvement and looking forward to use the new UI to debug our flaky test failures. The infra is hosted by Apache, so I don't see any compliance concerns as well regarding this change.

Here is an example build after this change: https://ge.apache.org/s/6ewlyxy4pt6ii

clayburn · 2023-05-05T15:16:32Z

Thank you for the contribution.

It would be super helpful (perhaps outside of this PR) if you could add a section in our documentation [1] on how to monitor the result of the scans. I am asking this because we had a similar question from a new contributor recently in the email list [2] and you sound like an expert in the CI build(s) at Apache :)

I would not classify myself as an expert in Apache CI builds 🙂, but we have quite a bit of experience helping OSS projects with setup and usage of Gradle Enterprise to benefit from the build insights.

I would be happy to contribute the documentation updates in a future PR.

I like this improvement and looking forward to use the new UI to debug our flaky test failures. The infra is hosted by Apache, so I don't see any compliance concerns as well regarding this change.

We've worked quite closely with the infra team on this rollout.

clayburn · 2023-05-23T20:48:23Z

I updated this PR to the latest version of Gradle Enterprise Gradle Plugin. Is there anything else I need to do to get this PR integrated?

divijvaidya · 2023-05-24T09:15:50Z

The next step here is to get attention of a committer to look into this PR. Usually @ijuma is interested in reviewing build related improvements. Or perhaps @mimaison may be interested to review this.

clayburn · 2023-05-24T13:08:35Z

The test failures appear to come from these two test cases across the following JDKs:

I'm not familiar with these tests, but I suspect these changes did not introduce these failures.

divijvaidya · 2023-06-16T18:15:13Z

@ijuma do you have any other questions wrt this PR? This is a useful feature to debug our flaky tests and I would like to ensure that your questions are addressed since you were involved in this review earlier.

mimaison · 2023-06-20T14:41:10Z

Thanks, this is a useful improvement. I was not aware the ASF had a GE instance. The changes look good to me.

@ijuma any concerns?

ijuma · 2023-06-20T15:30:31Z

As long as there are no restrictions due to this proprietary plugin, I am ok with it. That is, others (including several companies) run these builds too. Is there some official comms from Apache with regards to this?

mimaison · 2023-06-20T16:18:20Z

I did a quick search on the wiki and a few MLs and I can't find an official announcement. All I can find are a few ML threads mentioning ge.apache.org, several projects (Beam, Pulsar, Groovy, etc) already using it and a few Jiras about scans in INFRA .

clayburn · 2023-06-20T16:40:20Z

@ijuma - good question. I do not have an answer myself, but I can get in touch with those involved in the agreement between ASF and Gradle and get back to you on that case.

clayburn · 2023-06-26T16:56:10Z

@ijuma Under our Software License and Sponsorship Agreement, Apache's users and contributors may use the Gradle Enterprise instance and run any ASF builds with GE enabled solely in connection with the development of any ASF open source project. This applies even if the users are employed by a third party or making such contributions to an ASF open source project as part of their work as directed by a third party employer.

clayburn · 2023-06-29T13:25:30Z

@ijuma / @mimaison - does my previous answer address your concerns regarding licensing?

mimaison · 2023-07-12T08:47:19Z

Thanks for the clarifications @clayburn

…eep build insights (apache#13676)" This reverts commit 451fff8.

ijuma · 2023-07-18T23:31:23Z

If I'm reading this right, it seems very restrictive. It's very common for other companies to build AK for their own purposes. If they're not allowed to use gradle enterprise, then it should be disabled by default and enabled via a parameter in the Apache CI build.

…d insights (apache#13676) Reviewers: Mickael Maison <mickael.maison@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Divij Vaidya <diviv@amazon.com>

Fixed the conflict causing the jenkins job failure from AK-> CCS. Below are the PR which caused the conflict. In AK apache#14286 apache#13676 In CCS #901 For #901 and apache#14286 conflict I have kept the versioning changes of CCS in files settings.gradle streams/quickstart/java/pom.xml streams/quickstart/java/src/main/resources/archetype-resources/pom.xml streams/quickstart/pom.xml tests/kafkatest/__init__.py For apache#13676 i don't an actual conflict except few function added in AK. so i brought changes from AK

Fixed the conflict causing the jenkins job failure from AK-> CCS. Below are the PR which caused the conflict. In AK apache#14286 apache#13676 In CCS #901 For #901 and apache#14286 conflict I have kept the versioning changes of CCS in files gradle.properties settings.gradle streams/quickstart/java/pom.xml streams/quickstart/java/src/main/resources/archetype-resources/pom.xml streams/quickstart/pom.xml tests/kafkatest/__init__.py

clayburn commented May 4, 2023

View reviewed changes

divijvaidya reviewed May 5, 2023

View reviewed changes

divijvaidya approved these changes May 5, 2023

View reviewed changes

clayburn force-pushed the connect-to-ge.apache.org branch from 0c84c91 to 2fe78bc Compare May 23, 2023 19:56

divijvaidya requested a review from ijuma May 24, 2023 09:16

clayburn force-pushed the connect-to-ge.apache.org branch from 2fe78bc to ce768d0 Compare June 5, 2023 15:57

clayburn added 3 commits June 9, 2023 07:47

Capture build scans on ge.apache.org to benefit from deep build insights

b1e4826

Comment the rationale for obfuscating ip addresses from build scans

d0ed5d6

Update Gradle Enterprise plugin versions

a0353a4

clayburn force-pushed the connect-to-ge.apache.org branch from ce768d0 to a0353a4 Compare June 9, 2023 12:47

divijvaidya requested a review from mimaison June 9, 2023 13:36

mimaison approved these changes Jun 20, 2023

View reviewed changes

Update to Gradle Enterprise Gradle Plugin 3.13.4

72bf6bf

mimaison merged commit 451fff8 into apache:trunk Jul 12, 2023

clayburn deleted the connect-to-ge.apache.org branch July 17, 2023 13:07

jolshan added a commit to confluentinc/kafka that referenced this pull request Jul 18, 2023

Revert "MINOR: Capture build scans on ge.apache.org to benefit from d…

4fc716a

…eep build insights (apache#13676)" This reverts commit 451fff8.

honshu mentioned this pull request Aug 27, 2023

Fix the AK-> CCS jenkins job failing confluentinc/kafka#943

Closed

honshu mentioned this pull request Aug 27, 2023

{DO NOT MERGE] ccs jenkins fix confluentinc/kafka#944

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MINOR: Capture build scans on ge.apache.org to benefit from deep build insights #13676

MINOR: Capture build scans on ge.apache.org to benefit from deep build insights #13676

clayburn commented May 4, 2023

clayburn May 4, 2023

bmscomp May 5, 2023

clayburn May 5, 2023

ijuma May 24, 2023

clayburn May 24, 2023

divijvaidya left a comment

divijvaidya May 5, 2023

clayburn May 5, 2023

divijvaidya left a comment •

edited

Loading

clayburn commented May 5, 2023

clayburn commented May 23, 2023

divijvaidya commented May 24, 2023

clayburn commented May 24, 2023

divijvaidya commented Jun 16, 2023

mimaison commented Jun 20, 2023

ijuma commented Jun 20, 2023

mimaison commented Jun 20, 2023

clayburn commented Jun 20, 2023

clayburn commented Jun 26, 2023

clayburn commented Jun 29, 2023

mimaison commented Jul 12, 2023

ijuma commented Jul 18, 2023

MINOR: Capture build scans on ge.apache.org to benefit from deep build insights #13676

MINOR: Capture build scans on ge.apache.org to benefit from deep build insights #13676

Conversation

clayburn commented May 4, 2023

Committer Checklist (excluded from commit message)

clayburn May 4, 2023

Choose a reason for hiding this comment

bmscomp May 5, 2023

Choose a reason for hiding this comment

clayburn May 5, 2023

Choose a reason for hiding this comment

ijuma May 24, 2023

Choose a reason for hiding this comment

clayburn May 24, 2023

Choose a reason for hiding this comment

divijvaidya left a comment

Choose a reason for hiding this comment

divijvaidya May 5, 2023

Choose a reason for hiding this comment

clayburn May 5, 2023

Choose a reason for hiding this comment

divijvaidya left a comment • edited Loading

Choose a reason for hiding this comment

clayburn commented May 5, 2023

clayburn commented May 23, 2023

divijvaidya commented May 24, 2023

clayburn commented May 24, 2023

divijvaidya commented Jun 16, 2023

mimaison commented Jun 20, 2023

ijuma commented Jun 20, 2023

mimaison commented Jun 20, 2023

clayburn commented Jun 20, 2023

clayburn commented Jun 26, 2023

clayburn commented Jun 29, 2023

mimaison commented Jul 12, 2023

ijuma commented Jul 18, 2023

divijvaidya left a comment •

edited

Loading