Publish build scans to develocity.apache.org for build insights #2559

clayburn · 2025-09-13T20:40:30Z

@flyrain - this is the PR we discussed today at Community over Code.

This PR publishes a Build Scan for every CI build and for every local build from an authenticated Apache committer. The build will not fail if publishing fails.

The Build Scans of the Apache Polaris project are published to the Develocity instance at develocity.apache.org, hosted by the Apache Software Foundation and run in partnership between the ASF and Gradle. This Develocity instance has all features and extensions enabled and is freely available for use by the Apache Polaris project and all other Apache projects.

On this Develocity instance, Apache Polaris will have access not only to all of the published Build Scans but other aggregate data features such as:

Dashboards to view all historical Build Scans, along with performance trends over time
Build failure analytics for enhanced investigation and diagnosis of build failures
Test failure analytics to better understand trends and causes around slow, failing, and flaky tests

Please let me know if there are any questions about the value of Develocity or the changes in this pull request and I’d be happy to address them.

adutra · 2025-09-15T08:40:55Z

@clayburn this is great, thank you for this contribution!

Note: you need to run ./gradlew spotlessApply to fix the formatting issues before we can approve.

snazy · 2025-09-15T09:07:42Z

Thanks @clayburn for the PR.
It looks like it's addressing the same functionality as #475

clayburn · 2025-09-15T13:49:35Z

Thanks @adutra. I have pushed up the changes to pass the spotless check.

@snazy - I did not see your PR. A few of the updates in mine are necessary, like the server URL and the secret name. You'll also find that the Common Custom User Data Gradle Plugin in my PR will apply some extra tags and values similar to what you are doing. You can see those docs here. There are other things you are trying to do in that file though, so let me know you'd like to proceed here.

flyrain · 2025-09-15T16:50:03Z

@clayburn thanks for the PR. The develocity.apache.org looks nice to me. I think it will benefit Polaris. A basic question: does ASF Infra set the DEVELOCITY_ACCESS_KEY or Polaris PMC member/committer set it?

dimas-b · 2025-09-15T18:56:00Z

settings.gradle.kts

 }

+plugins {
+  id("com.gradle.develocity") version "4.1.1"


Would it be possible to use https://github.com/apache/polaris/blob/main/gradle/baselibs.versions.toml for version management (as we do for other build plugins)?

It's a "settings plugin", so sadly no.

@snazy is correct. Unfortunately this is not doable

dimas-b · 2025-09-15T18:57:48Z

settings.gradle.kts

+  projectId = "polaris"
+  buildScan {
+    uploadInBackground = !isCI
+    publishing.onlyIf { it.isAuthenticated }


Sending scans when authenticated SGTM... Still, it might be worth mentioning it in the main README for awareness. WDYT?

Apache's Develocity instance requires authentication. (See my other comment).

I added a bit to the README about build scan publication to the README.

dimas-b

Thanks for the PR @clayburn !

snazy

Overall LGTM, and I can add what's really missing from #475 in a follow-up PR.

I think what's missing is the actual secret being added to the apache/polaris GitHub repo (I don't see build scans being published from this PR).

Guess it's worth to add a link to the Gradle docs about how to use build scans in local builds in CONTRIBUTING.md. That however requires an Apache account. We can add support for non-Apache-committers in a follow-up, so they can use Gradle's public instance.

snazy · 2025-09-16T07:13:25Z

settings.gradle.kts

 }

+plugins {
+  id("com.gradle.develocity") version "4.1.1"


It's a "settings plugin", so sadly no.

snazy · 2025-09-16T07:33:48Z

settings.gradle.kts

+  projectId = "polaris"
+  buildScan {
+    uploadInBackground = !isCI
+    publishing.onlyIf { it.isAuthenticated }


Apache's Develocity instance requires authentication. (See my other comment).

clayburn · 2025-09-16T14:09:06Z

A basic question: does ASF Infra set the DEVELOCITY_ACCESS_KEY or Polaris PMC member/committer set it?

I think what's missing is the actual secret being added to the apache/polaris GitHub repo (I don't see build scans being published from this PR).

The DEVELOCITY_ACCESS_KEY secret is an organization secret at the apache org level. It is already present and used by other ASF projects. Unfortunately, with the way GitHub Actions works, pull requests from forks cannot access this secret, so they will not publish build scans. This is something we'd like to provide workflows for doing, but not something we have today.

If you want to verify this pull request, you are welcome to push this branch to the apache/polaris repo and run a build from that branch.

snazy · 2025-09-16T14:16:52Z

The DEVELOCITY_ACCESS_KEY secret is an organization secret at the apache org level.

Ah, true. Totally makes sense.

snazy

LGTM!

Let's ship it :)

jbonofre · 2025-09-16T14:25:31Z

For reference: https://infra.apache.org/gradle.html

jbonofre

It looks good to me, and compliant with https://cwiki.apache.org/confluence/display/INFRA/Project+Onboarding+Instructions+for+Develocity

Thanks !

flyrain

+1 Thanks for the context @clayburn and @jbonofre !

flyrain · 2025-09-16T16:48:22Z

Thanks a lot @clayburn for adding the integration! Thanks @snazy @dimas-b @jbonofre for the review!

* Avoid exceptions on ETag matches (apache#2578) Exceptions have runtime overhead, which is avoidable in this case. * Publish build scans to develocity.apache.org for build insights (apache#2559) * Site: add a blog for apache doris and polaris integration (apache#2571) * Bump: iceberg 1.10 (apache#2586) * Python client: add license check (apache#2580) * Python client: add license check * Python client: add license check * Python client: add license check * test * Enable license check * Fix license check for cryptography (apache#2591) * Update plugin com.gradle.common-custom-user-data-gradle-plugin to v2.4.0 (apache#2582) * Update plugin com.gradle.develocity to v4.2 (apache#2583) * Build: remove unnecessary openapigenerator plugin usages (apache#2592) * docs(changelog): Update the Changelog with 1.0.1, 1.0.0, & 0.9.0 Releases (apache#2587) * Revert "Update plugin com.gradle.develocity to v4.2 (apache#2583)" (apache#2594) This reverts commit 8cc0fad, build scans are not published to the ASF Develocity instance. * Add security report page on the website (apache#2538) This closes apache#2521 * CI: Fix Gradle cache usages (apache#2593) The GH action `ci-incr-build-cache-prepare` calls the `setup-gradle` action with the "right" parameters, so the call of `setup-gradle` in jobs that use `ci-incr-build-cache-prepare` is superfluous. But this also renders the caching ineffective as `setup-gradle` is called w/o `cache-read-only: true` in "child jobs", which then attempt to store their Gradle cache contents. This leads to attempty storing the Gradle cache in `Post Setup Gradle` steps of the "child" jobs, conflicting with other child jobs and also conflicting with `Store Gradle Cache` purpose. "Which cache" is then restored, is rather non-deterministic, but almost always at least "partial" leading to unnecessary (re)builds. * Last merged commit f3d53f0 --------- Co-authored-by: Dmitri Bourlatchkov <dmitri.bourlatchkov@gmail.com> Co-authored-by: Clay Johnson <cjohnson@gradle.com> Co-authored-by: Mingyu Chen (Rayner) <yunyou@selectdb.com> Co-authored-by: Prashant Singh <35593236+singhpk234@users.noreply.github.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Adam Christian <105929021+adam-christian-software@users.noreply.github.com> Co-authored-by: JB Onofré <jbonofre@apache.org>

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Sep 13, 2025

github-project-automation bot added this to Basic Kanban Board Sep 13, 2025

clayburn force-pushed the develocity branch from bc72786 to 567eb11 Compare September 15, 2025 13:37

clayburn force-pushed the develocity branch from 567eb11 to 2eac3c6 Compare September 15, 2025 13:50

flyrain previously approved these changes Sep 15, 2025

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Sep 15, 2025

dimas-b reviewed Sep 15, 2025

View reviewed changes

snazy reviewed Sep 16, 2025

View reviewed changes

Publish build scans to develocity.apache.org for build insights

16258cf

clayburn dismissed flyrain’s stale review via 16258cf September 16, 2025 14:03

clayburn force-pushed the develocity branch from 2eac3c6 to 16258cf Compare September 16, 2025 14:03

snazy approved these changes Sep 16, 2025

View reviewed changes

jbonofre approved these changes Sep 16, 2025

View reviewed changes

dimas-b approved these changes Sep 16, 2025

View reviewed changes

flyrain approved these changes Sep 16, 2025

View reviewed changes

flyrain merged commit 27ec215 into apache:main Sep 16, 2025
14 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Sep 16, 2025

clayburn deleted the develocity branch September 16, 2025 17:18

adam-christian-software referenced this pull request in adam-christian-software/polaris Sep 16, 2025

Publish build scans to develocity.apache.org for build insights (#2559)

c5d1d12

snazy mentioned this pull request Sep 17, 2025

CI/Build: Add Gradle Build Scans #475

Closed

Publish build scans to develocity.apache.org for build insights #2559

Publish build scans to develocity.apache.org for build insights #2559

Uh oh!

Conversation

clayburn commented Sep 13, 2025

Uh oh!

adutra commented Sep 15, 2025

Uh oh!

snazy commented Sep 15, 2025

Uh oh!

clayburn commented Sep 15, 2025

Uh oh!

flyrain commented Sep 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clayburn commented Sep 16, 2025

Uh oh!

snazy commented Sep 16, 2025

Uh oh!

snazy left a comment

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Sep 16, 2025

Uh oh!

jbonofre left a comment

Choose a reason for hiding this comment

Uh oh!

flyrain left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

flyrain commented Sep 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants