[feat][ci] Add integration to ge.apache.org Gradle Enterprise server #19133
Conversation
lhotari
requested review from
merlimat,
eolivelli,
tisonkun,
nicoloboschi and
michaeljmarshall
Codecov Report
@@ Coverage Diff @@
## master #19133 +/- ##
=============================================
- Coverage 47.22% 34.16% -13.07%
+ Complexity 10713 6485 -4228
=============================================
Files 713 607 -106
Lines 69697 57647 -12050
Branches 7485 5994 -1491
=============================================
- Hits 32914 19693 -13221
- Misses 33096 35302 +2206
+ Partials 3687 2652 -1035
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
/pulsarbot run-failure-checks |
There was a problem hiding this comment.
@lhotari Thanks for your contribution!
It should be good to try. I have some questions here:
- How do you imagine we make use of the scan report?
- How much time does the new step consume?
- Is it possible to replace codecov with this scan so we reduce too many reports?
|
Also, maybe we can tag |
there are many ways to make use of the scan reports. For example, the test execution reports are useful for analysing flaky tests and slow tests. Here's an example of test execution reports: For other parts of the build, the build scan contains detailed break downs of where time is spent in the build. It's simply a way to get a better understanding of what happens during the build. One of the challenges is that Gradle Enterprise doesn't seem to provide a nice way to group the related builds in list views. There are links in the build scan report header to find other builds in the same workflow run or for the same git commit hash.
The Gradle Enterprise maven extension collects the information and logs and sends them when the build finishes. It doesn't seem to add more than a few seconds to execution. One detail is that this solution is only enabled for builds that happen on the apache/pulsar branches. The reason for this is that Gradle Enterprise has a secret token that is used for publishing to ge.apache.org . If we'd like to publish reports for pull requests, the token would have to be made public. I don't think that there are plans for doing that at the moment.
This isn't a replacement for codecov. Currently codecov integration in apache/pulsar seems to be invalid and the results don't make sense. I think we should disable codecov reporting until it is fixed and it provides consistent results. |
@tisonkun I created lhotari#123 to ensure that the changes don't break builds in forks. You won't be able to publish build scans to ge.apache.org since the secret token is only available for builds on branches that exist in apache/pulsar repository. As a solution for forked builds, I think we could add a way to use public Gradle builds scans available at https://scans.gradle.com/ . Currently, the maven extension will only get activated when the token for Gradle Enterprise is available. The activation happens in .github/actions/gradle-enterprise/action.yml by copying |
|
There seems to be some compatibility issue with Gradle Enterprise maven extension and the way how we retry tests with TestNG. There's an error in build scan: https://ge.apache.org/s/lfbhlzzwce72c/failure?focused-exception-line=0-5#1 in GitHub Actions logs: https://github.com/apache/pulsar/actions/runs/3843180558/jobs/6546220728#step:11:1408 |
|
I have isolated the issue in https://github.com/lhotari/gradle-enterprise-duplicate-test-id-repro |
There's now a workaround in place to handle the issue. There was only one occurrence of the issue in the Pulsar code base. |
| import org.testng.annotations.Test; | ||
|
|
||
| @Test(groups = "broker-api") | ||
| public class SimpleSchemaWithSchemaValidationEnforcedTest extends SimpleSchemaTest { |
There was a problem hiding this comment.
Please file an issue and add a reference for this trick.
@tisonkun I have reported this issue https://github.com/lhotari/gradle-enterprise-duplicate-test-id-repro to Gradle support. They will work on a fix eventually. One of the workarounds that was suggested, was switching to use JUnit Jupiter to run tests with testng-engine. (the problem doesn't appear with testng-engine) This is probably something we would like to do if we would like to migrate to Junit 5 eventually. testng-engine doesn't support testng xml test suites that are used by Pulsar integration tests. |
Motivation
ASF Infra team together with Gradle Inc. has setup a dedicated Gradle Enterprise instance for Apache projects at https://ge.apache.org/ .
Gradle Enterprise works for both Gradle builds and Maven builds. There's a Maven extension which provides the integration for Maven builds, https://docs.gradle.com/enterprise/maven-extension/ .
You can find more information about the benefits of Gradle Enterprise on https://gradle.com . Gradle Enterprise is also for Maven builds!
Here's an example of builds scans for all Maven builds that are run as part of Pulsar CI workflow: https://ge.apache.org/scans?search.names=Git%20commit%20id%20short&search.timeZoneId=Europe/Helsinki&search.values=db7ab997&selection.buildScanB=ohtx53k6ymsgc
There are 69 maven builds that are run in total.
The build scans are also linked from the GitHub Actions results page, for example https://github.com/apache/pulsar/actions/runs/3839199234#summary-10438505199 shows this.
Mailing list discussion: https://lists.apache.org/thread/d36xbj8b7ntcyp85lv0730xfj4hodqxj
Modifications
Add configuration for Gradle Enterprise which will be activated only for builds in branches on apache/pulsar when a secret called GE_ACCESS_TOKEN exists.
A new environment variable called JOB_NAME is added so that the name of the GitHub Actions CI job can be passed on to Gradle Enterprise build scan values. GitHub Actions has a limitation that the job name isn't available as an environment variable.
Documentation
docdoc-requireddoc-not-neededdoc-complete