-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Moved Gradle check into OpenSearch repo #1665
Conversation
Can one of the admins verify this patch? |
Let's see if this actually succeeds on GHA. I was under the impression that it's too slow/takes too long. What are the time differences? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See comments.
.github/workflows/check.yml
Outdated
cd ../ | ||
REPORTS_DIR=`find ./search -name reports` | ||
zip -9 -q -r gradle_check_${BUILD_NUMBER}_reports.zip ${REPORTS_DIR} | ||
ls | grep "gradle_check_${BUILD_NUMBER}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this do?
.github/workflows/check.yml
Outdated
- name: Run Gradle | ||
run: | | ||
echo "./gradlew check --no-daemon --no-scan" | ||
./gradlew check --no-daemon --no-scan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAIK if this is the last line the build fails and we're done, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was following the script from the private repo. Let me know if you are suggesting to change the run
something like:
echo "./gradlew check --no-daemon --no-scan --parallel"
./gradlew check --no-daemon --no-scan ----parallel
GRADLE_CHECK_STATUS=$?
if [ "$GRADLE_CHECK_STATUS" != 0 ]
then
echo Gradle Check Failed!
exit $GRADLE_CHECK_STATUS
fi
`
.github/workflows/check.yml
Outdated
if [ "$GRADLE_CHECK_STATUS" != 0 ] | ||
then | ||
echo Gradle Check Failed! | ||
exit 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should return $GRADLE_CHECK_STATUS
.github/workflows/check.yml
Outdated
- name: Run Gradle | ||
run: | | ||
echo "./gradlew check --no-daemon --no-scan" | ||
./gradlew check --no-daemon --no-scan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we have multiple projects, we can leverage --parallel
, it will make the runs faster.
Ref: https://docs.gradle.org/current/userguide/performance.html
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense. Let me test it on my local.
This is similar to what I saw before, not sure if disabling or fixing or rewriting these tests is the thing to do. |
The time it took is almost the same as Jenkins run but for a failed build. |
I remembered previously @Rishikesh1159 did some experiments to run gradle check through Github Actions, the whole check took 120 minutes in the GitHub hosted machines. |
It's worth trying a little harder this time and seeing whether it's because some timeouts are being hit causing retries, or something else. Ideally we'd use GHA for gradle checks. |
❌ Gradle Wrapper Validation failure 0c818d8 :alert: Gradle Wrapper integrity has been altered |
Having gradle check in the open would bring in:
For next steps, I would:
|
@saratvemulapalli hosted runners inside AWS infra are a non-starter, this has been explored plenty - there are alternatives to GHA but they aren't better than our Jenkins instance otherwise |
Interesting, curious why though? |
✅ Gradle Precommit success 0c818d8 |
I don't have the detail, but I'll point you where to ask. There's also https://github.com/aws-actions/aws-codebuild-run-build, which may be a better solution. |
When I tried to run gradle check on Github Actions, the rate of Random Test fails was higher compared to jenkins CI. This is because of slower machines Github Actions uses. There were couple integTests constantly failing and I had exclude them to see if gradle check works on Github Actions. |
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
0c818d8
to
5d97934
Compare
@owaiskazi19 So this just works if we fix the timeouts/random failures? |
Yeah, it does and takes around 180 mins if there's no flaky tests. |
Can we parallelize, get this down to ~1h? |
I have it like |
Also, I think if we can get a better instance for GHA that would help to speed up the gradle check. |
The failure on GHA is because of https://github.com/opensearch-project/OpenSearch/blob/main/server/src/test/java/org/opensearch/indices/IndexingMemoryControllerTests.java#L488-L490 and https://github.com/opensearch-project/OpenSearch/blob/main/server/src/test/java/org/opensearch/indices/IndexingMemoryControllerTests.java#L497.
|
This is nice, do you know if all the tests (which are ignored on your fork) are flaky, and have issues opened ? |
I might be missing a few but for the most recent failures I have opened: #1852 and reopened: #39. |
I'm thinking multiple jobs across different kinds of tests. For example, If there's a way to generate a list of all the test tasks with |
The simplest I can think of is splitting BWC tests and the rest to start with. Because they take a lot of time to run on Jenkins too. |
Closing this one. Created #2166. |
Signed-off-by: Owais Kazi owaiskazi19@gmail.com
Description
Moved gradle check from private jenkins instance to OpenSearch repo.
Sample run: https://github.com/owaiskazi19/OpenSearch/runs/4705950671?check_suite_focus=true
Issues Resolved
Part of #2395
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.