-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI][flaky] reporting for PRs in GitHub #21853
Conversation
Jenkinsfile
Outdated
notifyBuildResult(prComment: true, slackComment: true, slackNotify: (isBranch() || isTag())) | ||
notifyBuildResult(prComment: true, | ||
slackComment: true, slackNotify: (isBranch() || isTag()) | ||
analyzeFlakey: true, flakyReportIdx: "reporter-beats-pipeline-master") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know if we should use as many indices as release branches we got and use the CHANGE_TARGET
for the PRs or BRANCH_NAME for the branches/tags.
But, anytime there is a new release, then we might need to create a new indice.
@cachedout , what are your thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this is an issue we need to figure out. I think that the thing to do here is to use one index per project and then include a field that we can filter against when searching for documents, but to achieve that I think we need to make some changes to the way we actually generate the flaky test analysis.
For now, I propose that we keep it as-is and address this in the test analyzer itself. Then we can come back and change the pipelines as-needed.
Jenkinsfile
Outdated
notifyBuildResult(prComment: true, slackComment: true, slackNotify: (isBranch() || isTag())) | ||
notifyBuildResult(prComment: true, | ||
slackComment: true, slackNotify: (isBranch() || isTag()) | ||
analyzeFlakey: true, flakyReportIdx: "reporter-beats-pipeline-master") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this is an issue we need to figure out. I think that the thing to do here is to use one index per project and then include a field that we can filter against when searching for documents, but to achieve that I think we need to make some changes to the way we actually generate the flaky test analysis.
For now, I propose that we keep it as-is and address this in the test analyzer itself. Then we can come back and change the pipelines as-needed.
jenkins run the tests please |
Jenkins run the tests please |
1 similar comment
Jenkins run the tests please |
…laky-test-analyser * upstream/master: (22 commits) [Ingest Manager] Prevent reporting ecs version twice (elastic#21616) [CI] Use google storage to keep artifacts (elastic#21910) Update docs.asciidoc (elastic#21849) Kubernetes leaderelection improvements (elastic#21896) Apply name changes to elastic agent docs (elastic#21549) Add 7.7.1 relnotes to 7.8 docs (elastic#21937) (elastic#21941) [libbeat] Fix potential deadlock in the disk queue + add more unit tests (elastic#21930) Refactor docker watcher to fix flaky test and other small issues (elastic#21851) [CI] Add stage name in the step (elastic#21887) [docs] Remove extra word in autodiscover docs (elastic#21871) [CI] lint stage doesn't produce test reports (elastic#21888) Add tests of reader of filestream input (elastic#21814) [Ingest Manager] Use local temp instead of system one (elastic#21883) chore: delegate variant pushes to the right method (elastic#21861) [CI] kind setup fails sometimes (elastic#21857) Fix panic on add_docker_metadata close (elastic#21882) Add tests for fileProspector in filestream input (elastic#21712) [Filebeat][okta] Fix okta pagination (elastic#21797) Add cloud.account.id into add_cloud_metadata for gcp (elastic#21776) Fix syslog RFC 5424 parsing in CheckPoint module (elastic#21854) ...
💚 Flaky test reportTests succeeded. Test stats 🧪
|
Jenkins run the tests please |
Jenkins run the tests please |
Build and Test / Filebeat Windows / test_close_renamed – test_harvester.Test Oct 21, 2020 @ 04:00:25.019 Build and Test / Libbeat / Libbeat oss / TestClientPublishEventKerberosAware – elasticsearch
…beats into feature/support-flaky-test-analyser * 'feature/support-flaky-test-analyser' of github.com:v1v/beats: [CI] Enable winlogbeat (elastic#22142)
This reverts commit 5532351.
This reverts commit 2ec9d4a.
This reverts commit 3a701bd.
This reverts commit 6e6e094.
This reverts commit ba9957b.
This reverts commit c6957e6.
…laky-test-analyser * upstream/master: Add new licence status: expired (elastic#22180) [filebeat][okta] Make cursor optional for okta and update docs (elastic#22091) Add documentation of filestream input (elastic#21615) [Ingest Manager] Skip flaky gateway tests elastic#22177 [CI] set env variable for the params (elastic#22143) Fix zeek connection pipeline (elastic#22151) Fix Google Cloud Function configuration file issues (elastic#22156) Remove old TODO on kubernetes node update (elastic#22074)
…laky-test-analyser * upstream/master: Add new licence status: expired (elastic#22180) [filebeat][okta] Make cursor optional for okta and update docs (elastic#22091) Add documentation of filestream input (elastic#21615) [Ingest Manager] Skip flaky gateway tests elastic#22177 [CI] set env variable for the params (elastic#22143) Fix zeek connection pipeline (elastic#22151) Fix Google Cloud Function configuration file issues (elastic#22156) Remove old TODO on kubernetes node update (elastic#22074)
…beats into feature/support-flaky-test-analyser * 'feature/support-flaky-test-analyser' of github.com:v1v/beats:
* upstream/master: (93 commits) Update commands used in the quick start (elastic#22248) Add interval documentation to `monitor` metricset (elastic#22152) [CI] enable x-pack/packetbeat in the CI (elastic#22252) Fix awscloudwatch input documentation (elastic#22247) Add support for different Azure Cloud environments in the metricbeat azure module (elastic#21044) [CI] support windows-2008-r2 (elastic#19791) protect against accessing undefined variables in sysmon module (elastic#22236) [CI] archive only if failed steps (elastic#22220) Add pe fields to Sysmon module (elastic#22217) [CI][flaky] Support 7.x branches and PRs (elastic#22197) Perfmon - Fix regular expressions to comply to multiple parentheses in instance name and object (elastic#22146) ci: improve linting speed (elastic#22103) Move cloudfoundry tags with metadata to common metadata fields (elastic#22150) [Docs] Update custom beat docs (elastic#22194) [Ingest Manager] Agent fix snapshot download for upgrade (elastic#22175) Update shared-autodiscover.asciidoc (elastic#21827) [DOCS] Warn about compression and Azure Event Hub for Kafka (elastic#21578) [CI][flaky] reporting for PRs in GitHub (elastic#21853) [Packetbeat] Create x-pack magefile (elastic#21979) [Elastic Agent] Fix deb/rpm installation (elastic#22153) ...
* upstream/master: (93 commits) Update commands used in the quick start (elastic#22248) Add interval documentation to `monitor` metricset (elastic#22152) [CI] enable x-pack/packetbeat in the CI (elastic#22252) Fix awscloudwatch input documentation (elastic#22247) Add support for different Azure Cloud environments in the metricbeat azure module (elastic#21044) [CI] support windows-2008-r2 (elastic#19791) protect against accessing undefined variables in sysmon module (elastic#22236) [CI] archive only if failed steps (elastic#22220) Add pe fields to Sysmon module (elastic#22217) [CI][flaky] Support 7.x branches and PRs (elastic#22197) Perfmon - Fix regular expressions to comply to multiple parentheses in instance name and object (elastic#22146) ci: improve linting speed (elastic#22103) Move cloudfoundry tags with metadata to common metadata fields (elastic#22150) [Docs] Update custom beat docs (elastic#22194) [Ingest Manager] Agent fix snapshot download for upgrade (elastic#22175) Update shared-autodiscover.asciidoc (elastic#21827) [DOCS] Warn about compression and Azure Event Hub for Kafka (elastic#21578) [CI][flaky] reporting for PRs in GitHub (elastic#21853) [Packetbeat] Create x-pack magefile (elastic#21979) [Elastic Agent] Fix deb/rpm installation (elastic#22153) ...
What does this PR do?
Enable the flaky test analyser using the
master
branch as a source of truth for the time being.Why is it important?
This will help to automate the flaky test analyser process:
flaky-test,ci-reported
Then contributors and reviewers will be able to understand the whole test context and whether those failures were already reported as flaky in the past.
IMPORTANT: the current GitHub issues automation won't create more than 3 issues per build, this will allow us to avoid any kind of corner case if there are suddenly a big number of flaky test failures.
What flaky tests failures have been found?
With the flaky test analyser:
Expand to view
With the
runbld
watcher analyser:Expand to view
Screenshots
When flaky tests are found but not reported
For instance, if there were two test failures which were flaky then you will see something like the below screenshot:
When flaky tests are found and reported
And a GH issue will be created
And if the GH issue was created previously then there will be a new comment:
Issues
Consumes elastic/apm-pipeline-library#754 and elastic/apm-pipeline-library#791
Follow ups