ci: improve linting speed #22103

kuisathaverat · 2020-10-22T16:38:35Z

What does this PR do?

It splits the make check target to allow parallelization.

Why is it important?

The make check takes about 15 min, let's see how much we can speed up it. Because we moved part of the Liting to the beats pipelines we have reduced the total build time about 40 min (1h 50min before the change now ~1h), and the first response to lest than 4 min (it was 15-17 min before the change), so developers will know if the PR pass the basics in 5 min(Linting)

Before

After

Checklist

~~[ ] My code follows the style guidelines of this project~~
~~[ ] I have commented my code, particularly in hard-to-understand areas~~
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
~~[ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

elasticmachine · 2020-10-22T16:38:40Z

Pinging @elastic/ingest-management (Team:Ingest Management)

elasticmachine · 2020-10-22T16:51:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Started by user Ivan Fernandez Calvo]
Start Time: 2020-10-28T10:41:11.680+0000
Duration: 63 min 2 sec

Test stats 🧪

Test	Results
Failed	0
Passed	16370
Skipped	1344
Total	17714

Steps errors

Expand to view the steps failures

Name: Compress
- Description: tar --exclude=test-build-artifacts-x-pack/winlogbeat-Lint-linux && ubuntu-18.tgz -czf test-build-artifacts-x-pack/winlogbeat-Lint-linux && ubuntu-18.tgz .
- Duration: 0 min 0 sec
- Start Time: 2020-10-28T10:55:50.337+0000
- log

jsoriano

This is great, my only concern is that we may be losing some check, or some condition to trigger some check. But the improvement may worth the risk of needing to solve or readjust some things in the following days/weeks.

After this change we could review the targets we have, there are many check/lint/fmt/update... targets and I am not sure if we need all of them.

developers will know if the PR pass the basics in 5 min(Linting)

That's a lot to say 🙂 We have to take into account that this 5 mins linting is only doing a very basic linting of the code. Generation of files (update and check targets) is the heavier part, the one that is easier to forget when committing, and will be checked later, taking some extra minutes.
I wonder if starting the stages in parallel may take much more resources. i.e a change in libbeat can trigger all builds, lint stage in libbeat might fail, but many other heavy stages for metricbeat, filebeat... are running. Could we abort and mark the build as failed immediately after any of the lint stages fail?

jsoriano · 2020-10-25T11:16:23Z

deploy/kubernetes/Jenkinsfile.yml

@@ -11,5 +11,7 @@ when:
    tags: true                 ## for all the tags
 platform: "linux && ubuntu-18" ## default label for all the stages
 stages:
+    lint:
+       make: "make -C deploy/kubernetes all"


This check should be executed when libbeat/docs/version.asciidoc changes.

(Long-term we should probably stop commiting these files and move them to docs build)

And I guess we need to check for changes so it detects anything.

Suggested change

make: "make -C deploy/kubernetes all"

make: |

make -C deploy/kubernetes all;

make check-no-changes;

@kuisathaverat I see you have added the trigger on changes in the version file, thanks!

But the other point in this comment about adding a check is still pending to be addressed.

It is https://github.com/elastic/beats/blob/master/deploy/kubernetes/Jenkinsfile.yml#L5

But the other point in this comment about adding a check is still pending to be addressed.

make -C deploy/kubernetes all is not checking anything, I think we also need to run make check-no-changes.

yep, we need to runmake check-no-changes also

Could you please add it?

jsoriano · 2020-10-25T11:16:40Z

Jenkinsfile

-            cmd(label: 'make check', script: 'make check')
+            cmd(label: "make check-python", script: "make check-python")
+            cmd(label: "make check-go", script: "make check-go")
+            cmd(label: "Check for changes", script: "make check-no-changes")


How will we decide what goes in this linting and what goes in the project-specific linting? For example this linting is not checking the license headers now.

I've moved the checks that are in a foreach in the main Makefile, so I think that every check that it is implemented on the beats should run only on beans. Indeed we should try to add check only to the beans to allow parallelize the task.

Have you tried make -j <n>?

the VMs are not really powerful machines that do not have too many cores and are not dedicated hardware, so this could decrease a little the time(maybe 5-10min or less) of the task but far away from this approach that uses a VM for each stage (40min).

jsoriano · 2020-10-25T11:18:37Z

filebeat/Jenkinsfile.yml

@@ -13,6 +13,11 @@ when:
    tags: true                 ## for all the tags
 platform: "linux && ubuntu-18" ## default label for all the stages
 stages:
+    Lint:


Nit. This stage is upercased in some files, and lowercased in others.

yep I saw it on the GitHub checks

jsoriano · 2020-10-25T11:20:00Z

x-pack/packetbeat/Jenkinsfile.yml

@@ -13,6 +14,12 @@ when:
    tags: true                 ## for all the tags
 platform: "linux && ubuntu-18" ## default label for all the stages
 stages:
+# TODO confirm this works


Does it work? 🙂

I have deleted the file is not used, @v1v has to confirm it

It is not used indeed, but I left there in case it will be supported in the future, so we can delete it for simplicity

I have to re-add the file because there is a magefile on the master branch, I've to check if we can backport this to all branches

Jenkinsfile

dev-tools/Jenkinsfile.yml

kuisathaverat · 2020-10-27T19:15:27Z

/test

elasticmachine · 2020-10-28T11:44:33Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	16370
Skipped	1344
Total	17714

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # x-pack/packetbeat/Jenkinsfile.yml

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # Jenkinsfile.yml # auditbeat/Jenkinsfile.yml # filebeat/Jenkinsfile.yml # heartbeat/Jenkinsfile.yml # journalbeat/Jenkinsfile.yml # libbeat/Jenkinsfile.yml # packetbeat/Jenkinsfile.yml # x-pack/auditbeat/Jenkinsfile.yml # x-pack/elastic-agent/Jenkinsfile.yml # x-pack/filebeat/Jenkinsfile.yml # x-pack/functionbeat/Jenkinsfile.yml # x-pack/libbeat/Jenkinsfile.yml # x-pack/packetbeat/Jenkinsfile.yml

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # x-pack/packetbeat/Jenkinsfile.yml

jsoriano · 2020-10-28T12:42:45Z

Hey @kuisathaverat, there was a pending issue #22103 (comment), it seems to me that we are losing this check after this change. Could you please add it?

* upstream/master: (93 commits) Update commands used in the quick start (elastic#22248) Add interval documentation to `monitor` metricset (elastic#22152) [CI] enable x-pack/packetbeat in the CI (elastic#22252) Fix awscloudwatch input documentation (elastic#22247) Add support for different Azure Cloud environments in the metricbeat azure module (elastic#21044) [CI] support windows-2008-r2 (elastic#19791) protect against accessing undefined variables in sysmon module (elastic#22236) [CI] archive only if failed steps (elastic#22220) Add pe fields to Sysmon module (elastic#22217) [CI][flaky] Support 7.x branches and PRs (elastic#22197) Perfmon - Fix regular expressions to comply to multiple parentheses in instance name and object (elastic#22146) ci: improve linting speed (elastic#22103) Move cloudfoundry tags with metadata to common metadata fields (elastic#22150) [Docs] Update custom beat docs (elastic#22194) [Ingest Manager] Agent fix snapshot download for upgrade (elastic#22175) Update shared-autodiscover.asciidoc (elastic#21827) [DOCS] Warn about compression and Azure Event Hub for Kafka (elastic#21578) [CI][flaky] reporting for PRs in GitHub (elastic#21853) [Packetbeat] Create x-pack magefile (elastic#21979) [Elastic Agent] Fix deb/rpm installation (elastic#22153) ...

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # Jenkinsfile # Makefile # auditbeat/Jenkinsfile.yml # filebeat/Jenkinsfile.yml # heartbeat/Jenkinsfile.yml # journalbeat/Jenkinsfile.yml # libbeat/Jenkinsfile.yml # metricbeat/Jenkinsfile.yml # packetbeat/Jenkinsfile.yml # winlogbeat/Jenkinsfile.yml # x-pack/auditbeat/Jenkinsfile.yml # x-pack/dockerlogbeat/Jenkinsfile.yml # x-pack/elastic-agent/Jenkinsfile.yml # x-pack/filebeat/Jenkinsfile.yml # x-pack/functionbeat/Jenkinsfile.yml # x-pack/libbeat/Jenkinsfile.yml # x-pack/packetbeat/Jenkinsfile.yml # x-pack/winlogbeat/Jenkinsfile.yml

* ci: improve linting speed (#22103) * ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # x-pack/packetbeat/Jenkinsfile.yml * Apply suggestions from code review Co-authored-by: cachedout <mike.place@elastic.co> Co-authored-by: cachedout <mike.place@elastic.co>

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # x-pack/packetbeat/Jenkinsfile.yml

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # Jenkinsfile.yml # auditbeat/Jenkinsfile.yml # filebeat/Jenkinsfile.yml # heartbeat/Jenkinsfile.yml # journalbeat/Jenkinsfile.yml # libbeat/Jenkinsfile.yml # packetbeat/Jenkinsfile.yml # x-pack/auditbeat/Jenkinsfile.yml # x-pack/elastic-agent/Jenkinsfile.yml # x-pack/filebeat/Jenkinsfile.yml # x-pack/functionbeat/Jenkinsfile.yml # x-pack/libbeat/Jenkinsfile.yml # x-pack/packetbeat/Jenkinsfile.yml

* ci: improve linting speed (#22103) * ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # Jenkinsfile # Makefile # auditbeat/Jenkinsfile.yml # filebeat/Jenkinsfile.yml # heartbeat/Jenkinsfile.yml # journalbeat/Jenkinsfile.yml # libbeat/Jenkinsfile.yml # metricbeat/Jenkinsfile.yml # packetbeat/Jenkinsfile.yml # winlogbeat/Jenkinsfile.yml # x-pack/auditbeat/Jenkinsfile.yml # x-pack/dockerlogbeat/Jenkinsfile.yml # x-pack/elastic-agent/Jenkinsfile.yml # x-pack/filebeat/Jenkinsfile.yml # x-pack/functionbeat/Jenkinsfile.yml # x-pack/libbeat/Jenkinsfile.yml # x-pack/packetbeat/Jenkinsfile.yml # x-pack/winlogbeat/Jenkinsfile.yml * Apply suggestions from code review * Update Makefile * Update Makefile * Update Jenkinsfile * Apply suggestions from code review

* ci: improve linting speed * fix: use multiple agents for the matrix * fix: add python-env * fix: set make folder parameter * feat: move k8s check and dev-tools to their own stages * fix: lint in a single stage is faster * fix: suggested changes * chore:change triggers * chore: update for packetbeat # Conflicts: # Jenkinsfile.yml # auditbeat/Jenkinsfile.yml # filebeat/Jenkinsfile.yml # heartbeat/Jenkinsfile.yml # journalbeat/Jenkinsfile.yml # libbeat/Jenkinsfile.yml # packetbeat/Jenkinsfile.yml # x-pack/auditbeat/Jenkinsfile.yml # x-pack/elastic-agent/Jenkinsfile.yml # x-pack/filebeat/Jenkinsfile.yml # x-pack/functionbeat/Jenkinsfile.yml # x-pack/libbeat/Jenkinsfile.yml # x-pack/packetbeat/Jenkinsfile.yml

ci: improve linting speed

be932d7

kuisathaverat added automation ci Team:Automation Label for the Observability productivity team labels Oct 22, 2020

kuisathaverat requested a review from a team October 22, 2020 16:38

kuisathaverat requested review from a team as code owners October 22, 2020 16:38

kuisathaverat self-assigned this Oct 22, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label Team:Ingest Management and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 22, 2020

kuisathaverat removed the Team:Ingest Management label Oct 22, 2020

kuisathaverat requested a review from jsoriano October 22, 2020 16:39

mdelapenya approved these changes Oct 22, 2020

View reviewed changes

botelastic bot added the Team:Ingest Management label Oct 22, 2020

kuisathaverat force-pushed the speed_up_lint branch from 89ad5b9 to be932d7 Compare October 22, 2020 16:41

fix: use multiple agents for the matrix

362b2e5

kuisathaverat added 4 commits October 22, 2020 19:09

fix: add python-env

59d0018

fix: set make folder parameter

c682f3b

feat: move k8s check and dev-tools to their own stages

284add9

fix: lint in a single stage is faster

36d600a

kuisathaverat requested review from andrewkroh and urso October 23, 2020 09:29

jsoriano reviewed Oct 25, 2020

View reviewed changes

jsoriano added needs_backport PR is waiting to be backported to other branches. v6.8.13 v7.10.0 v7.11.0 labels Oct 25, 2020

kuisathaverat added 4 commits October 26, 2020 10:52

chore:change triggers

b28b754

Merge branch 'master' of github.com:elastic/beats into speed_up_lint

0b9e387

Merge branch 'master' of github.com:elastic/beats into speed_up_lint

8e33fce

chore: update for packetbeat

81aebd2

kuisathaverat merged commit 46fc975 into elastic:master Oct 28, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Oct 28, 2020

This was referenced Oct 28, 2020

Cherry-pick to 7.x: ci: improve linting speed (#22103) #22221

Merged

Cherry-pick to 7.9: ci: improve linting speed (#22103) #22222

Merged

kuisathaverat mentioned this pull request Oct 28, 2020

Cherry-pick to 7.10: ci: improve linting speed (#22103) #22223

Merged

kuisathaverat mentioned this pull request Nov 3, 2020

Cherry-pick to 6.8: ci: improve linting speed (#22103) #22382

Merged

kuisathaverat removed the [zube]: Done label Dec 1, 2020

This was referenced Dec 14, 2020

[CI] Run linting for only docs #23099

Merged

[CI] X-pack changes should trigger OSS changes if doc changes #23169

Closed

kuisathaverat deleted the speed_up_lint branch December 24, 2020 11:06

v1v mentioned this pull request Apr 20, 2021

CI: split stages in categories #25112

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: improve linting speed #22103

ci: improve linting speed #22103

kuisathaverat commented Oct 22, 2020 •

edited

Loading

elasticmachine commented Oct 22, 2020

elasticmachine commented Oct 22, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

jsoriano left a comment

jsoriano Oct 25, 2020

jsoriano Oct 27, 2020

kuisathaverat Oct 28, 2020

jsoriano Oct 28, 2020

kuisathaverat Oct 28, 2020

jsoriano Oct 28, 2020

jsoriano Oct 25, 2020

kuisathaverat Oct 26, 2020

urso Oct 26, 2020

kuisathaverat Oct 27, 2020 •

edited

Loading

jsoriano Oct 25, 2020

kuisathaverat Oct 26, 2020

jsoriano Oct 25, 2020

kuisathaverat Oct 26, 2020

v1v Oct 26, 2020

kuisathaverat Oct 27, 2020

kuisathaverat commented Oct 27, 2020

elasticmachine commented Oct 28, 2020

Test stats 🧪

jsoriano commented Oct 28, 2020

-       make: "make -C deploy/kubernetes all"
+       make: |
+         make -C deploy/kubernetes all;
+         make check-no-changes;

ci: improve linting speed #22103

ci: improve linting speed #22103

Conversation

kuisathaverat commented Oct 22, 2020 • edited Loading

What does this PR do?

Why is it important?

Before

After

Checklist

elasticmachine commented Oct 22, 2020

elasticmachine commented Oct 22, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

jsoriano left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuisathaverat Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kuisathaverat commented Oct 27, 2020

elasticmachine commented Oct 28, 2020

💚 Flaky test report

Test stats 🧪

jsoriano commented Oct 28, 2020

kuisathaverat commented Oct 22, 2020 •

edited

Loading

elasticmachine commented Oct 22, 2020 •

edited by jenkins-beats-ci bot

Loading

kuisathaverat Oct 27, 2020 •

edited

Loading