Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Enables APM collection #81731

Merged
merged 16 commits into from
Dec 3, 2020
Merged

[CI] Enables APM collection #81731

merged 16 commits into from
Dec 3, 2020

Conversation

tylersmalley
Copy link
Contributor

@tylersmalley tylersmalley commented Oct 27, 2020 •
edited
Loading

Enables APM collection for functional tests on tracked branches, with a 1% sample rate on pull requests.

Stats are accessible at: https://ela.st/kibana-ops-ci-stats

[CI] Enables APM collection
5c3601e 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Contributor Author

Early checks appear to increase CI functional test times by 10% This doesn't seem enough to justify preventing us from proceeding, but it's worth investigating to understand if there is anything causing large increases in some tests that could be improved.

job master 81731 diff
kibana-ciGroup1 0:05:44 0:06:11 7.28%
kibana-ciGroup2 0:17:00 0:18:00 5.56%
kibana-ciGroup3 0:24:00 0:24:00 0.00%
kibana-ciGroup4 0:22:00 0:22:00 0.00%
kibana-ciGroup5 0:13:00 0:14:00 7.14%
kibana-ciGroup6
kibana-ciGroup7
kibana-ciGroup8 0:15:00 0:15:00 0.00%
kibana-ciGroup9
kibana-ciGroup10 0:15:00 0:16:00 6.25%
kibana-ciGroup11 0:24:00 0:25:00 4.00%
kibana-ciGroup12 0:29:00 0:30:00 3.33%
xpack-kibana-ciGroup1 0:53:00 1:02:00 14.52%
xpack-kibana-ciGroup2 0:40:00 0:44:00 9.09%
xpack-kibana-ciGroup3 0:51:00 0:59:00 13.56%
xpack-kibana-ciGroup4 0:51:00 0:55:00 7.27%
xpack-kibana-ciGroup5 0:48:00 0:50:00 4.00%
xpack-kibana-ciGroup6
xpack-kibana-ciGroup7
xpack-kibana-ciGroup8 1:00:00 1:25:00 29.41%
xpack-kibana-ciGroup9
xpack-kibana-ciGroup10 0:41:00 0:43:00 4.65%
8:28:44 9:28:11 10.46%

@tylersmalley
Copy link
Contributor Author

@IgorGuz2000 any luck with these failures?

@tylersmalley tylersmalley mentioned this pull request Nov 16, 2020
Closed
Skip Security Endpoint test
3288738 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@elastic elastic deleted a comment from apmmachine Nov 19, 2020
Tyler Smalley added 5 commits November 23, 2020 16:26
Updates token
c7ee220 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Enable APM for FTR only
d93b0a7 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Merge remote-tracking branch 'upstream/master' into enable-apm-ci
8bc8914
Set sample rate
f380562 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Decrease sample rate to 1%
963addf 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@mshustov
Copy link
Contributor

mshustov commented Dec 1, 2020 •
edited
Loading

@tylersmalley you can adjust API config to reduce performance overhead. see #78792 (comment)

# optional metrics to adjust performance 
# see https://www.elastic.co/guide/en/apm/agent/nodejs/master/configuration.html
elastic.apm.centralConfig: false
elastic.apm.breakdownMetrics: false
elastic.apm.transactionSampleRate: 0.1
elastic.apm.metricsInterval: '120s'

Tyler Smalley added 2 commits December 1, 2020 07:53
Increase metricsInterval
3ac2e10 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Increase metricsInterval for distributable
1aafb07 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Contributor Author

@restrry, thanks. Since the functional tests run on the distribution, we were just missing the metricsInterval here: https://github.com/elastic/kibana/blob/master/packages/kbn-apm-config-loader/src/config.ts#L30-L49

For the sampleRate - I was taking 100% for the tracked branches (master/7.x/7.10), and 1% on PR's. Do you think we should reduce that rate even on the tracked branches?

@tylersmalley tylersmalley marked this pull request as ready for review December 1, 2020 23:13
@tylersmalley tylersmalley requested review from vigneshshanmugam and a team as code owners December 1, 2020 23:13
@tylersmalley tylersmalley added release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.11.0 v8.0.0 labels Dec 1, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-operations (Team:Operations)

@tylersmalley
Copy link
Contributor Author

@elasticmachine merge upstream

vigneshshanmugam
vigneshshanmugam approved these changes Dec 2, 2020
View reviewed changes
Copy link
Member

@vigneshshanmugam vigneshshanmugam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@mshustov
Copy link
Contributor

mshustov commented Dec 2, 2020

For the sampleRate - I was taking 100% for the tracked branches (master/7.x/7.10), and 1% on PR's. Do you think we should reduce that rate even on the tracked branches?

I'd expect it's all or nothing for PRs.
With master/7.x/7.10: it depends on how often we are going to gather those. If we collect metrics on every build (>20 per day), maybe 100% is overhead?

spalger
spalger approved these changes Dec 2, 2020
View reviewed changes
Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Disable for PRs and set to 10% sample rate
9331fcf 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley
Copy link
Contributor Author

@restrry, I have disabled entirely for PR's and set the sample rate to 10% for when it does run on the tracked branches per your suggestion. If we find this is insufficient we can increase it.

Tyler Smalley added 2 commits December 2, 2020 11:34
Allow disabling by env and pass sample rate to frontend
dc4bb17 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Adds prId label and removes jobName
626fdd5 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
@tylersmalley tylersmalley merged commit 74ef540 into elastic:master Dec 3, 2020
@tylersmalley tylersmalley mentioned this pull request Dec 3, 2020
Merged
tylersmalley pushed a commit to tylersmalley/kibana that referenced this pull request Dec 3, 2020
[CI] Enables APM collection (elastic#81731)
f9b6492 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
tylersmalley pushed a commit that referenced this pull request Dec 3, 2020
[CI] Enables APM collection (#81731) (#84929)
9a8351c 
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
gmmorris added a commit to gmmorris/kibana that referenced this pull request Dec 4, 2020
@gmmorris
Merge branch 'master' into alerts/custom-recovery-action-group
8940592 
* master: (28 commits)
  [Actions] fixes bug where severity is auto selected but not applied to the action in PagerDuty (elastic#84891)
  Only attempt to rollover signals index if version < builtin version (elastic#84982)
  skip flaky suite (elastic#84978)
  skip lens rollup tests
  Add geo containment tracking alert type (elastic#84151)
  Changed rollup tests to use test user rather than elastic super user. (elastic#79567)
  skip 'should allow creation of lens xy chart' elastic#84957
  [APM] Add log_level/sanitize_field_names config options to Python Agent (elastic#84810)
  [Maps] geo line source (elastic#76572)
  [data.search] Move search method inside session service and add tests (elastic#84715)
  skip lens drag and drop test.  elastic#84941
  [Ingest Node Pipelines] Integrate painless autocomplete (elastic#84554)
  [Lens] allow drag and drop reorder on xyChart for y dimension (elastic#84640)
  [Lens] Fix error when selecting the current field again (elastic#84817)
  [Metrics UI] Add metadata tab to node details flyout (elastic#84454)
  [CI] Enables APM collection (elastic#81731)
  [Workplace Search] Migrate Sources Schema tree (elastic#84847)
  Disable checking for conflicts when copying saved objects (elastic#83575)
  [SECURITY_SOLUTION] delete advanced Policy fields when they are empty (elastic#84368)
  y18n 4.0.0 -> 4.0.1 (elastic#84905)
  ...
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@peterschretlen peterschretlen mentioned this pull request Dec 22, 2020
Closed
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigneshshanmugam vigneshshanmugam vigneshshanmugam approved these changes

@jbudz jbudz jbudz approved these changes

@spalger spalger spalger approved these changes

Assignees
No one assigned
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Operations Team label for Operations Team v7.11.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@tylersmalley @mshustov @elasticmachine @kibanamachine @spalger @jbudz @vigneshshanmugam