[Metricbeat] Add Linux pressure metricset #27355

b-deam · 2021-08-13T05:29:07Z

What does this PR do?

This PR aims to introduce an additional metricset that collects Pressure Stall Information (PSI) for CPU, Memory, and IO. These are available from kernel >=4.20.

These metrics are incredibly useful for debugging system performance, and more can be read about these metrics below:

Why is it important?

These are valuable OS metrics that are useful for both debugging and monitoring system performance.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Tests need work.

How to test this PR locally

Pull down, run on a Linux host (ubuntu2004 works from Vagrantfile) that:
- Is on kernel >=4.20
- /proc/pressure is available on

Related issues

Closes [Metricbeat][System module] psi: pressure stall information #27223

…tricset

elasticmachine · 2021-08-13T05:44:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-19T03:21:52.887+0000
Duration: 81 min 24 sec
Commit: c4cbb2a

Test stats 🧪

Test	Results
Failed	0
Passed	9086
Skipped	2418
Total	11504

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9086
Skipped	2418
Total	11504

fearful-symmetry

This is looking good! Just a few small things, mostly beats cruft, we need to change. If you want, I can take the PR over from here; I don't want to make you feel like you're obligated to spend your time coding to the various metricbeat idioms.

fearful-symmetry · 2021-08-13T16:42:10Z

metricbeat/module/linux/pressure/_meta/fields.yml

+      type: group
+      description: Linux pressure stall information
+      fields:
+        - name: cpu.some.10


In system/process cgroup data, we're reporting these as 10.pct and so on. We'll also need to add format: percent

Cool. I tried to follow similar conventions for total, in that it becomes total.time.us - does this make sense here?

metricbeat/module/linux/pressure/pressure.go

metricbeat/module/linux/pressure/pressure_test.go

b-deam · 2021-08-16T08:48:00Z

If you want, I can take the PR over from here; I don't want to make you feel like you're obligated to spend your time coding to the various metricbeat idioms.

Not a problem - I think I've addressed the comments so far but feel free to make any small changes if I've otherwise messed things up there.

fearful-symmetry · 2021-08-16T16:06:57Z

Real quick, you'll need to run mage fmt update in the main metricbeat directory.

b-deam · 2021-08-16T23:05:13Z

Real quick, you'll need to run mage fmt update in the main metricbeat directory.

Hmm.. I tried this but I don't see any changes. Any ideas?

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ make update  
mage update
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/build/fields/fields.all.yml
exec: go list -m
exec: go list -m              

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ mage fmt update
>> fmt - go-licenser: Adding missing headers
>> fmt - goimports: Formatting Go code
>> fmt - autopep8: Formatting Python code
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/fields.yml
>> Building metricbeat.yml for linux/amd64
>> Building metricbeat.reference.yml for linux/amd64
>> Building metricbeat.docker.yml for linux/amd64
Generated fields.yml for metricbeat to /Users/bradleydeam/go/src/github.com/b-deam/beats/metricbeat/build/fields/fields.all.yml
exec: go list -m
exec: go list -m

bradleydeam in ~/go/src/github.com/b-deam/beats/metricbeat on branch linux-pressure-metricset $ git status
On branch linux-pressure-metricset
Your branch is up to date with 'origin/linux-pressure-metricset'.

nothing to commit, working tree clean

fearful-symmetry · 2021-08-17T16:27:19Z

@b-deam I think you'll need to re-merge upstream changes? A few things may have changed in the doc build.

…tricset

b-deam · 2021-08-18T01:48:12Z

@b-deam I think you'll need to re-merge upstream changes? A few things may have changed in the doc build.

🤦 - ofc. Done. 🤞 tests are good!

elasticmachine · 2021-08-18T05:25:20Z

Pinging @elastic/integrations (Team:Integrations)

metricbeat/module/linux/pressure/pressure.go

metricbeat/module/linux/pressure/pressure_test.go

fearful-symmetry

LGTM.

…tricset

b-deam · 2021-08-19T03:22:26Z

LGTM.

I re-ran a test to make sure it all looked good and noticed that I'd messed up the fields/mapping so that the metrics were actually recorded as:

linux.pressure.pressure.*

This should be fixed now..

b-deam · 2021-08-19T05:04:10Z

Thanks for all the help @fearful-symmetry!

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/modules_list.asciidoc # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go

* Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/docs/modules/linux.asciidoc # metricbeat/docs/modules/linux/pressure.asciidoc # metricbeat/docs/modules_list.asciidoc # metricbeat/include/list_common.go # metricbeat/metricbeat.reference.yml # metricbeat/module/linux/_meta/config.yml # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go # metricbeat/module/linux/pressure/pressure_test.go # metricbeat/modules.d/linux.yml.disabled # x-pack/metricbeat/metricbeat.reference.yml

) * [Metricbeat] Add Linux pressure metricset (#27355) * Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/fields.asciidoc # metricbeat/docs/modules/linux.asciidoc # metricbeat/docs/modules/linux/pressure.asciidoc # metricbeat/docs/modules_list.asciidoc # metricbeat/include/list_common.go # metricbeat/metricbeat.reference.yml # metricbeat/module/linux/_meta/config.yml # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go # metricbeat/module/linux/pressure/pressure_test.go # metricbeat/modules.d/linux.yml.disabled # x-pack/metricbeat/metricbeat.reference.yml * fix merge Co-authored-by: Brad Deam <54515790+b-deam@users.noreply.github.com> Co-authored-by: Alex Kristiansen <alex.kristiansen@elastic.co>

…9036) * [Metricbeat] Add Linux pressure metricset (#27355) * Adds new linux pressure metricset * Tidy up code * Be consistent in default config * Run integration tests * Make update * make update v3 * Address comments * Format tests * Remove unused imports from tests * Mage format * Update changelog * simplify loop * Fix fields/mapping (cherry picked from commit e49f257) # Conflicts: # metricbeat/docs/modules_list.asciidoc # metricbeat/module/linux/fields.go # metricbeat/module/linux/pressure/pressure.go * Fix conflicts * Formatting Co-authored-by: Brad Deam <54515790+b-deam@users.noreply.github.com> Co-authored-by: Brad Deam <bradley.deam@elastic.co>

b-deam added 5 commits August 12, 2021 17:47

Adds new linux pressure metricset

425ef9b

Tidy up code

c6a1ce0

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

baa4dc9

…tricset

Be consistent in default config

b63496b

Run integration tests

f3f531a

b-deam added enhancement Metricbeat Metricbeat Team:Integrations Label for the Integrations team :integrations labels Aug 13, 2021

b-deam self-assigned this Aug 13, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Aug 13, 2021

b-deam changed the title ~~Linux pressure metricset~~ [Metricbeat] Add Linux pressure metricset Aug 13, 2021

b-deam requested a review from fearful-symmetry August 13, 2021 05:29

b-deam added 2 commits August 13, 2021 16:19

Make update

4da2423

make update v3

4d572d6

fearful-symmetry reviewed Aug 13, 2021

View reviewed changes

fearful-symmetry self-assigned this Aug 13, 2021

b-deam added 3 commits August 16, 2021 17:59

Address comments

a470b1e

Format tests

2c41126

Remove unused imports from tests

d46e1ea

b-deam added 2 commits August 18, 2021 11:14

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

e4c39dd

…tricset

Mage format

ac5f0ee

Update changelog

b496aeb

b-deam marked this pull request as ready for review August 18, 2021 05:25

fearful-symmetry reviewed Aug 18, 2021

View reviewed changes

metricbeat/module/linux/pressure/pressure.go Show resolved Hide resolved

fearful-symmetry reviewed Aug 18, 2021

View reviewed changes

metricbeat/module/linux/pressure/pressure_test.go Outdated Show resolved Hide resolved

fearful-symmetry approved these changes Aug 18, 2021

View reviewed changes

b-deam added 3 commits August 19, 2021 09:18

Merge remote-tracking branch 'upstream/master' into linux-pressure-me…

4afa1f6

…tricset

simplify loop

4451182

Fix fields/mapping

c4cbb2a

b-deam merged commit e49f257 into elastic:master Aug 19, 2021

andresrc added backport-v7.16.0 Automated backport with mergify backport-v8.0.0 Automated backport with mergify labels Nov 18, 2021

mergify bot mentioned this pull request Nov 18, 2021

[7.16](backport #27355) [Metricbeat] Add Linux pressure metricset #29036

Merged

mergify bot mentioned this pull request Nov 18, 2021

[8.0](backport #27355) [Metricbeat] Add Linux pressure metricset #29037

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add Linux pressure metricset #27355

[Metricbeat] Add Linux pressure metricset #27355

b-deam commented Aug 13, 2021 •

edited

Loading

elasticmachine commented Aug 13, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

fearful-symmetry left a comment

fearful-symmetry Aug 13, 2021

b-deam Aug 16, 2021

b-deam commented Aug 16, 2021

fearful-symmetry commented Aug 16, 2021

b-deam commented Aug 16, 2021

fearful-symmetry commented Aug 17, 2021

b-deam commented Aug 18, 2021

elasticmachine commented Aug 18, 2021

fearful-symmetry left a comment

b-deam commented Aug 19, 2021

b-deam commented Aug 19, 2021

[Metricbeat] Add Linux pressure metricset #27355

[Metricbeat] Add Linux pressure metricset #27355

Conversation

b-deam commented Aug 13, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Aug 13, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

fearful-symmetry left a comment

Choose a reason for hiding this comment

fearful-symmetry Aug 13, 2021

Choose a reason for hiding this comment

b-deam Aug 16, 2021

Choose a reason for hiding this comment

b-deam commented Aug 16, 2021

fearful-symmetry commented Aug 16, 2021

b-deam commented Aug 16, 2021

fearful-symmetry commented Aug 17, 2021

b-deam commented Aug 18, 2021

elasticmachine commented Aug 18, 2021

fearful-symmetry left a comment

Choose a reason for hiding this comment

b-deam commented Aug 19, 2021

b-deam commented Aug 19, 2021

b-deam commented Aug 13, 2021 •

edited

Loading

elasticmachine commented Aug 13, 2021 •

edited by jenkins-beats-ci bot

Loading