Filesystem refactor #31001

fearful-symmetry · 2022-03-24T21:15:56Z

What does this PR do?

This is a refactor of the system/filesystem and system/fsstat metricsets for the purpose of fixing #30590 Also also continuing to depricate the gosigar library.

I'm still in the process of testing this, but I'm currently juggling 3 other things so I at least need to get the PR in so I don't loose track of what I'm doing.

Why is it important?

This fixes a bug, removes code from metricbeat, and removes deprecated gosigar code.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Fixes #30590

elasticmachine · 2022-03-24T21:15:59Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

mergify · 2022-03-24T21:16:06Z

This pull request does not have a backport label. Could you fix it @fearful-symmetry? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-03-24T21:37:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-05-23T19:08:12.996+0000
Duration: 54 min 4 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3581
Skipped	885
Total	4466

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

libbeat/metric/system/filesystem/filesystem.go

metricbeat/module/system/filesystem/filesystem_test.go

mergify · 2022-04-27T14:38:13Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filesystem-refactor upstream/filesystem-refactor
git merge upstream/main
git push upstream filesystem-refactor

mergify · 2022-04-28T12:54:44Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filesystem-refactor upstream/filesystem-refactor
git merge upstream/main
git push upstream filesystem-refactor

jlind23 · 2022-05-02T06:59:21Z

/test

mergify · 2022-05-02T11:02:44Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filesystem-refactor upstream/filesystem-refactor
git merge upstream/main
git push upstream filesystem-refactor

kvch · 2022-05-20T07:27:01Z

libbeat/opt/opt.go

@@ -33,15 +33,15 @@ type Int struct {
 	value  int
 }


Do you mind opening a PR to elastic-agent-libs to update the opt package there as well?

Oh, it's already in there. Huh.

cmacknz · 2022-05-20T15:12:00Z

Looking at this again it's looked like it has changed a fair bit.

This fixes a bug, removes code from metricbeat, and removes deprecated gosigar code.

What is the bug that was fixed here? It's hard to spot in all the code? Also you need a CHANGELOG entry for it.

fearful-symmetry · 2022-05-20T17:18:46Z

@cmacknz yup, hence why I asked for another review in slack. The original code wasn't respecting hostfs, which was the bug. Changelog incoming...

fearful-symmetry · 2022-05-20T18:15:34Z

Alright, changlog added, reverted some of the opt changes.

mergify · 2022-05-20T19:59:41Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filesystem-refactor upstream/filesystem-refactor
git merge upstream/main
git push upstream filesystem-refactor

cmacknz · 2022-05-20T20:09:00Z

metricbeat/module/system/filesystem/_meta/fields.yml

+    - name: options
+      type: keyword
+      description: >
+        The options present on the filesystem mount.


Should this option be mentioned in the CHANGELOG?

cmacknz

LGTM again, I can't spot anything problematic.

mergify · 2022-05-23T10:45:33Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filesystem-refactor upstream/filesystem-refactor
git merge upstream/main
git push upstream filesystem-refactor

toby-sutor · 2022-07-07T10:54:23Z

It seems like this made it into version 8.3.1 as per https://github.com/elastic/beats/blob/v8.3.1/CHANGELOG.asciidoc#L34.

* first pass at filesystem refactoring * cleanup code * final cleanup * cleanup, docs * license headers * make linter happy * try to hack around linter platform issues * fix function sig * minor fixes, formatting * try to make linter happy * more linter fixes * hopefully the last linter cleanup * try to fix unit tests * fix tests * refactor after merges from elastic-agent libs * make linter happy * try to update deps * okay, try that * fix merge * add changelog, revert opt changes * fix broken merge * mage update (cherry picked from commit 1d4a7ca) # Conflicts: # go.sum # metricbeat/module/system/fields.go # metricbeat/module/system/filesystem/_meta/fields.yml # metricbeat/module/system/filesystem/filesystem.go # metricbeat/module/system/filesystem/filesystem_test.go # metricbeat/module/system/filesystem/helper.go # metricbeat/module/system/fsstat/fsstat.go # metricbeat/module/system/test_system.py

* Filesystem refactor (#31001) * first pass at filesystem refactoring * cleanup code * final cleanup * cleanup, docs * license headers * make linter happy * try to hack around linter platform issues * fix function sig * minor fixes, formatting * try to make linter happy * more linter fixes * hopefully the last linter cleanup * try to fix unit tests * fix tests * refactor after merges from elastic-agent libs * make linter happy * try to update deps * okay, try that * fix merge * add changelog, revert opt changes * fix broken merge * mage update (cherry picked from commit 1d4a7ca) # Conflicts: # go.sum # metricbeat/module/system/fields.go # metricbeat/module/system/filesystem/_meta/fields.yml # metricbeat/module/system/filesystem/filesystem.go # metricbeat/module/system/filesystem/filesystem_test.go # metricbeat/module/system/filesystem/helper.go # metricbeat/module/system/fsstat/fsstat.go # metricbeat/module/system/test_system.py * attempt to clean up backport * update notice * try to make linter happy * still fixing up backport * still un-breaking merge * fix changelog Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: Alex Kristiansen <alex.kristiansen@elastic.co>

* first pass at filesystem refactoring * cleanup code * final cleanup * cleanup, docs * license headers * make linter happy * try to hack around linter platform issues * fix function sig * minor fixes, formatting * try to make linter happy * more linter fixes * hopefully the last linter cleanup * try to fix unit tests * fix tests * refactor after merges from elastic-agent libs * make linter happy * try to update deps * okay, try that * fix merge * add changelog, revert opt changes * fix broken merge * mage update

fearful-symmetry added 4 commits March 21, 2022 15:08

first pass at filesystem refactoring

eb68207

cleanup code

9451489

final cleanup

ab92ff4

cleanup, docs

fbf111a

fearful-symmetry added bug enhancement Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team labels Mar 24, 2022

fearful-symmetry self-assigned this Mar 24, 2022

fearful-symmetry requested a review from a team as a code owner March 24, 2022 21:15

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 24, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Mar 24, 2022

fearful-symmetry added 3 commits March 24, 2022 16:48

license headers

bbeb709

make linter happy

8a63bd6

try to hack around linter platform issues

a8beb8d

cmacknz reviewed Mar 28, 2022

View reviewed changes

libbeat/metric/system/filesystem/filesystem.go Outdated Show resolved Hide resolved

cmacknz reviewed Mar 28, 2022

View reviewed changes

libbeat/metric/system/filesystem/filesystem.go Outdated Show resolved Hide resolved

cmacknz reviewed Mar 28, 2022

View reviewed changes

metricbeat/module/system/filesystem/filesystem_test.go Outdated Show resolved Hide resolved

fearful-symmetry added 9 commits March 28, 2022 12:15

fix function sig

bf3d88c

minor fixes, formatting

c300931

try to make linter happy

2e6cd47

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

a64be0f

more linter fixes

412bc71

hopefully the last linter cleanup

0a6dc6a

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

ba99b6c

try to fix unit tests

d6f9ace

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

845a55b

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

4c947ea

fearful-symmetry added 2 commits April 29, 2022 11:09

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

1577fbb

fix merge

e7c00be

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

d84f297

kvch reviewed May 20, 2022

View reviewed changes

cmacknz self-requested a review May 20, 2022 15:11

fearful-symmetry added 2 commits May 20, 2022 10:53

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

db9b5fa

add changelog, revert opt changes

b6c37b5

cmacknz reviewed May 20, 2022

View reviewed changes

cmacknz approved these changes May 20, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

bcc33fd

fearful-symmetry added 3 commits May 23, 2022 11:15

Merge remote-tracking branch 'upstream/main' into filesystem-refactor

0b2e039

fix broken merge

6891ca9

mage update

4b7bac2

fearful-symmetry merged commit 1d4a7ca into elastic:main May 23, 2022

cmacknz added the backport-7.17 Automated backport to the 7.17 branch with mergify label Sep 8, 2022

mergify bot mentioned this pull request Sep 8, 2022

[7.17](backport #31001) Filesystem refactor #33022

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Filesystem refactor #31001

Filesystem refactor #31001

fearful-symmetry commented Mar 24, 2022 •

edited by jlind23

Loading

elasticmachine commented Mar 24, 2022

mergify bot commented Mar 24, 2022

elasticmachine commented Mar 24, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Apr 27, 2022

mergify bot commented Apr 28, 2022

jlind23 commented May 2, 2022

mergify bot commented May 2, 2022

kvch May 20, 2022

fearful-symmetry May 20, 2022

cmacknz commented May 20, 2022

fearful-symmetry commented May 20, 2022

fearful-symmetry commented May 20, 2022

mergify bot commented May 20, 2022

cmacknz May 20, 2022

cmacknz left a comment

mergify bot commented May 23, 2022

toby-sutor commented Jul 7, 2022

Filesystem refactor #31001

Filesystem refactor #31001

Conversation

fearful-symmetry commented Mar 24, 2022 • edited by jlind23 Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Mar 24, 2022

mergify bot commented Mar 24, 2022

elasticmachine commented Mar 24, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Apr 27, 2022

mergify bot commented Apr 28, 2022

jlind23 commented May 2, 2022

mergify bot commented May 2, 2022

kvch May 20, 2022

Choose a reason for hiding this comment

fearful-symmetry May 20, 2022

Choose a reason for hiding this comment

cmacknz commented May 20, 2022

fearful-symmetry commented May 20, 2022

fearful-symmetry commented May 20, 2022

mergify bot commented May 20, 2022

cmacknz May 20, 2022

Choose a reason for hiding this comment

cmacknz left a comment

Choose a reason for hiding this comment

mergify bot commented May 23, 2022

toby-sutor commented Jul 7, 2022

fearful-symmetry commented Mar 24, 2022 •

edited by jlind23

Loading

elasticmachine commented Mar 24, 2022 •

edited by jenkins-beats-ci bot

Loading