[libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30534

AndersonQ · 2022-02-22T14:32:32Z

What does this PR do?

Improve logging on filebeat to show better what happens when it's parsing the configs.

Why is it important?

Stop omitting errors and will help to investigate #30533

Checklist

My code follows the style guidelines of this project
~~[ ] I have commented my code, particularly in hard-to-understand areas~~
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates Filebeat running under Elastic-Agent not harvesting logs after restart #30533

mergify · 2022-02-22T14:33:01Z

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-02-22T14:37:02Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

botelastic · 2022-02-22T14:37:22Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2022-02-22T14:42:24Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-02T16:09:07.464+0000
Duration: 133 min 46 sec

Test stats 🧪

Test	Results
Failed	0
Passed	42995
Skipped	3846
Total	46841

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

mergify · 2022-02-22T15:25:46Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-improve-logs upstream/filebeat-improve-logs
git merge upstream/main
git push upstream filebeat-improve-logs

ph

Added a small suggestion around the wording of the log message for the harvester, other than that LGTM.

ph · 2022-02-22T18:43:46Z

filebeat/beater/crawler.go

@@ -105,7 +105,7 @@ func (c *crawler) Start(
 		}()
 	}

-	log.Infof("Loading and starting Inputs completed. Enabled inputs: %v", len(c.inputs))
+	log.Infof("Loading and starting Inputs completed. Enabled inputs: %d", len(c.inputs))



@AndersonQ The len() is a good start but let's be more precise like having a detailed count for each input. IE. Filestream input: 2, same comment for this log statement.

I'm logging the input config keys now on startInput now.
The inputs field doesn't have much information to log, it's an implementation of Runner :/

filebeat/input/log/harvester.go

ph · 2022-02-22T18:54:25Z

x-pack/libbeat/management/manager.go

@@ -256,8 +257,8 @@ func (cm *Manager) apply(blocks ConfigBlocks) error {
 	}

 	// Unset missing configs
-	for name := range missing {
-		if missing[name] {
+	for name, isMissing := range missing {


+1 nice change.

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

AndersonQ · 2022-02-24T15:46:14Z

/test

AndersonQ · 2022-03-01T10:47:00Z

/test

AndersonQ · 2022-03-02T12:30:29Z

/test

ph

Changes LGTM.

This PRs improves logging for the management and filebeat input.
We should do something similar for to others beats too. But's let's get this merged.

ph · 2022-03-02T14:11:56Z

filebeat/beater/crawler.go

+	// TODO: Either use debug or remove it after https://github.com/elastic/beats/pull/30534
+	// is fixed.
+	c.log.Infof("starting input, keys present on the config: %v",
+		config.FlattenedKeys())


++ for not leaking senstitive information.

mergify · 2022-03-02T15:20:58Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b filebeat-improve-logs upstream/filebeat-improve-logs
git merge upstream/main
git push upstream filebeat-improve-logs

…30534) - log error when parsing config block and disabled input on filebeat - adjust tests and add an optional error message for 'wait_until' on libbeat python tests (cherry picked from commit b25fdf6) # Conflicts: # filebeat/beater/crawler.go # libbeat/tests/system/beat/beat.py

… input * log error when parsing config block and disabled input on filebeat (#30534) - log error when parsing config block and disabled input on filebeat - adjust tests and add an optional error message for 'wait_until' on libbeat python tests (cherry picked from commit b25fdf6) Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co>

…d input * log error when parsing config block and disabled input on filebeat (#30534) - log error when parsing config block and disabled input on filebeat - adjust tests and add an optional error message for 'wait_until' on libbeat python tests (cherry picked from commit b25fdf6) Co-authored-by: Anderson Queiroz <anderson.queiroz@elastic.co>

…lastic#30534) - log error when parsing config block and disabled input on filebeat - adjust tests and add an optional error message for 'wait_until' on libbeat python tests (cherry picked from commit b25fdf6) # Conflicts: # filebeat/beater/crawler.go # libbeat/tests/system/beat/beat.py

log error and disabled input on filebeat

3e9f4df

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Feb 22, 2022

mergify bot assigned AndersonQ Feb 22, 2022

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 22, 2022

changelog

61cb76c

AndersonQ requested a review from ph February 22, 2022 14:36

AndersonQ added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Feb 22, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 22, 2022

AndersonQ added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify needs_team Indicates that the issue/PR needs a Team:* label labels Feb 22, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Feb 22, 2022

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Feb 22, 2022

AndersonQ mentioned this pull request Feb 22, 2022

Filebeat running under Elastic-Agent not harvesting logs after restart #30533

Closed

AndersonQ changed the title ~~log error and disabled input on filebeat~~ [libbeat | filebeat] Log error when parsing config block and disabled input on filebeat Feb 22, 2022

ph approved these changes Feb 22, 2022

View reviewed changes

AndersonQ and others added 4 commits February 23, 2022 10:07

Update filebeat/input/log/harvester.go

272303c

Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>

pr changes

1876918

more logs

6c92a39

.

9a408e2

ph approved these changes Feb 23, 2022

View reviewed changes

Merge branch 'main' into filebeat-improve-logs

904801b

AndersonQ added 4 commits March 1, 2022 14:35

a test

cd9a4f8

.

3c88ab6

fix test

3fdfa7c

add an optional error message for 'wait_until'

0b369bc

ph approved these changes Mar 2, 2022

View reviewed changes

Merge branch 'main' into filebeat-improve-logs

8b2aaec

AndersonQ merged commit b25fdf6 into main Mar 2, 2022

AndersonQ deleted the filebeat-improve-logs branch March 2, 2022 18:33

mergify bot mentioned this pull request Mar 2, 2022

[7.17](backport #30534) [libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30653

Merged

mergify bot mentioned this pull request Mar 2, 2022

[8.0](backport #30534) [libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30654

Merged

mergify bot mentioned this pull request Mar 2, 2022

[8.1](backport #30534) [libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30655

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30534

[libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30534

AndersonQ commented Feb 22, 2022 •

edited

Loading

mergify bot commented Feb 22, 2022

elasticmachine commented Feb 22, 2022

botelastic bot commented Feb 22, 2022

elasticmachine commented Feb 22, 2022 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

mergify bot commented Feb 22, 2022

ph left a comment

ph Feb 22, 2022

AndersonQ Feb 23, 2022

ph Feb 22, 2022

AndersonQ commented Feb 24, 2022

AndersonQ commented Mar 1, 2022

AndersonQ commented Mar 2, 2022

ph left a comment

ph Mar 2, 2022

mergify bot commented Mar 2, 2022

[libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30534

[libbeat | filebeat] Log error when parsing config block and disabled input on filebeat #30534

Conversation

AndersonQ commented Feb 22, 2022 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

mergify bot commented Feb 22, 2022

elasticmachine commented Feb 22, 2022

botelastic bot commented Feb 22, 2022

elasticmachine commented Feb 22, 2022 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

mergify bot commented Feb 22, 2022

ph left a comment

Choose a reason for hiding this comment

ph Feb 22, 2022

Choose a reason for hiding this comment

AndersonQ Feb 23, 2022

Choose a reason for hiding this comment

ph Feb 22, 2022

Choose a reason for hiding this comment

AndersonQ commented Feb 24, 2022

AndersonQ commented Mar 1, 2022

AndersonQ commented Mar 2, 2022

ph left a comment

Choose a reason for hiding this comment

ph Mar 2, 2022

Choose a reason for hiding this comment

mergify bot commented Mar 2, 2022

AndersonQ commented Feb 22, 2022 •

edited

Loading

elasticmachine commented Feb 22, 2022 •

edited by jenkins-beats-ci bot

Loading