Add start of elastic-agent diagnostics command #28265

michel-laterman · 2021-10-06T01:29:31Z

What does this PR do?

This PR starts the elastic-agent diagnostics command.
The beats info ("/") HTTP endpoint has been changed to add more data about the running beat including git commit and ephemeral ID.
Currently the diagnostics command will gather beats metadata information from the endpoint and display them along with agent version information.

Why is it important?

The diagnostics command is meant to be used to gather diagnostics information from a running elastic-agent (and beats) that can be used for debugging.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation PR here
[] ~~I have made corresponding change to the default configuration files~~
[] ~~I have added tests that prove my fix is effective or that my feature works~~
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run the agent normally then use: elastic-agent diagnostics

Related issues

Relates Beats diagnostics tooling #21965

Logs

bash-4.2$ elastic-agent diagnostics
{BeatMeta:[{Beat:filebeat Name:filebeat_monitoring Hostname:docker-fleet-agent ID:f5e55217-16fc-4b34-be14-2e60096807b5 EphemeralID:08b22db1-4716-42ce-b1e6-8bcdafcc6776 Version:8.0.0 BuildCommit:ba9eebf2f62f3a827008dbfaf0f814464b0a63f9 BuildTime:2021-09-15 21:13:30 +0000 UTC Username:elastic-agent UserID:1000 UserGID:1000 BinaryArchitecture:amd64 RouteKey:default ElasticLicensed:true Error:} {Beat:metricbeat Name:metricbeat_monitoring Hostname:docker-fleet-agent ID:46a16e22-533d-461d-9916-6dba8fb9ed05 EphemeralID:d090f30f-7662-4abb-9b89-59e95b7b3cd8 Version:8.0.0 BuildCommit:ba9eebf2f62f3a827008dbfaf0f814464b0a63f9 BuildTime:2021-09-15 21:17:17 +0000 UTC Username:elastic-agent UserID:1000 UserGID:1000 BinaryArchitecture:amd64 RouteKey:default ElasticLicensed:true Error:} {Beat:filebeat Name:filebeat Hostname:docker-fleet-agent ID:f5e55217-16fc-4b34-be14-2e60096807b5 EphemeralID:08b22db1-4716-42ce-b1e6-8bcdafcc6776 Version:8.0.0 BuildCommit:ba9eebf2f62f3a827008dbfaf0f814464b0a63f9 BuildTime:2021-09-15 21:13:30 +0000 UTC Username:elastic-agent UserID:1000 UserGID:1000 BinaryArchitecture:amd64 RouteKey:default ElasticLicensed:true Error:} {Beat:metricbeat Name:metricbeat Hostname:docker-fleet-agent ID:46a16e22-533d-461d-9916-6dba8fb9ed05 EphemeralID:d090f30f-7662-4abb-9b89-59e95b7b3cd8 Version:8.0.0 BuildCommit:ba9eebf2f62f3a827008dbfaf0f814464b0a63f9 BuildTime:2021-09-15 21:17:17 +0000 UTC Username:elastic-agent UserID:1000 UserGID:1000 BinaryArchitecture:amd64 RouteKey:default ElasticLicensed:true Error:}] AgentVersion:{Version:8.0.0 Commit:473cb596588bc7da634e21e6615a0a6c90180be8 BuildTime:2021-10-06 15:31:20 +0000 UTC Snapshot:true}}

Add initial command that will gather meta-data from running beats HTTP endpoints and elastic-agent version information.

mergify · 2021-10-06T01:29:57Z

This pull request does not have a backport label. Could you fix it @michel-laterman? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-10-06T01:57:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-14T21:38:22.115+0000
Duration: 155 min 1 sec
Commit: a72a7fb

Test stats 🧪

Test	Results
Failed	0
Passed	53802
Skipped	5346
Total	59148

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

michalpristas · 2021-10-06T12:44:48Z

could you diff a change in output in desciption? it would make easier for people having strict requirements on format using this endpoint finding out what should be added

michalpristas

looks good, i havent tested it so far. i will today/tomorrow

libbeat/cmd/instance/beat.go

michalpristas · 2021-10-06T12:49:42Z

x-pack/elastic-agent/pkg/agent/application/pipeline/stream/operator_stream.go

@@ -20,6 +21,10 @@ type stater interface {
 	State() map[string]state.State
 }

+type specer interface {


probably you meant specser. i dont know if it is a real world but it sounds funny :-)
no more naming comments from me

I think specser is a bit too strange, it does more to hide what the interface is doing than specer

michel-laterman · 2021-10-06T17:27:53Z

/test

ruflin

What is the format of the output? Can we make it human readable or at least add an option for it?

ruflin · 2021-10-12T11:06:05Z

x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

+
+	diag, err := getDiagnostics(innerCtx)
+	if err == context.DeadlineExceeded {
+		return errors.New("timed out after 30 seconds trying to connect to Elastic Agent daemon")


30s sounds like a long time for a timeout for a local diagnostic commands. I would expect the diagnostic command to be pretty quick or is there some "remote" dependency?

copied timeout from the status command

ruflin · 2021-10-12T11:07:01Z

x-pack/elastic-agent/pkg/agent/control/client/client.go

@@ -53,6 +53,25 @@ type ApplicationStatus struct {
 	Payload map[string]interface{}
 }

+// BeatMeta is the running version and ID inforation for a running beat.
+type BeatMeta struct {


What about endpoint and other processes we run? If possible we should keep the naming not specific to Beats only.

good point, Do we know the kind of meta-data endpoint or another process would return? Or should I change this to just be a generic map[string]interface{} to capture everything?

elasticmachine · 2021-10-14T16:14:55Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

blakerouse

Overall looks really good. Just got one question.

blakerouse · 2021-10-14T17:11:59Z

x-pack/elastic-agent/pkg/core/socket/dial.go

+// DialContext returns a function that can be used to dial a local unix-domain socket.
+func DialContext(socket string) func(context.Context, string, string) (net.Conn, error) {
+	return func(_ context.Context, _, _ string) (net.Conn, error) {
+		return net.Dial("unix", socket)


Is it not possible to get net.Dial to take a context? Is there no net.DialContext?

This PR starts the elastic-agent diagnostics command. The beats info ("/") HTTP endpoint has been changed to add more data about the running beat including git commit and ephemeral ID. Currently the diagnostics command will gather beats metadata information from the endpoint and display them along with agent version information. (cherry picked from commit 887e40a)

This PR starts the elastic-agent diagnostics command. The beats info ("/") HTTP endpoint has been changed to add more data about the running beat including git commit and ephemeral ID. Currently the diagnostics command will gather beats metadata information from the endpoint and display them along with agent version information. (cherry picked from commit 887e40a) Co-authored-by: Michel Laterman <82832767+michel-laterman@users.noreply.github.com>

This PR starts the elastic-agent diagnostics command. The beats info ("/") HTTP endpoint has been changed to add more data about the running beat including git commit and ephemeral ID. Currently the diagnostics command will gather beats metadata information from the endpoint and display them along with agent version information.

This PR starts the elastic-agent diagnostics command. The beats info ("/") HTTP endpoint has been changed to add more data about the running beat including git commit and ephemeral ID. Currently the diagnostics command will gather beats metadata information from the endpoint and display them along with agent version information. (cherry picked from commit 887e40a) # Conflicts: # libbeat/cmd/instance/beat.go # x-pack/elastic-agent/pkg/agent/cmd/diagnostics.go

michel-laterman added 3 commits October 5, 2021 13:23

Add extra metadata to beat info registry

0d508f1

Add new BeatMeta grpc command

8ab225f

Add initial diagnostics command

981a540

Add initial command that will gather meta-data from running beats HTTP endpoints and elastic-agent version information.

michel-laterman added the enhancement label Oct 6, 2021

michel-laterman requested review from ruflin, michalpristas and blakerouse October 6, 2021 01:29

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 6, 2021

michel-laterman added the Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team label Oct 6, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 6, 2021

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Oct 6, 2021

michalpristas approved these changes Oct 6, 2021

View reviewed changes

Ran make update

473cb59

michel-laterman added backport-v7.16.0 Automated backport with mergify and removed backport-skip Skip notification from the automated backport with mergify labels Oct 6, 2021

michel-laterman added 2 commits October 6, 2021 11:17

Rename libbeat version variable add CHANGElOG

58d3caf

Merge remote-tracking branch 'origin/master' into diagnostics

e1fa500

ruflin reviewed Oct 12, 2021

View reviewed changes

michel-laterman added 6 commits October 12, 2021 12:35

Rename BeatMeta to ProcMeta

78b2f78

Merge remote-tracking branch 'origin/master' into diagnostics

0114819

Add output options for diagnostics

040c9ed

Fix human output

c0d1dae

Fix issues with linting

2e6b199

Update for go 1.17.1

9421335

michel-laterman mentioned this pull request Oct 13, 2021

Add elastic-agent diagnostics documentation elastic/observability-docs#1164

Merged

michel-laterman marked this pull request as ready for review October 14, 2021 16:14

blakerouse approved these changes Oct 14, 2021

View reviewed changes

michel-laterman added 2 commits October 14, 2021 13:47

Change unix dialer to use contexts, add unittest

e3c4a28

Add license header

a72a7fb

michel-laterman merged commit 887e40a into elastic:master Oct 15, 2021

michel-laterman deleted the diagnostics branch October 15, 2021 02:26

mergify bot mentioned this pull request Oct 15, 2021

[7.x](backport #28265) Add start of elastic-agent diagnostics command #28460

Merged

barkbay mentioned this pull request Oct 25, 2021

[libbeat] APM Server does not start when running as an arbitrary user ID #28617

Closed

david-kow mentioned this pull request Oct 28, 2021

Fall back to os.Get[ug]id if user.Current() fails #28696

Merged

6 tasks

michel-laterman added the backport-v8.0.0 Automated backport with mergify label Nov 10, 2021

mergify bot mentioned this pull request Nov 10, 2021

[7.16](backport #28265) Add start of elastic-agent diagnostics command #28916

Closed

mergify bot mentioned this pull request Nov 10, 2021

[8.0](backport #28265) Add start of elastic-agent diagnostics command #28917

Closed

P1llus mentioned this pull request Nov 15, 2021

Libbeat diagnostic tooling #28977

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add start of elastic-agent diagnostics command #28265

Add start of elastic-agent diagnostics command #28265

michel-laterman commented Oct 6, 2021 •

edited

Loading

mergify bot commented Oct 6, 2021

elasticmachine commented Oct 6, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

michalpristas commented Oct 6, 2021

michalpristas left a comment

michalpristas Oct 6, 2021

michel-laterman Oct 12, 2021

michel-laterman commented Oct 6, 2021

ruflin left a comment

ruflin Oct 12, 2021

michel-laterman Oct 12, 2021

ruflin Oct 12, 2021

michel-laterman Oct 12, 2021

elasticmachine commented Oct 14, 2021

blakerouse left a comment

blakerouse Oct 14, 2021

michel-laterman Oct 14, 2021

Add start of elastic-agent diagnostics command #28265

Add start of elastic-agent diagnostics command #28265

Conversation

michel-laterman commented Oct 6, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

Logs

mergify bot commented Oct 6, 2021

elasticmachine commented Oct 6, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

michalpristas commented Oct 6, 2021

michalpristas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michel-laterman commented Oct 6, 2021

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Oct 14, 2021

blakerouse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michel-laterman commented Oct 6, 2021 •

edited

Loading

elasticmachine commented Oct 6, 2021 •

edited by jenkins-beats-ci bot

Loading