[Fleet] Add agent incoming data endpoint and presentational component #127177

criamico · 2022-03-08T17:27:33Z

Summary

Adding an endpoint agent_status/data that queries logs-*-*,metrics-*-*,traces-*-*,synthetics-*-* to verify that a set of agents is receiving data since the last 5 minutes.
Querying the new endpoint from a react component to signal that a newly enrolled agent has received data.

Received data (two possible buttons are shown)

Loading:

Testing steps

Make sure you have an agent running and ingesting data
Send a request like this

  curl --location --request GET 'localhost:5601/<YOUR_PATH>/api/fleet/agent_status/data?agentsId=[<ANEXISTINGAGENTID>, <ANOTHEREXISTINGAGENTID>]' -u elastic:changeme

You should get a response similar to this one:

    {"items": [
        {
        "bb59252d-ad7f-4289-9dbe-6808582638f6":
            {
               "data": true
             }
        },
        {
        "9a6edd86-ab0e-4683-8706-2a2a0e36b977":
            {
              "data": false
            }
        }
     ]
   }

To test the component I added it here where agentsIds is an array of strings (the agents ids to check).

Test the case when the button shows "Install APM agent":

<ConfirmIncomingData agentsIds={agentsCheckData} installedPolicy={{ name: 'apm', version: '1.2.0' }} />

Test the case when the button shows "View Assets":

<ConfirmIncomingData agentsIds={agentsCheckData} installedPolicy={{ name: 'nginx', version: '1.2.0' }} />

Test the case when no button is shown (similar to current behavior in "add agent"):
<ConfirmIncomingData agentsIds={agentsCheckData} />

Example of the ES query used:

POST /logs-*-*,metrics-*-*,traces-*-*,synthetics-*-*/_search?allow_partial_search_results=true&_source=false&timeout=5s&size=0
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "agent.id": [
              "fa892bb4-b7da-4022-8aaa-4126ba9d2362",
              "9a6edd86-ab0e-4683-8706-2a2a0e36b977"
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-5m",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "agent_ids": {
      "terms": {
        "field": "agent.id",
        "size": 5
      }
    }
  }
}

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2022-03-09T17:34:19Z

Pinging @elastic/fleet (Team:Fleet)

criamico · 2022-03-09T17:34:59Z

@elasticmachine merge upstream

x-pack/plugins/fleet/server/services/agents/status.ts

x-pack/plugins/fleet/public/components/agent_enrollment_flyout/confirm_incoming_data.tsx

nchaulet · 2022-03-09T18:41:39Z

x-pack/plugins/fleet/server/services/agents/status.ts

+  agentsIds: string[]
+) {
+  try {
+    const searchResult = await esClient.search({


we had previously some performance issue on a similar query with aggregation in the datastream tab does the allow_partial_search_results and timeout here prevent this to throw an error?

Good question, I'm not that familiar with this type of query. There are some info in this doc but I'm going to verify with somebody in the es team to get some guidance.

I changed must to filter and added size:0 option based on this conversation: https://elastic.slack.com/archives/C0D8ST60Y/p1646929886103879

nchaulet

Tested locally and the API worked well! 🚀

What do you think of adding an api integration test for the new route? ES query and aggregations are easy to break and it will make the maintenance and future improvements on that API easier

kpollich

Tested locally and looks good. One very very minor nitpick in the code but otherwise all is well. Thanks for your work on this! 🚀

x-pack/plugins/fleet/public/hooks/use_get_agent_incoming_data.tsx

criamico · 2022-03-10T16:03:37Z

Thanks @kpollich and @nchaulet for your reviews! I pushed another commit that adds a dynamic button. This is meant to work the same way as the "View assets" button that we currently have in the flyout.
If it's displayed when installing an integration it will show "View assets" and link to the integration page, if it's the apm case will point to the apm page. For reference, it's like the two cases below, however I had to rebuild the functionality since currently the whole step is injected into the flyout.

criamico · 2022-03-14T11:25:02Z

@elasticmachine merge upstream

kibana-ci · 2022-03-14T12:47:07Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	568	569	+1

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	1241	1249	+8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	662.9KB	662.9KB	+2.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	110.3KB	110.5KB	+240.0B

Unknown metric groups

API count

id	before	after	diff
`fleet`	1358	1366	+8

History

💔 Build #29475 failed 9325922
💚 Build #28933 succeeded 05ac744
💔 Build #28485 failed 901b3a7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @criamico

kpollich

Ran through again with the new button and all looks great. 🚀

criamico · 2022-03-14T13:46:44Z

What do you think of adding an api integration test for the new route? ES query and aggregations are easy to break and it will make the maintenance and future improvements on that API easier

I've been trying to archive the indices for the integration tests with this command but it doesn't work for me:
node scripts/es_archiver save x-pack/test/functional/es_archives/fleet/incoming_data '.metrics-system*,.logs-system*'
I searched for a solution on slack but none of them worked. Do you know how I could get this data in another way?

nchaulet · 2022-03-14T14:31:02Z

@criamico Looks like es archiver do not support datastream #69061

I think you can still manually write some data to the datastream and delete the datastream with the esClient available in the test framework (not as user friendly but it should work)

criamico · 2022-03-15T15:34:04Z

I'm merging this and I'll add the integration tests in a subsequent PR.

kibanamachine · 2022-03-17T15:46:19Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 127177 or prevent reminders by adding the backport:skip label.

…elastic#127177) * [Fleet] Create endpoint to check if agent has incoming data * Document new endpoint * Improvements to component * Update endpoint schema * Remove button for now * Address review comments * Add dynamic button functionality * Add option to hide button and improve query Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

criamico added 3 commits March 7, 2022 15:53

[Fleet] Create endpoint to check if agent has incoming data

5549f7b

Document new endpoint

2288887

Improvements to component

901b3a7

criamico self-assigned this Mar 8, 2022

criamico added Team:Fleet Team label for Observability Data Collection Fleet team release_note:skip Skip the PR/issue when compiling release notes labels Mar 8, 2022

criamico added 2 commits March 9, 2022 17:15

Update endpoint schema

288dd07

Remove button for now

6cf25fe

criamico marked this pull request as ready for review March 9, 2022 17:34

criamico requested a review from a team as a code owner March 9, 2022 17:34

criamico added the v8.2.0 label Mar 9, 2022

Merge branch 'main' into agent_flyout/agent_incoming_data

05ac744

nchaulet reviewed Mar 9, 2022

View reviewed changes

x-pack/plugins/fleet/server/services/agents/status.ts Show resolved Hide resolved

nchaulet reviewed Mar 9, 2022

View reviewed changes

x-pack/plugins/fleet/public/components/agent_enrollment_flyout/confirm_incoming_data.tsx Outdated Show resolved Hide resolved

nchaulet reviewed Mar 9, 2022

View reviewed changes

criamico changed the title ~~Agent flyout/agent incoming data~~ [Fleet] Add agent incoming data endpoint and presentational component Mar 10, 2022

criamico mentioned this pull request Mar 10, 2022

[Fleet] Agent Install Flyout Redesign #125534

Closed

20 tasks

Address review comments

8cdfe4d

nchaulet approved these changes Mar 10, 2022

View reviewed changes

kpollich self-requested a review March 10, 2022 15:34

kpollich approved these changes Mar 10, 2022

View reviewed changes

x-pack/plugins/fleet/public/hooks/use_get_agent_incoming_data.tsx Outdated Show resolved Hide resolved

Add dynamic button functionality

9325922

Add option to hide button and improve query

e9a03f5

Merge branch 'main' into agent_flyout/agent_incoming_data

d3d50c3

kpollich self-requested a review March 14, 2022 11:44

kpollich approved these changes Mar 14, 2022

View reviewed changes

criamico merged commit 2f876bc into elastic:main Mar 15, 2022

criamico mentioned this pull request Mar 16, 2022

[Fleet] Dynamic "view incoming data" button behaviour #125920

Closed

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 17, 2022

criamico added the backport:skip This commit does not require backporting label Mar 18, 2022

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 18, 2022

criamico deleted the agent_flyout/agent_incoming_data branch July 12, 2022 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add agent incoming data endpoint and presentational component #127177

[Fleet] Add agent incoming data endpoint and presentational component #127177

criamico commented Mar 8, 2022 •

edited

Loading

elasticmachine commented Mar 9, 2022

criamico commented Mar 9, 2022

nchaulet Mar 9, 2022

criamico Mar 10, 2022

criamico Mar 14, 2022

nchaulet left a comment •

edited

Loading

kpollich left a comment

criamico commented Mar 10, 2022

criamico commented Mar 14, 2022

kibana-ci commented Mar 14, 2022

API count

kpollich left a comment

criamico commented Mar 14, 2022 •

edited

Loading

nchaulet commented Mar 14, 2022

criamico commented Mar 15, 2022

kibanamachine commented Mar 17, 2022

[Fleet] Add agent incoming data endpoint and presentational component #127177

[Fleet] Add agent incoming data endpoint and presentational component #127177

Conversation

criamico commented Mar 8, 2022 • edited Loading

Summary

Testing steps

Example of the ES query used:

Checklist

Risk Matrix

For maintainers

elasticmachine commented Mar 9, 2022

criamico commented Mar 9, 2022

nchaulet Mar 9, 2022

Choose a reason for hiding this comment

criamico Mar 10, 2022

Choose a reason for hiding this comment

criamico Mar 14, 2022

Choose a reason for hiding this comment

nchaulet left a comment • edited Loading

Choose a reason for hiding this comment

kpollich left a comment

Choose a reason for hiding this comment

criamico commented Mar 10, 2022

criamico commented Mar 14, 2022

kibana-ci commented Mar 14, 2022

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

kpollich left a comment

Choose a reason for hiding this comment

criamico commented Mar 14, 2022 • edited Loading

nchaulet commented Mar 14, 2022

criamico commented Mar 15, 2022

kibanamachine commented Mar 17, 2022

criamico commented Mar 8, 2022 •

edited

Loading

nchaulet left a comment •

edited

Loading

criamico commented Mar 14, 2022 •

edited

Loading