[Synthetics] stream results back for project monitors #138069

dominiqueclarke · 2022-08-04T02:20:17Z

Summary

After elastic/uptime#475, the project monitors api became much slower, often resulting in timeouts on the synthetic-agent side for even a small number of monitors, sometimes as low as 5. This PR accounts for that by streaming updates back to the synthetics agent. The final update in the stream remains the same, and consists of a full summary of the created, updated, deleted, and failed monitors.

EDIT: Added by @justinkambic:

Testing this PR

It is possible to test the push command from Synthetics with only a Kibana/ES cluster. It's also possible to test this E2E, with some additional configuration.

Testing `push` only

You can configure Kibana to do this in a few minutes.

Start up Kibana against a clean ES cluster
Navigate to Uptime
Enable Monitor Management

4. Click `Private Locations`, and follow the flow to make an Agent Policy

5. Navigate back to Monitor Management, follow the flow to add a Private Location

6. Create an [API key](https://www.elastic.co/guide/en/kibana/master/api-keys.html) for your admin user in Kibana. 7. Checkout the [synthetics branch that supports push streaming](https://github.com/elastic/synthetics/pull/576). 8. `npm run build && npm link`. Then, make a new directory for the project you will push. `mkdir {PROJECT_NAME}`. For the sake of clarity, make this project folder the same as the private location name you specified. 9. Use the build Synthetics CLI to create your project. `node dist/cli init {PROJECT_PATH}`. Follow the prompts to configure your project. You should see your private location on the list of choices in the CLI, select that one. 10. Enter your project dir and run `npm link @elastic/synthetics`. 11. After Synthetics builds your project, run `push` to see if your project's monitors get loaded to your Kibana instance. Example: `env SYNTHETICS_API_KEY={API_KEY} npm run push`. You should see output like the example below:

> preparing all monitors
> creating all monitors
> addition and completion of single task: monitor created successfully
> adding and removing multiple tasks: monitor created successfully
> example-monitor: monitor created successfully
✓ Pushed

Testing this patch e2e

To do this, you will largely need to replicate the previous testing example, but with some additional configuration.

Firstly, set up the elastic-package utility.

Run elastic-package stack up -d -v --version 8.5.0-SNAPSHOT to spin up a deployment on 8.5 that we can use for testing purposes.
Modify your local Kibana to run on port 5602 with server.port: 5602. elastic-package also uses https by default now, so you'll need to modify your host config to account for this if you were using http previously. Example: elasticsearch.hosts: https://localhost:9200.
Before we can run Kibana, we need to update the kibana_system password for it.

curl -X POST https://localhost:9200/_security/user/kibana_system/_password\?pretty -u elastic:changeme -H 'Content-Type: application/json' -d '{"password": "changeme" }' --insecure

Start Kibana and log in, create a Private Location with the Elastic Agent policy.

4. Define an API key for your user. 5. Follow the instructions in the previous steps to push monitors. 6. Navigate to Uptime, you should be able to observer your monitors running:

Additional testing

Try configuring project monitors for private locations in different spaces to ensure it's working.
You can also create error states in Kibana's server by modifying the PR code.

dominiqueclarke · 2022-08-04T02:23:22Z

src/plugins/bfetch/server/plugin.ts

-        }
-      );
+      };
+      const routeHandler: RequestHandler<any, any, any> = async (context, request, response) => {


I'd like to get these types correct.

dominiqueclarke · 2022-08-04T02:34:52Z

src/plugins/bfetch/server/plugin.ts

+      const routeDefinition = {
+        path: `/${removeLeadingSlash(path)}`,
+        validate: {
+          body: schema.any(),


I suppose I need to remove this for get

dominiqueclarke · 2022-08-04T02:37:22Z

x-pack/plugins/synthetics/server/routes/monitor_cruds/add_monitor_project_stream.ts

+      failedMonitors: pushMonitorFormatter.failedMonitors,
+      failedStaleMonitors: pushMonitorFormatter.failedStaleMonitors,
+    });
+    subject?.complete();


I suppose I need a try catch around everything, to ensure that complete is always called in the event of errors and prevent the connection from hanging.

Think in the case of error, you would need .error from the observer. Thats how Observer pattern works IIRC.

vadimkibana

bfetch code changes LGTM.

elasticmachine · 2022-08-11T16:09:27Z

Pinging @elastic/uptime (Team:uptime)

awahab07 · 2022-08-15T09:40:51Z

@elasticmachine merge upstream

kibana-ci · 2022-08-15T10:41:47Z

💚 Build Succeeded

Buildkite Build
Commit: e7914d6

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`bfetch`	69	71	+2

Unknown metric groups

API count

id	before	after	diff
`bfetch`	78	80	+2

History

💚 Build #64838 succeeded 0aae657
💚 Build #64413 succeeded 76f229a
💔 Build #64355 failed 44da1eedb60a50dd2a9f25fe4794173ebcc86b71
💔 Build #64102 failed 0dd6864f1bfd93bded6762c9ec014a099485c659
💔 Build #64038 failed 2be554bed829f80268907b094ace1f8e867afddb
💚 Build #63962 succeeded 35d1f05345e608ac70c7d3ed51d42ac00e51d7e2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @justinkambic

awahab07

LGTM

End to end tested the PR following the instructions but using a cloud 8.5 snapshot and everything works.

dominiqueclarke · 2022-08-15T19:34:28Z

x-pack/plugins/synthetics/server/routes/monitor_cruds/add_monitor_project.ts

+      });
+    } catch (error) {
+      subject?.error(error);
+    } finally {


I thought I added this? I guess I didn't push my last commit. :(

No worries, we got it in there 😄

dominiqueclarke

LGTM! 🚢

* stream results back for private locations * update types * consolidate apis * adjust tests * Fix API tests. * Define types more clearly. * Reintroduce default method. * Add error handling to observable. Co-authored-by: Justin Kambic <jk@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 4990478)

kibanamachine · 2022-08-15T20:16:36Z

💚 All backports created successfully

Status	Branch	Result
✅	8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

elastic#138848) * stream results back for private locations * update types * consolidate apis * adjust tests * Fix API tests. * Define types more clearly. * Reintroduce default method. * Add error handling to observable. Co-authored-by: Justin Kambic <jk@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 4990478) Co-authored-by: Dominique Clarke <dominique.clarke@elastic.co>

* stream results back for private locations * update types * consolidate apis * adjust tests * Fix API tests. * Define types more clearly. * Reintroduce default method. * Add error handling to observable. Co-authored-by: Justin Kambic <jk@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

dominiqueclarke changed the title ~~[DRAFT] [Synthetics] stream results back for project monitors to account for long running private location tasks~~ [DRAFT] [Synthetics] stream results back for project monitors Aug 4, 2022

dominiqueclarke commented Aug 4, 2022

View reviewed changes

dominiqueclarke force-pushed the fix/synthetics-private-locations-project-stream branch from b81c4e9 to c1d6913 Compare August 4, 2022 02:57

elastic deleted a comment from kibana-ci Aug 4, 2022

dominiqueclarke force-pushed the fix/synthetics-private-locations-project-stream branch from 8fc2377 to 8d60180 Compare August 4, 2022 20:51

justinkambic mentioned this pull request Aug 9, 2022

feat: support streaming mode for push command elastic/synthetics#576

Merged

dominiqueclarke and others added 8 commits August 10, 2022 16:01

stream results back for private locations

2e13274

update types

8776750

consolidate apis

d571dfe

adjust tests

2c92aa1

Fix API tests.

e623d96

Define types more clearly.

4786893

Reintroduce default method.

0646c8f

Add error handling to observable.

76f229a

justinkambic force-pushed the fix/synthetics-private-locations-project-stream branch from 44da1ee to 76f229a Compare August 10, 2022 21:39

justinkambic marked this pull request as ready for review August 10, 2022 23:07

justinkambic requested review from a team as code owners August 10, 2022 23:07

vadimkibana approved these changes Aug 11, 2022

View reviewed changes

botelastic bot added Feature:ExpressionLanguage Interpreter expression language (aka canvas pipeline) Team:Uptime - DEPRECATED Synthetics & RUM sub-team of Application Observability labels Aug 11, 2022

justinkambic changed the title ~~[DRAFT] [Synthetics] stream results back for project monitors~~ [Synthetics] stream results back for project monitors Aug 11, 2022

justinkambic added release_note:skip Skip the PR/issue when compiling release notes v8.4.0 v8.5.0 bug Fixes for quality problems that affect the customer experience labels Aug 11, 2022

justinkambic self-assigned this Aug 11, 2022

Merge branch 'main' into fix/synthetics-private-locations-project-stream

0aae657

Merge branch 'main' into fix/synthetics-private-locations-project-stream

e7914d6

awahab07 approved these changes Aug 15, 2022

View reviewed changes

dominiqueclarke commented Aug 15, 2022

View reviewed changes

justinkambic merged commit 4990478 into elastic:main Aug 15, 2022

kibanamachine mentioned this pull request Aug 15, 2022

[8.4] [Synthetics] stream results back for project monitors (#138069) #138848

Merged

dominiqueclarke deleted the fix/synthetics-private-locations-project-stream branch August 16, 2022 16:11

vadimkibana mentioned this pull request Mar 26, 2024

[Research] bsearch investigation #166206

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Synthetics] stream results back for project monitors #138069

[Synthetics] stream results back for project monitors #138069

dominiqueclarke commented Aug 4, 2022 •

edited by kibanamachine

Loading

dominiqueclarke Aug 4, 2022

dominiqueclarke Aug 4, 2022 •

edited

Loading

dominiqueclarke Aug 4, 2022

vigneshshanmugam Aug 4, 2022

vadimkibana left a comment

elasticmachine commented Aug 11, 2022

awahab07 commented Aug 15, 2022

kibana-ci commented Aug 15, 2022

API count

awahab07 left a comment

dominiqueclarke Aug 15, 2022

justinkambic Aug 15, 2022

dominiqueclarke left a comment

kibanamachine commented Aug 15, 2022

[Synthetics] stream results back for project monitors #138069

[Synthetics] stream results back for project monitors #138069

Conversation

dominiqueclarke commented Aug 4, 2022 • edited by kibanamachine Loading

Summary

Testing this PR

Testing push only

Testing this patch e2e

Additional testing

dominiqueclarke Aug 4, 2022

Choose a reason for hiding this comment

dominiqueclarke Aug 4, 2022 • edited Loading

Choose a reason for hiding this comment

dominiqueclarke Aug 4, 2022

Choose a reason for hiding this comment

vigneshshanmugam Aug 4, 2022

Choose a reason for hiding this comment

vadimkibana left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 11, 2022

awahab07 commented Aug 15, 2022

kibana-ci commented Aug 15, 2022

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

API count

History

awahab07 left a comment

Choose a reason for hiding this comment

dominiqueclarke Aug 15, 2022

Choose a reason for hiding this comment

justinkambic Aug 15, 2022

Choose a reason for hiding this comment

dominiqueclarke left a comment

Choose a reason for hiding this comment

kibanamachine commented Aug 15, 2022

💚 All backports created successfully

Questions ?

dominiqueclarke commented Aug 4, 2022 •

edited by kibanamachine

Loading

Testing `push` only

dominiqueclarke Aug 4, 2022 •

edited

Loading