Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Heartbeat] Always use synthetics data streams for browser #32064

Merged

Conversation

andrewvc
Copy link
Contributor

A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a heartbeat data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use synthetics* data streams which breaks them out by type.

We can't do this in any sort of simple way with the 'heartbeat' prefix. This shouldn't be a practical issue for most users since in 8.x all kibana clusters have synthetics templates pre-installed. Users who have explicitly set their kibana index pattern to only heartbeat-* with no synthetics-* will be affected. Given that we are in beta this potential breakage is worth it for the long term benefits to users.

This also implies Kibana has been setup with Elasticsearch, which is overwhelmingly the case. If kibana is not setup then the mappings for this synthetics index will be incorrect, but this is likely not a real-world concern.

See also elastic/observability-docs#1944

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Check against a fresh stack with the synthetics integration not installed.

A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type.

We can't do this in any sort of simple way with the 'heartbeat' prefix.
This shouldn't be a practical issue for most users since in 8.x all
kibana clusters have synthetics templates pre-installed. Users who have
explicitly set their kibana index pattern to only `heartbeat-*` with no
`synthetics-*` will be affected. Given that we are in beta this
potential breakage is worth it for the long term benefits to users.

This also implies Kibana has been setup with Elasticsearch, which is
overwhelmingly the case. If kibana is not setup then the mappings for
this synthetics index will be incorrect, but this is likely not a
real-world concern.

See also elastic/observability-docs#1944
@andrewvc andrewvc added enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Jun 23, 2022
@andrewvc andrewvc self-assigned this Jun 23, 2022
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 23, 2022
@andrewvc andrewvc requested a review from emilioalvap June 23, 2022 04:06
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 23, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-06-23T21:16:25.646+0000

  • Duration: 49 min 13 sec

Test stats 🧪

Test Results
Failed 0
Passed 1606
Skipped 22
Total 1628

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

@emilioalvap
Copy link
Collaborator

emilioalvap commented Jun 23, 2022

LGTM, tested with a fresh install and heartbeat.yml config only. Browser monitor is configured to use synthetics-*, unlike other types of monitors. Synthetics data view:

image

HB data view:

image

Tested with:

heartbeat.monitors:
- type: http
  enabled: true
  id: my-monitor
  name: My Monitor
  urls: ["http://localhost:9200"]
  schedule: '@every 1m'
- type: icmp
  enabled: true
  id: my-monitor-2
  schedule: "@every 1m"
  hosts: [8.8.8.8]
- type: tcp
  enabled: true
  id: my-monitor-3
  schedule: "@every 1m"
  hosts: ["google.es"]
- type: browser
  enabled: true
  id: my-monitor-4
  schedule: "@every 1m"
  name: "My monitor 2"
  source:
    inline:
      script: |
         step("load homepage", async () => {
           await page.goto('https://www.elastic.co');
         });

@andrewvc andrewvc marked this pull request as ready for review June 23, 2022 21:11
@andrewvc andrewvc requested a review from a team as a code owner June 23, 2022 21:11
@elasticmachine
Copy link
Collaborator

Pinging @elastic/uptime (Team:Uptime)

@andrewvc andrewvc merged commit e99181e into elastic:main Jun 23, 2022
@dominiqueclarke
Copy link

LGTM. Data for browser monitors was properly segmented into data streams. No data was written to heartbeat.

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023
* [Heartbeat] Always use synthetics data streams for browser

A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type.

We can't do this in any sort of simple way with the 'heartbeat' prefix.
This shouldn't be a practical issue for most users since in 8.x all
kibana clusters have synthetics templates pre-installed. Users who have
explicitly set their kibana index pattern to only `heartbeat-*` with no
`synthetics-*` will be affected. Given that we are in beta this
potential breakage is worth it for the long term benefits to users.

This also implies Kibana has been setup with Elasticsearch, which is
overwhelmingly the case. If kibana is not setup then the mappings for
this synthetics index will be incorrect, but this is likely not a
real-world concern.

See also elastic/observability-docs#1944

* Add changelog, improve docs copy, and add deprecating warning to `index` setting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants