[Heartbeat] Always use synthetics data streams for browser (#32064)

* [Heartbeat] Always use synthetics data streams for browser

A frequent UX issue for our users is wanting to set different retention periods for network / screenshot data. Our default settings for browser monitors really hurt here since we lump them all into a `heartbeat` data stream unless users are using fleet. This addresses this issue by defaulting browser monitors to use `synthetics*` data streams which breaks them out by type.

We can't do this in any sort of simple way with the 'heartbeat' prefix.
This shouldn't be a practical issue for most users since in 8.x all
kibana clusters have synthetics templates pre-installed. Users who have
explicitly set their kibana index pattern to only `heartbeat-*` with no
`synthetics-*` will be affected. Given that we are in beta this
potential breakage is worth it for the long term benefits to users.

This also implies Kibana has been setup with Elasticsearch, which is
overwhelmingly the case. If kibana is not setup then the mappings for
this synthetics index will be incorrect, but this is likely not a
real-world concern.

See also elastic/observability-docs#1944

* Add changelog, improve docs copy, and add deprecating warning to `index` setting

CHANGELOG.next.asciidoc

@@ -19,6 +19,8 @@ https://github.com/elastic/beats/compare/v8.2.0\...main[Check the HEAD diff]
 - Add a new `salesforce` module to collect data from salesforce. {pull}31486[31486]
 
 *Heartbeat*
+- Browser monitors (beta) no write to the `synthetics-*` index prefix. {pull}32064[32064]
+- Setting a custom index for a given monitor is now deprecated. Streams are preferred. {pull}32064[32064]
 
 
 *Metricbeat*

heartbeat/docs/monitors/monitor-common-options.asciidoc

@@ -143,6 +143,20 @@ A list of processors to apply to the data generated by the monitor.
 See <<filtering-and-enhancing-data>> for information about specifying
 processors in your config.
 
+[float]
+[[monitor-data-stream]]
+
+Contains options pertaining to data stream naming, following the conventions followed by [Fleet Data Streams](https://www.elastic.co/guide/en/fleet/current/data-streams.html). By default Heartbeat will
+write to a datastream named `heartbeat-VERSION` except in the case of `browser` monitors, which will
+always follow the Fleet convention of `type-dataset-namespace`, where the type is always synthetics
+unless overriden.
+
+
+```yaml
+# To enable data streams with the default namespace
+data_stream.namespace: default
+```
+
 [float]
 [[monitor-pipeline]]
 ===== `pipeline`
@@ -156,8 +170,9 @@ input is used.
 
 [float]
 [[monitor-index]]
-===== `index`
+===== `index` (deprecated)
 
+This setting is now deprecated in favor of the `data_stream` option.
 If present, this formatted string overrides the index for events from this input
 (for elasticsearch outputs), or sets the `raw_index` field of the event's
 metadata (for other outputs). This string can only refer to the agent name and

heartbeat/monitors/factory.go

@@ -205,6 +205,15 @@ func preProcessors(info beat.Info, settings publishSettings, monitorType string)
 	// Always set event.dataset
 	procs.AddProcessor(actions.NewAddFields(mapstr.M{"event": mapstr.M{"dataset": dataset}}, true, true))
 
+	// always use synthetics data streams for browser monitors, there is no good reason not to
+	// the default `heartbeat` data stream won't split out network and screenshot data.
+	// at some point we should make all monitors use the `synthetics` datastreams and retire
+	// the heartbeat one, but browser is the only beta one, and it would be a breaking change
+	// to do so otherwise.
+	if monitorType == "browser" && settings.DataStream == nil {
+		settings.DataStream = &add_data_stream.DataStream{}
+	}
+
 	if settings.DataStream != nil {
 		ds := *settings.DataStream
 		if ds.Type == "" {
@@ -218,6 +227,7 @@ func preProcessors(info beat.Info, settings publishSettings, monitorType string)
 	}
 
 	if !settings.Index.IsEmpty() {
+		logp.L().Warn("Deprecated use of 'index' setting in heartbeat monitor, use 'data_stream' instead!")
 		proc, err := indexProcessor(&settings.Index, info)
 		if err != nil {
 			return nil, err

heartbeat/monitors/factory_test.go

@@ -48,19 +48,31 @@ func TestPreProcessors(t *testing.T) {
 		wantProc           bool
 		wantErr            bool
 	}{
-		"no settings should yield no processor": {
+		"no settings should yield no processor for lightweight monitor": {
 			publishSettings{},
 			"",
 			nil,
-			"browser",
+			"http",
 			false,
 			false,
 		},
+		"no settings should yield a data stream processor for browsers": {
+			publishSettings{},
+			"synthetics-browser-default",
+			&add_data_stream.DataStream{
+				Namespace: "default",
+				Dataset:   "browser",
+				Type:      "synthetics",
+			},
+			"browser",
+			true,
+			false,
+		},
 		"exact index should be used exactly": {
 			publishSettings{Index: *fmtstr.MustCompileEvent("test")},
 			"test",
 			nil,
-			"browser",
+			"http",
 			true,
 			false,
 		},