Merge branch 'main' into ES-9378

docs/reference/connector/docs/connectors-salesforce.asciidoc

@@ -200,7 +200,7 @@ Once the permissions are set, assign the Profiles, Permission Set or Permission
 Follow these steps in Salesforce:
 
 1. Navigate to `Administration` under the `Users` section.
-2. Select `Users` and choose the user to set the permissions to. 
+2. Select `Users` and choose the user to set the permissions to.
 3. Set the `Profile`, `Permission Set` or `Permission Set Groups` created in the earlier steps.
 
 [discrete#es-connectors-salesforce-sync-rules]
@@ -249,7 +249,7 @@ Allowed values are *SOQL* and *SOSL*.
 [
   {
     "query": "FIND {Salesforce} IN ALL FIELDS",
-    "language": "SOSL" 
+    "language": "SOSL"
   }
 ]
 ----
@@ -381,7 +381,13 @@ See <<es-connectors-content-extraction,content extraction>> for more specifics o
 [discrete#es-connectors-salesforce-known-issues]
 ===== Known issues
 
-There are currently no known issues for this connector.
+* *DLS feature is "type-level" not "document-level"*
++
+Salesforce DLS, added in 8.13.0, does not accomodate specific access controls to specific Salesforce Objects.
+Instead, if a given user/group can have access to _any_ Objects of a given type (`Case`, `Lead`, `Opportunity`, etc), that user/group will appear in the `\_allow_access_control` list for _all_ of the Objects of that type.
+See https://github.com/elastic/connectors/issues/3028 for more details.
++
+
 Refer to <<es-connectors-known-issues,connector known issues>> for a list of known issues for all connectors.
 
 [discrete#es-connectors-salesforce-security]
@@ -396,7 +402,7 @@ This connector is built with the {connectors-python}[Elastic connector framework
 
 View the {connectors-python}/connectors/sources/salesforce.py[source code for this connector^] (branch _{connectors-branch}_, compatible with Elastic _{minor-version}_).
 
-// Closing the collapsible section 
+// Closing the collapsible section
 ===============
 
 
@@ -598,7 +604,7 @@ Once the permissions are set, assign the Profiles, Permission Set or Permission
 Follow these steps in Salesforce:
 
 1. Navigate to `Administration` under the `Users` section.
-2. Select `Users` and choose the user to set the permissions to. 
+2. Select `Users` and choose the user to set the permissions to.
 3. Set the `Profile`, `Permission Set` or `Permission Set Groups` created in the earlier steps.
 
 [discrete#es-connectors-salesforce-client-sync-rules]
@@ -648,7 +654,7 @@ Allowed values are *SOQL* and *SOSL*.
 [
   {
     "query": "FIND {Salesforce} IN ALL FIELDS",
-    "language": "SOSL" 
+    "language": "SOSL"
   }
 ]
 ----
@@ -781,7 +787,13 @@ See <<es-connectors-content-extraction,content extraction>> for more specifics o
 [discrete#es-connectors-salesforce-client-known-issues]
 ===== Known issues
 
-There are currently no known issues for this connector.
+* *DLS feature is "type-level" not "document-level"*
++
+Salesforce DLS, added in 8.13.0, does not accomodate specific access controls to specific Salesforce Objects.
+Instead, if a given user/group can have access to _any_ Objects of a given type (`Case`, `Lead`, `Opportunity`, etc), that user/group will appear in the `\_allow_access_control` list for _all_ of the Objects of that type.
+See https://github.com/elastic/connectors/issues/3028 for more details.
++
+
 Refer to <<es-connectors-known-issues,connector known issues>> for a list of known issues for all connectors.
 
 [discrete#es-connectors-salesforce-client-security]
@@ -797,5 +809,5 @@ This connector is built with the {connectors-python}[Elastic connector framework
 View the {connectors-python}/connectors/sources/salesforce.py[source code for this connector^] (branch _{connectors-branch}_, compatible with Elastic _{minor-version}_).
 
 
-// Closing the collapsible section 
+// Closing the collapsible section
 ===============

docs/reference/data-streams/tsds.asciidoc

@@ -17,7 +17,7 @@ metrics data. Only use a TSDS if you typically add metrics data to {es} in near
 real-time and `@timestamp` order.
 
 A TSDS is only intended for metrics data. For other timestamped data, such as
-logs or traces, use a regular data stream.
+logs or traces, use a <<logs-data-stream,logs data stream>> or regular data stream.
 
 [discrete]
 [[differences-from-regular-data-stream]]

docs/reference/index-modules.asciidoc

@@ -113,10 +113,9 @@ Index mode supports the following values:
 
 `standard`::: Standard indexing with default settings.
 
-`time_series`::: Index mode optimized for storage of metrics documented in <<tsds-index-settings,TSDS Settings>>.
+`tsds`::: _(data streams only)_ Index mode optimized for storage of metrics. For more information, see <<tsds-index-settings>>.
 
-`logsdb`::: Index mode optimized for storage of logs. It applies default sort settings on the `hostname` and `timestamp` fields and uses <<synthetic-source,synthetic `_source`>>. <<index-modules-index-sorting,Index sorting>> on different fields is still allowed.
-preview:[]
+`logsdb`::: _(data streams only)_ Index mode optimized for <<logs-data-stream,logs>>.
 
 [[routing-partition-size]] `index.routing_partition_size`::
 

docs/reference/indices/index-mgmt.asciidoc

@@ -67,7 +67,7 @@ This value is the time period for which your data is guaranteed to be stored. Da
 Elasticsearch at a later time.
 
 [role="screenshot"]
-image::images/index-mgmt/management-data-stream.png[Data stream details]
+image::images/index-mgmt/management-data-stream-fields.png[Data stream details]
 
 * To view more information about a data stream, such as its generation or its
 current index lifecycle policy, click the stream's name. From this view, you can navigate to *Discover* to

docs/reference/indices/put-index-template.asciidoc

@@ -115,10 +115,10 @@ See <<create-index-template,create an index template>>.
 
 `index_mode`::
 (Optional, string) Type of data stream to create. Valid values are `null`
-(regular data stream) and `time_series` (<<tsds,time series data stream>>).
+(standard data stream), `time_series` (<<tsds,time series data stream>>) and `logsdb` 
+(<<logs-data-stream,logs data stream>>).
 +
-If `time_series`, each backing index has an `index.mode` index setting of
-`time_series`.
+The template's `index_mode` sets the `index.mode` of the backing index.
 =====
 
 `index_patterns`::

docs/reference/mapping/fields/synthetic-source.asciidoc

@@ -1,17 +1,10 @@
 [[synthetic-source]]
 ==== Synthetic `_source`
 
-IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
-(indices that have `index.mode` set to `time_series`). For other indices,
-synthetic `_source` is in technical preview. Features in technical preview may
-be changed or removed in a future release. Elastic will work to fix
-any issues, but features in technical preview are not subject to the support SLA
-of official GA features.
-
 Though very handy to have around, the source field takes up a significant amount
 of space on disk. Instead of storing source documents on disk exactly as you
 send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
-Enable this by using the value `synthetic` for the index setting `index.mapping.source.mode`:
+To enable this https://www.elastic.co/subscriptions[subscription] feature, use the value `synthetic` for the index setting `index.mapping.source.mode`:
 
 [source,console,id=enable-synthetic-source-example]
 ----
@@ -30,7 +23,7 @@ PUT idx
 ----
 // TESTSETUP
 
-While this on the fly reconstruction is *generally* slower than saving the source
+While this on-the-fly reconstruction is _generally_ slower than saving the source
 documents verbatim and loading them at query time, it saves a lot of storage
 space. Additional latency can be avoided by not loading `_source` field in queries when it is not needed.
 

muted-tests.yml

@@ -171,9 +171,6 @@ tests:
 - class: org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT
   method: test {p0=synonyms/90_synonyms_reloading_for_synset/Reload analyzers for specific synonym set}
   issue: https://github.com/elastic/elasticsearch/issues/116777
-- class: org.elasticsearch.reservedstate.service.FileSettingsServiceTests
-  method: testStopWorksInMiddleOfProcessing
-  issue: https://github.com/elastic/elasticsearch/issues/117591
 - class: "org.elasticsearch.xpack.esql.qa.multi_node.EsqlSpecIT"
   method: "test {scoring.*}"
   issue: https://github.com/elastic/elasticsearch/issues/117641
@@ -317,6 +314,9 @@ tests:
   issue: https://github.com/elastic/elasticsearch/issues/118220
 - class: org.elasticsearch.xpack.esql.action.EsqlActionBreakerIT
   issue: https://github.com/elastic/elasticsearch/issues/118238
+- class: org.elasticsearch.reservedstate.service.FileSettingsServiceTests
+  method: testInvalidJSON
+  issue: https://github.com/elastic/elasticsearch/issues/116521
 
 # Examples:
 #

server/src/main/java/org/elasticsearch/reservedstate/service/FileSettingsService.java

@@ -37,8 +37,6 @@
 import java.util.List;
 import java.util.Map;
 import java.util.concurrent.ExecutionException;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.atomic.AtomicReference;
 
 import static org.elasticsearch.health.HealthStatus.GREEN;
 import static org.elasticsearch.health.HealthStatus.YELLOW;
@@ -122,6 +120,18 @@ public void handleSnapshotRestore(ClusterState clusterState, Metadata.Builder md
         }
     }
 
+    @Override
+    protected void doStart() {
+        healthIndicatorService.startOccurred();
+        super.doStart();
+    }
+
+    @Override
+    protected void doStop() {
+        super.doStop();
+        healthIndicatorService.stopOccurred();
+    }
+
     /**
      * If the file settings metadata version is set to zero, then we have restored from
      * a snapshot and must reprocess the file.
@@ -211,6 +221,7 @@ protected void processInitialFileMissing() throws ExecutionException, Interrupte
 
     public static class FileSettingsHealthIndicatorService implements HealthIndicatorService {
         static final String NAME = "file_settings";
+        static final String INACTIVE_SYMPTOM = "File-based settings are inactive";
         static final String NO_CHANGES_SYMPTOM = "No file-based setting changes have occurred";
         static final String SUCCESS_SYMPTOM = "The most recent file-based settings were applied successfully";
         static final String FAILURE_SYMPTOM = "The most recent file-based settings encountered an error";
@@ -225,21 +236,33 @@ public static class FileSettingsHealthIndicatorService implements HealthIndicato
             )
         );
 
-        private final AtomicLong changeCount = new AtomicLong(0);
-        private final AtomicLong failureStreak = new AtomicLong(0);
-        private final AtomicReference<String> mostRecentFailure = new AtomicReference<>();
+        private boolean isActive = false;
+        private long changeCount = 0;
+        private long failureStreak = 0;
+        private String mostRecentFailure = null;
 
-        public void changeOccurred() {
-            changeCount.incrementAndGet();
+        public synchronized void startOccurred() {
+            isActive = true;
+            failureStreak = 0;
         }
 
-        public void successOccurred() {
-            failureStreak.set(0);
+        public synchronized void stopOccurred() {
+            isActive = false;
+            mostRecentFailure = null;
         }
 
-        public void failureOccurred(String description) {
-            failureStreak.incrementAndGet();
-            mostRecentFailure.set(description);
+        public synchronized void changeOccurred() {
+            ++changeCount;
+        }
+
+        public synchronized void successOccurred() {
+            failureStreak = 0;
+            mostRecentFailure = null;
+        }
+
+        public synchronized void failureOccurred(String description) {
+            ++failureStreak;
+            mostRecentFailure = description;
         }
 
         @Override
@@ -248,18 +271,20 @@ public String name() {
         }
 
         @Override
-        public HealthIndicatorResult calculate(boolean verbose, int maxAffectedResourcesCount, HealthInfo healthInfo) {
-            if (0 == changeCount.get()) {
+        public synchronized HealthIndicatorResult calculate(boolean verbose, int maxAffectedResourcesCount, HealthInfo healthInfo) {
+            if (isActive == false) {
+                return createIndicator(GREEN, INACTIVE_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of());
+            }
+            if (0 == changeCount) {
                 return createIndicator(GREEN, NO_CHANGES_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of());
             }
-            long numFailures = failureStreak.get();
-            if (0 == numFailures) {
+            if (0 == failureStreak) {
                 return createIndicator(GREEN, SUCCESS_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of());
             } else {
                 return createIndicator(
                     YELLOW,
                     FAILURE_SYMPTOM,
-                    new SimpleHealthIndicatorDetails(Map.of("failure_streak", numFailures, "most_recent_failure", mostRecentFailure.get())),
+                    new SimpleHealthIndicatorDetails(Map.of("failure_streak", failureStreak, "most_recent_failure", mostRecentFailure)),
                     STALE_SETTINGS_IMPACT,
                     List.of()
                 );

server/src/test/java/org/elasticsearch/reservedstate/service/FileSettingsHealthIndicatorServiceTests.java

@@ -22,6 +22,7 @@
 import static org.elasticsearch.health.HealthStatus.GREEN;
 import static org.elasticsearch.health.HealthStatus.YELLOW;
 import static org.elasticsearch.reservedstate.service.FileSettingsService.FileSettingsHealthIndicatorService.FAILURE_SYMPTOM;
+import static org.elasticsearch.reservedstate.service.FileSettingsService.FileSettingsHealthIndicatorService.INACTIVE_SYMPTOM;
 import static org.elasticsearch.reservedstate.service.FileSettingsService.FileSettingsHealthIndicatorService.NO_CHANGES_SYMPTOM;
 import static org.elasticsearch.reservedstate.service.FileSettingsService.FileSettingsHealthIndicatorService.STALE_SETTINGS_IMPACT;
 import static org.elasticsearch.reservedstate.service.FileSettingsService.FileSettingsHealthIndicatorService.SUCCESS_SYMPTOM;
@@ -39,14 +40,27 @@ public void initialize() {
         healthIndicatorService = new FileSettingsHealthIndicatorService();
     }
 
-    public void testInitiallyGreen() {
+    public void testInitiallyGreen() {}
+
+    public void testStartAndStop() {
+        assertEquals(
+            new HealthIndicatorResult("file_settings", GREEN, INACTIVE_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of()),
+            healthIndicatorService.calculate(false, null)
+        );
+        healthIndicatorService.startOccurred();
         assertEquals(
             new HealthIndicatorResult("file_settings", GREEN, NO_CHANGES_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of()),
             healthIndicatorService.calculate(false, null)
         );
+        healthIndicatorService.stopOccurred();
+        assertEquals(
+            new HealthIndicatorResult("file_settings", GREEN, INACTIVE_SYMPTOM, HealthIndicatorDetails.EMPTY, List.of(), List.of()),
+            healthIndicatorService.calculate(false, null)
+        );
     }
 
     public void testGreenYellowYellowGreen() {
+        healthIndicatorService.startOccurred();
         healthIndicatorService.changeOccurred();
         // This is a strange case: a change occurred, but neither success nor failure have been reported yet.
         // While the change is still in progress, we don't change the status.