Implement Client Side Operation Timeout (CSOT) feature (#1215)

Introduce a unified Client Side Operation Timeout (CSOT) as an Alpha feature to simplify the multitude of existing
timeout settings. This consolidation merges various complex timeout settings into a single, overarching operation
timeout option, enhancing usability and predictability. Existing timeout options will continue to be honored if the
operation timeout is not set. The options `serverSelectionTimeoutMS` and `connectTimeoutMS` are still honored even
if the operation timeout is set.

Key Changes:

- Implement CSOT across various layers of the driver, applying the new operation 
  timeout configuration to govern execution time in areas such as Authentication, 
  Connection Monitoring and Pooling, Server Discovery and Monitoring, CRUD, GridFS, 
  CSFLE, Transactions, Handshake, and Server Selection.

- Annotate CSOT API as Alpha, indicating that this public API is in 
  the early stages of development and may undergo incompatible changes.

- Centralize the setting of maxTimeMS just before sending commands to the server. 
  This optimization enhances performance and ensures precise enforcement of timeouts.

- Introduce TimeoutContext to centralize timeout-related logic, enhancing the 
  clarity and ease of maintenance for timeout management within the driver.

- Enable a new retry mode when CSOT is active, allowing operations to retry until the timeout is reached.
  This improves resiliency compared to the previous behavior of retrying only once.

- Enhance MongoClient, MongoDatabase, MongoCollection, and ClientSession to support 
  setting and inheriting the operation timeout.

- Introduce MongoCluster, a new conceptual entity representing server deployments, 
  configurable with specific writeConcern, timeout, and read preference.

- Account for minimum Round Trip Time (RTT) for socket reads and writes when CSOT is enabled to ensure timeouts
  are adjusted for network latency, avoiding connection churn.

- Standardize CSOT error handling by transforming timeout errors into the new `MongoOperationTimeoutException`.

---------

Co-authored-by: Ross Lawley <ross@mongodb.com>
Co-authored-by: Maxim Katcharov <maxim.katcharov@mongodb.com>
Co-authored-by: Jeff Yemin <jeff.yemin@mongodb.com>
Co-authored-by: Viacheslav Babanin <slav.babanin@mongodb.com>
Co-authored-by: Valentin Kovalenko <valentin.kovalenko@mongodb.com>

Author: 5 people

README.md

@@ -54,6 +54,17 @@ almost always be binary compatible with prior minor releases from the same major
 Patch 5.x.y increments (such as 5.0.0 -> 5.0.1, 5.1.1 -> 5.1.2, etc) will occur for bug fixes only and will always be binary compatible
 with prior patch releases of the same minor release branch.
 
+#### @Alpha
+
+APIs marked with the `@Alpha` annotation are in the early stages of development, subject to incompatible changes, 
+or even removal, in a future release and may lack some intended features. An APIs bearing `@Alpha` annotation may 
+contain known issues affecting functionality, performance, and stability. They are also exempt from any compatibility 
+guarantees made by its containing library.
+
+It is inadvisable for <i>applications</i> to use Alpha APIs in production environments or for <i>libraries</i>
+(which get included on users' CLASSPATHs, outside the library developers' control) to depend on these APIs. Alpha APIs
+are intended for <b>experimental purposes</b> only.
+
 #### @Beta
 
 APIs marked with the `@Beta` annotation at the class or method level are subject to change. They can be modified in any way, or even

THIRD-PARTY-NOTICES

@@ -37,7 +37,10 @@ https://github.com/mongodb/mongo-java-driver.
     See the License for the specific language governing permissions and
     limitations under the License.
 
-3) The following files: Beta.java
+3) The following files:
+
+    Alpha.java (formerly Beta.java)
+    Beta.java
 
     Copyright 2010 The Guava Authors
     Copyright 2011 The Guava Authors

bson/src/main/org/bson/assertions/Assertions.java

@@ -116,6 +116,19 @@ public static <T> T assertNotNull(@Nullable final T value) throws AssertionError
         return value;
     }
 
+    /**
+     * Throw AssertionError if the condition if false.
+     *
+     * @param name      the name of the state that is being checked
+     * @param condition the condition about the parameter to check
+     * @throws AssertionError if the condition is false
+     */
+    public static void assertTrue(final String name, final boolean condition) {
+        if (!condition) {
+            throw new AssertionError("state should be: " + assertNotNull(name));
+        }
+    }
+
     /**
      * Cast an object to the given class and return it, or throw IllegalArgumentException if it's not assignable to that class.
      *

build.gradle

@@ -158,6 +158,7 @@ configure(scalaProjects) {
                     "-unchecked",
                     "-language:reflectiveCalls",
                     "-Wconf:cat=deprecation:ws,any:e",
+                    "-Wconf:msg=While parsing annotations in:silent",
                     "-Xlint:strict-unsealed-patmat"
             ]
         }

config/checkstyle/suppressions.xml

@@ -29,6 +29,7 @@
     <suppress checks="MethodLength" files="QuickTour"/>
     <suppress checks="Regexp" files="Tour"/>
 
+    <suppress checks="FileLength" files="UnifiedCrudHelper"/>
     <suppress checks="MethodLength" files="PojoRoundTripTest"/>
     <suppress checks="MethodLength" files="AbstractUnifiedTest"/>
     <suppress checks="MethodLength" files="AbstractClientSideEncryptionTest"/>
@@ -87,6 +88,7 @@
     <suppress checks="ParameterNumber" files="Operations"/>
     <suppress checks="ParameterNumber" files="ChangeStreamDocument"/>
     <suppress checks="ParameterNumber" files="StructuredLogger"/>
+    <suppress checks="ParameterNumber" files="MongoClusterImpl"/>
 
     <!--Legacy code that has not yet been cleaned-->
     <suppress checks="FinalClass" files="AggregationOptions"/>

config/spotbugs/exclude.xml

@@ -229,7 +229,7 @@
     -->
     <Match>
         <!-- MongoDB status: "False Positive", SpotBugs rank: 13 -->
-        <Class name="com.mongodb.kotlin.client.coroutine.MongoClient"/>
+        <Class name="com.mongodb.kotlin.client.coroutine.MongoCluster"/>
         <Method name="startSession"/>
         <Bug pattern="NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE"/>
     </Match>
@@ -239,4 +239,25 @@
         <Bug pattern="NP_NONNULL_PARAM_VIOLATION"/>
     </Match>
 
+    <!-- Ignoring await return; intended to be used in a loop -->
+    <Match>
+        <Class name="com.mongodb.internal.time.Timeout"/>
+        <Bug pattern="RV_RETURN_VALUE_IGNORED_BAD_PRACTICE"/>
+    </Match>
+    <Match>
+        <Class name="com.mongodb.internal.time.Timeout"/>
+        <Bug pattern="RV_RETURN_VALUE_IGNORED"/>
+    </Match>
+    <Match>
+        <Class name="com.mongodb.internal.time.Timeout"/>
+        <Bug pattern="WA_AWAIT_NOT_IN_LOOP"/>
+    </Match>
+
+    <!-- Void method returning null but @NotNull API -->
+    <Match>
+        <Class name="com.mongodb.internal.operation.DropIndexOperation"/>
+        <Method name="execute"/>
+        <Bug pattern="NP_NONNULL_RETURN_VIOLATION"/>
+    </Match>
+
 </FindBugsFilter>

driver-core/build.gradle

@@ -58,6 +58,7 @@ dependencies {
     implementation "org.mongodb:mongodb-crypt:$mongoCryptVersion", optional
 
     testImplementation project(':bson').sourceSets.test.output
+    testImplementation('org.junit.jupiter:junit-jupiter-api')
     testRuntimeOnly "io.netty:netty-tcnative-boringssl-static"
 
     classifiers.forEach {

driver-core/src/main/com/mongodb/AwsCredential.java

@@ -17,6 +17,7 @@
 package com.mongodb;
 
 import com.mongodb.annotations.Beta;
+import com.mongodb.annotations.Reason;
 import com.mongodb.lang.Nullable;
 
 import static com.mongodb.assertions.Assertions.notNull;
@@ -28,7 +29,7 @@
  * @see MongoCredential#AWS_CREDENTIAL_PROVIDER_KEY
  * @since 4.4
  */
-@Beta(Beta.Reason.CLIENT)
+@Beta(Reason.CLIENT)
 public final class AwsCredential {
     private final String accessKeyId;
     private final String secretAccessKey;

driver-core/src/main/com/mongodb/ClientEncryptionSettings.java

@@ -16,15 +16,21 @@
 
 package com.mongodb;
 
+import com.mongodb.annotations.Alpha;
 import com.mongodb.annotations.NotThreadSafe;
+import com.mongodb.annotations.Reason;
+import com.mongodb.lang.Nullable;
 
 import javax.net.ssl.SSLContext;
 import java.util.HashMap;
 import java.util.Map;
+import java.util.concurrent.TimeUnit;
 import java.util.function.Supplier;
 
 import static com.mongodb.assertions.Assertions.notNull;
+import static com.mongodb.internal.TimeoutSettings.convertAndValidateTimeout;
 import static java.util.Collections.unmodifiableMap;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
 
 /**
  * The client-side settings for data key creation and explicit encryption.
@@ -42,6 +48,8 @@ public final class ClientEncryptionSettings {
     private final Map<String, Map<String, Object>> kmsProviders;
     private final Map<String, Supplier<Map<String, Object>>> kmsProviderPropertySuppliers;
     private final Map<String, SSLContext> kmsProviderSslContextMap;
+    @Nullable
+    private final Long timeoutMS;
     /**
      * A builder for {@code ClientEncryptionSettings} so that {@code ClientEncryptionSettings} can be immutable, and to support easier
      * construction through chaining.
@@ -53,6 +61,8 @@ public static final class Builder {
         private Map<String, Map<String, Object>> kmsProviders;
         private Map<String, Supplier<Map<String, Object>>> kmsProviderPropertySuppliers = new HashMap<>();
         private Map<String, SSLContext> kmsProviderSslContextMap = new HashMap<>();
+        @Nullable
+        private Long timeoutMS;
 
         /**
          * Sets the {@link MongoClientSettings} that will be used to access the key vault.
@@ -120,6 +130,43 @@ public Builder kmsProviderSslContextMap(final Map<String, SSLContext> kmsProvide
             return this;
         }
 
+        /**
+         * Sets the time limit for the full execution of an operation.
+         *
+         * <ul>
+         *   <li>{@code null} means that the timeout mechanism for operations will defer to using:
+         *    <ul>
+         *        <li>{@code waitQueueTimeoutMS}: The maximum wait time in milliseconds that a thread may wait for a connection to become
+         *        available</li>
+         *        <li>{@code socketTimeoutMS}: How long a send or receive on a socket can take before timing out.</li>
+         *        <li>{@code wTimeoutMS}: How long the server will wait for the write concern to be fulfilled before timing out.</li>
+         *        <li>{@code maxTimeMS}: The cumulative time limit for processing operations on a cursor.
+         *        See: <a href="https://docs.mongodb.com/manual/reference/method/cursor.maxTimeMS">cursor.maxTimeMS</a>.</li>
+         *        <li>{@code maxCommitTimeMS}: The maximum amount of time to allow a single {@code commitTransaction} command to execute.
+         *        See: {@link TransactionOptions#getMaxCommitTime}.</li>
+         *   </ul>
+         *   </li>
+         *   <li>{@code 0} means infinite timeout.</li>
+         *    <li>{@code > 0} The time limit to use for the full execution of an operation.</li>
+         * </ul>
+         *
+         * <p><strong>Note:</strong> The timeout set through this method overrides the timeout defined in the key vault client settings
+         * specified in {@link #keyVaultMongoClientSettings(MongoClientSettings)}.
+         * Essentially, for operations that require accessing the key vault, the remaining timeout from the initial operation
+         * determines the duration allowed for key vault access.</p>
+         *
+         * @param timeout the timeout
+         * @param timeUnit the time unit
+         * @return this
+         * @since 5.2
+         * @see #getTimeout
+         */
+        @Alpha(Reason.CLIENT)
+        public ClientEncryptionSettings.Builder timeout(final long timeout, final TimeUnit timeUnit) {
+            this.timeoutMS = convertAndValidateTimeout(timeout, timeUnit);
+            return this;
+        }
+
         /**
          * Build an instance of {@code ClientEncryptionSettings}.
          *
@@ -253,12 +300,46 @@ public Map<String, SSLContext> getKmsProviderSslContextMap() {
         return unmodifiableMap(kmsProviderSslContextMap);
     }
 
+    /**
+     * The time limit for the full execution of an operation.
+     *
+     * <p>If set the following deprecated options will be ignored:
+     * {@code waitQueueTimeoutMS}, {@code socketTimeoutMS}, {@code wTimeoutMS}, {@code maxTimeMS} and {@code maxCommitTimeMS}</p>
+     *
+     * <ul>
+     *   <li>{@code null} means that the timeout mechanism for operations will defer to using:
+     *    <ul>
+     *        <li>{@code waitQueueTimeoutMS}: The maximum wait time in milliseconds that a thread may wait for a connection to become
+     *        available</li>
+     *        <li>{@code socketTimeoutMS}: How long a send or receive on a socket can take before timing out.</li>
+     *        <li>{@code wTimeoutMS}: How long the server will wait for the write concern to be fulfilled before timing out.</li>
+     *        <li>{@code maxTimeMS}: The cumulative time limit for processing operations on a cursor.
+     *        See: <a href="https://docs.mongodb.com/manual/reference/method/cursor.maxTimeMS">cursor.maxTimeMS</a>.</li>
+     *        <li>{@code maxCommitTimeMS}: The maximum amount of time to allow a single {@code commitTransaction} command to execute.
+     *        See: {@link TransactionOptions#getMaxCommitTime}.</li>
+     *   </ul>
+     *   </li>
+     *   <li>{@code 0} means infinite timeout.</li>
+     *    <li>{@code > 0} The time limit to use for the full execution of an operation.</li>
+     * </ul>
+     *
+     * @param timeUnit the time unit
+     * @return the timeout in the given time unit
+     * @since 5.2
+     */
+    @Alpha(Reason.CLIENT)
+    @Nullable
+    public Long getTimeout(final TimeUnit timeUnit) {
+        return timeoutMS == null ? null : timeUnit.convert(timeoutMS, MILLISECONDS);
+    }
+
     private ClientEncryptionSettings(final Builder builder) {
         this.keyVaultMongoClientSettings = notNull("keyVaultMongoClientSettings", builder.keyVaultMongoClientSettings);
         this.keyVaultNamespace = notNull("keyVaultNamespace", builder.keyVaultNamespace);
         this.kmsProviders = notNull("kmsProviders", builder.kmsProviders);
         this.kmsProviderPropertySuppliers = notNull("kmsProviderPropertySuppliers", builder.kmsProviderPropertySuppliers);
         this.kmsProviderSslContextMap = notNull("kmsProviderSslContextMap", builder.kmsProviderSslContextMap);
+        this.timeoutMS = builder.timeoutMS;
     }
 
 }