Merge pull request #2913 from vespa-engine/bjorncs/container-threadpool-configuration

Bjorncs/container threadpool configuration

Author: baldersheim

en/performance/container-tuning.html

@@ -22,7 +22,7 @@ <h2 id="container-worker-threads">Container worker threads</h2>
   Most components including request handlers use the container's <em>default thread pool</em>,
   which is controlled by a shared executor instance.
   Any component can utilize the default pool by injecting an <code>java.util.concurrent.Executor</code> instance.
-  Some built-in components have dedicated thread pools - such as the Jetty server, the search handler and the feed handler.
+  Some built-in components have dedicated thread pools - such as the Jetty server and the search handler.
   These thread pools are injected through special wiring in the config model and
   are not easily accessible from other components.
 </p>
@@ -38,46 +38,45 @@ <h2 id="container-worker-threads">Container worker threads</h2>
 <p>
   The container will pre-start the minimum number of worker threads,
   so even an idle container may report running several hundred threads.
-  For thread pools with fixed size (<em>min == max</em>), all threads are pre-started.
+  The thread pool is pre-started with the number of thread specified in the <code>threads</code> parameter.
   Note that tuning the capacity upwards increases the risk of high GC pressure
   as concurrency becomes higher with more in-flight requests.
   The GC pressure is a function of number of in-flight requests, the time it takes to complete the request
   and the amount of garbage produced per request.
   Increasing the queue size will allow the application to handle shorter traffic bursts without rejecting requests,
   although increasing the average latency for those requests that are queued up.
   Large queues will also increase heap consumption in overload situations.
-  Extra threads will be created once the queue is full (if <em>max &gt; min</em>), and are destroyed after an idle timeout.
+  Extra threads will be created once the queue is full (when <code>boost</code> is specified), and are destroyed after an idle timeout.
   If all threads are occupied, requests are rejected with a 503 response.
 </p>
 <p>
   The effective thread pool configuration and utilization statistics can be observed through the
   <a href="../operations/container.html#container-metrics">Container Metrics</a>.
   See <a href="../operations/container.html#thread-pool-metrics">Thread Pool Metrics</a> for a list of metrics exported.
 </p>
-<p>Example configuration override:</p>
+{% include note.html content=' If the queue size is set to 0 the metric measuring the queue size -
+<code>jdisc.thread_pool.work_queue.size</code> - will instead switch to measure how many threads are active.'%}
+
+<h3 id="container-worker-threads-min">Lower limit</h3>
+The container will override any configuration if the effective value is below a fixed minimum. This is to
+reduce the risk of certain deadlock scenarios and improve concurrency for low-resource environments.
+<ul>
+    <li>Minimum 8 threads.</li>
+    <li>Minimum 650 queue capacity (if queue is not disabled).</li>
+</ul>
+
+<h3 id="container-worker-threads-example">Example</h3>
 <pre>{% highlight xml %}
 <container id="container" version="1.0">
 
     <search>
         <!-- Search handler thread pool -->
         <threadpool>
-            <max-threads>500</max-threads>
-            <min-threads>500</min-threads>
-            <queue-size>0</queue-size>
+            <threads boost="12">4</threads>
+            <queue>100</queue>
         </threadpool>
     </search>
 
-    <document-api>
-        <!-- Feed handler thread pool -->
-        <http-client-api>
-            <threadpool>
-                <max-threads>50</max-threads>
-                <min-threads>10</min-threads>
-                <queue-size>1000</queue-size>
-            </threadpool>
-        </http-client-api>
-    </document-api>
-
     <!-- Default thread pool -->
     <config name="container.handler.threadpool">
         <maxthreads>200</maxthreads>

en/reference/services-container.html

@@ -50,11 +50,6 @@
         <a href="#tracelevel">tracelevel</a>
         <a href="#mbusport">mbusport</a>
         <a href="#ignore-undefined-fields">ignore-undefined-fields</a>
-        <a href="#http-client-api">http-client-api</a>
-            <a href="#http-client-api-threadpool">threadpool</a>
-                <a href="#http-client-api-threadpool">max-threads</a>
-                <a href="#http-client-api-threadpool">min-threads</a>
-                <a href="#http-client-api-threadpool">queue-size</a>
     <a href="../stateless-model-evaluation.html">model-evaluation</a>
         <a href="../stateless-model-evaluation.html#onnx-inference-options">onnx</a>
     <a href="#document">document [type, class, bundle]</a>
@@ -397,13 +392,6 @@ <h2 id="document-api">document-api</h2>
         </p>
       </td>
     </tr>
-    <tr>
-        <th>http-client-api</th>
-        <td>optional</td>
-        <td></td>
-        <td></td>
-        <td><p id="document-api.http-client-api">Configuration for the Vespa HTTP client API</p></td>
-    </tr>
   </tbody>
 </table>
 <p>Example:</p>
@@ -420,48 +408,11 @@ <h2 id="document-api">document-api</h2>
     &lt;route&gt;default&lt;/route&gt;
     &lt;timeout&gt;250.5&lt;/timeout&gt;
     &lt;tracelevel&gt;3&lt;/tracelevel&gt;
-    &lt;http-client-api&gt;
-        &lt;threadpool&gt;
-            &lt;max-threads&gt;50&lt;/max-threads&gt;
-            &lt;min-threads&gt;10&lt;/min-threads&gt;
-            &lt;queue-size&gt;1000&lt;/queue-size&gt;
-        &lt;/threadpool&gt;
-    &lt;/http-client-api&gt;
 &lt;document-api&gt;
 </pre>
 
 
 
-<h2 id="http-client-api">http-client-api</h2>
-<pre>
-&lt;http-client-api&gt;
-    &lt;threadpool&gt;
-        &lt;max-threads&gt;50&lt;/max-threads&gt;
-        &lt;min-threads&gt;10&lt;/min-threads&gt;
-        &lt;queue-size&gt;1000&lt;/queue-size&gt;
-    &lt;/threadpool&gt;
-&lt;/http-client-api&gt;
-</pre>
-<p>Children elements:</p>
-<table class="table">
-    <thead>
-    <tr><th>Name</th><th>Required</th><th>Value</th><th>Default</th><th>Description</th></tr>
-    </thead><tbody>
-    <tr><th>threadpool</th>
-        <td>optional</td>
-        <td></td>
-        <td></td>
-        <td></td>
-        <td>
-          <p id="http-client-api-threadpool">
-            Contains configuration of the threadpool for the http client api handler.
-            The pool is initialized with minimum number of threads during startup.
-            Additional threads will be created on demand once the request queue is full.
-            Requests are rejected once maximum threads are reached, all threads are busy and the request queue is full.
-          </p>
-        </td>
-    </tr></tbody>
-</table>
 
 
 

en/reference/services-search.html

@@ -37,9 +37,8 @@
             <a href="#node">node</a>
     <a href="#renderer">renderer [id, class, bundle]</a>
     <a href="#threadpool">threadpool</a>
-        <a href="#threadpool-max-threads">max-threads</a>
-        <a href="#threadpool-min-threads">min-threads</a>
-        <a href="#threadpool-queue-size">queue-size</a>
+        <a href="#threadpool-threads">threads [ boost ]</a>
+        <a href="#threadpool-queue">queue</a>
 </pre>
 <p>
   <a href="config-files.html#generic-configuration-in-services-xml">config</a>
@@ -597,29 +596,36 @@ <h2 id="node">node</h2>
 
 <h2 id="threadpool">threadpool</h2>
 <p>
-  Contains configuration of the threadpool for the jdisc search handler.
-  The pool is initialized with minimum number of threads during startup.
-  Additional threads will be created on demand once the request queue is full.
-  Requests are rejected once maximum threads are reached, all threads are busy and the request queue is full.
+  Specifies configuration for the thread pool for the jdisc search handler. All parameters are relative to the number of CPU cores -
+  if a node has 8 vCPU with <code>threads=4</code>, the thread pool will have 32 threads.
+  Same for queue size - if <code>queue=10</code>, the queue will have capacity for 80 entries.
+  If the <code>boost</code> attribute is specified, additional threads will be created on demand once the request queue is full.
+  These threads are then destructed after idling for a fixed amount of time.
+  Requests are rejected once the maximum number of allowed threads is reached, all threads are busy and the request queue is full.
+  See <a href="container-tuning.html">Container Tuning</a> for more details.
 </p>
 
-
-
-<h2 id="threadpool-max-threads">max-threads</h2>
-<p>
-  Maximum number of threads in pool.
-</p>
-
-
-
-<h2 id="threadpool-min-threads">min-threads</h2>
+<h2 id="threadpool-threads">threads</h2>
 <p>
-  Minimum number of threads in pool.
+  The number of permanent threads relative to CPU cores. Default value is <code>2</code>.
 </p>
+<table class="table">
+  <thead>
+  <tr><th>Attribute</th><th>Required</th><th>Value</th><th>Default</th><th>Description</th></tr>
+  </thead><tbody>
+<tr><th>boost</th>
+  <td>optional</td>
+  <td>number</td>
+  <td></td>
+  <td>
+    <p id="threads.boost">
+      The number of additional threads relative to CPU cores. Default value is <code>2</code>.
+    </p>
+  </td></tr>
+</tbody>
+</table>
 
-
-
-<h2 id="threadpool-queue-size">queue-size</h2>
+<h2 id="threadpool-queue">queue</h2>
 <p>
-  Request queue size.
+  The size of the request queue relative to CPU cores. Specify <code>0</code> to disable queuing. Default value is <code>40</code>.
 </p>