Add documentation for graceful shutdown

doc/admin-guide/graceful-shutdown.en.rst

@@ -0,0 +1,173 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+.. include:: ../common.defs
+
+.. _admin-graceful-shutdown:
+
+Graceful Shutdown
+*****************
+
+Overview
+========
+
+|TS| provides a graceful shutdown feature that allows active client connections
+to complete naturally while signaling to clients that they should close their
+connections. This is useful when preparing a server for maintenance, removing it
+from a load balancer rotation, or performing other operations where you want to
+reduce active connections without abruptly terminating traffic.
+
+The :ref:`traffic_ctl server drain <traffic-control-command-server-drain>`
+command enables this draining mode.
+
+.. _drain-command:
+
+The Drain Command
+=================
+
+Basic Usage
+-----------
+
+To enable drain mode:
+
+.. code-block:: bash
+
+   traffic_ctl server drain
+
+To disable drain mode and return to normal operation:
+
+.. code-block:: bash
+
+   traffic_ctl server drain --undo
+
+How It Works
+------------
+
+When you run :ref:`traffic_ctl server drain
+<traffic-control-command-server-drain>`, the following occurs:
+
+1. A global drain flag is set internally (``TSSystemState::draining = true``).
+2. The ``proxy.process.proxy.draining`` metric is set to 1.
+3. All subsequent client responses are affected as described below.
+
+Behavior by Protocol
+--------------------
+
+HTTP/1.1 Sessions
+~~~~~~~~~~~~~~~~~
+
+For HTTP/1.1 connections, |TS| automatically adds a ``Connection: close`` header
+to all responses. When clients receive this header, compliant clients will:
+
+- Complete the current request/response.
+- Close the connection after receiving the response.
+- Establish a new connection if additional requests are needed.
+
+HTTP/2 Sessions
+~~~~~~~~~~~~~~~
+
+For HTTP/2 connections, |TS| initiates the HTTP/2 graceful shutdown procedure:
+
+- Sends a GOAWAY frame to clients.
+- Clients stop creating new streams on the connection.
+- Existing streams are allowed to complete.
+- Clients close the connection when all active streams finish.
+
+Advantages Over Manual Methods
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The drain command is simpler and more complete than manually setting
+``Connection: close`` headers (for example, via the header_rewrite plugin)
+because it:
+
+- Works automatically without requiring configuration file changes.
+- Handles both HTTP/1.1 and HTTP/2 protocols properly.
+- Can be easily toggled on and off with the ``--undo`` flag.
+- Requires no server restart or configuration reload.
+
+What Drain Does NOT Do
+-----------------------
+
+It's important to understand the limitations of drain mode. Namely, the drain
+mode is something of a soft request to clients to drain current connections
+without other explicit action by |TS|. Specifically:
+
+- **Does NOT close listening sockets**: |TS| continues accepting new connections
+  from clients.
+- **Does NOT terminate existing connections**: Connections remain open until
+  clients close them or they time out naturally.
+- **Does NOT reject new requests**: New connections are accepted and processed
+  normally, but they receive the drain signals described above.
+
+Monitoring Drain Status
+========================
+
+You can check whether |TS| is in drain mode using:
+
+.. code-block:: bash
+
+   traffic_ctl server status
+
+This will show a JSON response including:
+
+.. code-block:: json
+
+   {
+      "initialized_done": "true",
+      "is_ssl_handshaking_stopped": "false",
+      "is_draining": "true",
+      "is_event_system_shut_down": "false"
+   }
+
+You can also monitor the ``proxy.process.proxy.draining`` metric, which will
+be 1 when drain mode is active and 0 otherwise.
+
+Best Practices
+==============
+
+When using drain mode for maintenance or other operations:
+
+1. Remove the server from your load balancer rotation (or otherwise reduce traffic to it).
+2. Run :program:`traffic_ctl server drain`.
+3. Monitor active connections (via metrics, the use of ``ss``, ``top``, etc.).
+4. Wait for connections to drain naturally or until you reach an acceptable level.
+5. Proceed with maintenance or other operations.
+
+Additional recommendations:
+
+- **Combine with load balancer controls**: Use drain mode in conjunction with
+  your load balancer's session persistence and health check features for the
+  smoothest transitions.
+
+- **Monitor connection counts**: Watch your connection metrics to see the effect
+  of drain mode and determine when enough connections have closed.
+
+- **Use** ``--undo`` **if needed**: If you need to return a server to service
+  quickly, use :program:`traffic_ctl server drain --undo` to disable drain mode.
+
+- **Test in non-production first**: Test your drain procedures in a staging
+  environment to understand timing and behavior with your specific traffic
+  patterns.
+
+
+See Also
+========
+
+- :ref:`traffic_ctl server drain <traffic-control-command-server-drain>`
+- :ref:`traffic_ctl server status <traffic-control-command-server-status>`
+- :ts:cv:`proxy.config.stop.shutdown_timeout`
+

doc/admin-guide/index.en.rst

@@ -31,6 +31,7 @@ Table of Contents:
    installation/index.en
    configuration/index.en
    interaction/index.en
+   graceful-shutdown.en
    security/index.en
    storage/index.en
    plugins/index.en

doc/appendices/command-line/traffic_ctl.en.rst

@@ -378,14 +378,63 @@ traffic_ctl server
 
 .. program:: traffic_ctl server
 
+.. _traffic-control-command-server-drain:
+
 .. program:: traffic_ctl server
 .. option:: drain
 
-   :ref:`admin_server_start_drain`
+   Enable graceful connection draining mode. When enabled, |TS| signals clients
+   to close their connections gracefully by:
+
+   - Adding ``Connection: close`` headers to HTTP/1.1 responses
+   - Sending GOAWAY frames to HTTP/2 clients for graceful shutdown
+
+   This allows active connections to complete naturally while encouraging clients
+   to establish new connections. |TS| continues accepting new connections while
+   in drain mode.
+
+   .. note::
+
+      Drain mode does NOT close listening sockets or terminate existing connections.
+      It only signals to clients that they should close their connections.
+
+   .. option:: --undo, -U
+
+      Disable drain mode and return to normal operation.
+
+   .. option:: --no-new-connection, -N
+
+      This option is accepted but not yet implemented. It has no effect in the
+      current version.
+
+   Examples:
+
+   Enable drain mode:
+
+   .. code-block:: bash
+
+      $ traffic_ctl server drain
+
+   Disable drain mode:
+
+   .. code-block:: bash
+
+      $ traffic_ctl server drain --undo
+
+   Check drain status:
+
+   .. code-block:: bash
+
+      $ traffic_ctl server status | jq '.is_draining'
+      "false"
+
+   For detailed information about drain behavior and best practices, see
+   :ref:`admin-graceful-shutdown`.
 
-   :ref:`admin_server_stop_drain`
+   See also:
 
-   Drop the number of active client connections.
+   - :ref:`admin_server_start_drain` (JSONRPC API)
+   - :ref:`admin_server_stop_drain` (JSONRPC API)
 
 
 .. _traffic-control-command-server-status:

doc/developer-guide/jsonrpc/jsonrpc-api.en.rst

@@ -1120,7 +1120,7 @@ admin_server_stop_drain
 Description
 ~~~~~~~~~~~
 
-Stop the drain requests process. Recover server from the drain mode
+Stop the process of draining connections. See :ref:`admin-graceful-shutdown` for more information.
 
 Parameters
 ~~~~~~~~~~
@@ -1157,7 +1157,7 @@ admin_server_start_drain
 Description
 ~~~~~~~~~~~
 
-Drain TS requests.
+Drain TS connections. See :ref:`admin-graceful-shutdown` for more information.
 
 Parameters
 ~~~~~~~~~~