cache_range_requests plugin optional support for special IMS header

doc/admin-guide/plugins/cache_range_requests.en.rst

@@ -0,0 +1,168 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+
+.. include:: ../../common.defs
+
+.. _admin-plugins-cache-range-requests:
+
+
+Cache Range Requests Plugin
+***************************
+
+Description
+===========
+
+Most origin servers support HTTP/1.1 range requests (rfc 7233).
+ATS internally handles range request caching in one of 2 ways:
+
+* Don't cache range requests.
+* Only server range requests from a wholly cached object.
+
+This plugin allows you to remap individual range requests so that they
+are stored as individual objects in the ATS cache when subsequent range
+requests are likely to use the same range.  This spreads range requests
+over multiple stripes thereby reducing I/O wait and system load averages.
+
+:program:`cache_range_requests` reads the range request header byte range
+value and then creates a new ``cache key URL`` using the original request
+url with the range value appended to it.  The range header is removed
+where appropriate from the requests and the origin server response code
+is changed from a 206 to a 200 to insure that the object is written to
+cache using the new cache key url.  The response code sent to the client
+will be changed back to a 206 and all requests to the origin server will
+contain the range header so that the correct response is received.
+
+The :program:`cache_range_requests` plugin by itself has no logic to
+efficiently manage overlapping ranges.  It is best to use this plugin
+in conjunction with a smart client that only requests predetermined
+non overlapping cache ranges (request blocking) or as a helper for the
+:program:`slice` plugin.
+
+Only requests which contain the ``Range: <units>=`` GET header
+will be served by the :program:`cache_range_requests` plugin.
+
+If/when ATS implements partial object caching this plugin will
+become deprecated.
+
+*NOTE* Given a multi range request the :program:`cache_range_requests`
+only processes the first range and ignores the rest.
+
+How to run the plugin
+=====================
+
+The plugin can run as a global plugin (a single global instance configured
+using :file:`plugin.config`) or as per-remap plugin (a separate instance
+configured per remap rule in :file:`remap.config`).
+
+Global instance
+---------------
+
+.. code::
+
+  $ cat plugin.config
+  cache_range_request.so
+
+
+Per-remap instance
+------------------
+
+.. code::
+
+  $cat remap.config
+  map http://www.example.com http://www.origin.com \
+      @plugin=cache_range_requests.so
+
+
+If both global and per-remap instance are used the per-remap configuration
+would take precedence (per-remap configuration would be applied and the
+global configuration ignored).
+
+Plugin options
+==============
+
+
+Parent Selection as Cache Key
+-----------------------------
+
+.. option:: --ps-cachekey
+.. option:: -p
+
+Without this option parent selection is based solely on the hash of a
+URL Path a URL is requested from the same upstream parent cache listed
+in parent.config
+
+
+With this option parent selection is based on the full ``cache key URL``
+which includes information about the partial content range.  In this mode,
+all requests (include partial content) will use consistent hashing method
+for parent selection.
+
+
+X-CRR-IMS header support
+------------------------
+
+.. option:: --consider-ims
+.. option:: -c
+
+To support slice plugin self healing an option to force revalidation
+after cache lookup complete was added.  This option is triggered by a
+special header:
+
+.. code::
+
+    X-CRR-IMS: Tue, 19 Nov 2019 13:26:45 GMT
+
+When this header is provided and a `cache hit fresh` is encoutered the
+``Date`` header of the object in cache is compared to this header date
+value.  If the cache date is *less* than this IMS date then the object
+is marked as STALE and an appropriate If-Modified-Since or If-Match
+request along with this X-CRR-IMS header is passed up to the parent.
+
+In order for this to properly work in a CDN each cache in the
+chain *SHOULD* also contain a remap rule with the
+:program:`cache_range_requests` plugin with this option set.
+
+
+Configuration examples
+======================
+
+Global plugin
+-------------
+
+.. code::
+
+    cache_range_requests.so --ps-cachekey --consider-ims
+
+or
+
+.. code::
+
+    cache_range_requests.so -p -c
+
+Remap plugin
+------------
+
+.. code::
+
+    map http://ats http://parent @plugin=cache_range_requests.so @pparam=--ps-cachekey @pparam=--consider-ims
+
+or
+
+.. code::
+
+    map http://ats http://parent @plugin=cache_range_requests.so @pparam=-p @pparam=-c

doc/admin-guide/plugins/index.en.rst

@@ -50,6 +50,7 @@ Plugins that are considered stable are installed by default in |TS| releases.
    Background Fetch <background_fetch.en>
    Cache Key Manipulation <cachekey.en>
    Cache Promotion Policies <cache_promote.en>
+   Cache Range Requests <cache_range_requests.en>
    Combo Handler <combo_handler.en>
    Configuration Remap <conf_remap.en>
    Cookie Remap <cookie_remap.en>
@@ -63,7 +64,6 @@ Plugins that are considered stable are installed by default in |TS| releases.
    Regex Remap <regex_remap.en>
    Regex Revalidate <regex_revalidate.en>
    Remap Purge <remap_purge.en>
-   Slice <slice.en>
    Stats over HTTP <stats_over_http.en>
    TCPInfo <tcpinfo.en>
    XDebug <xdebug.en>
@@ -83,6 +83,9 @@ Plugins that are considered stable are installed by default in |TS| releases.
 :doc:`Cache Promotion Policies <cache_promote.en>`
    Allows for control over which assets should be written to cache, or not.
 
+:doc:`Cache Range Requests <cache_range_requests.en>`
+   Cache ranges by adding the range request header to the cache key.
+
 :doc:`Combo Handler <combo_handler.en>`
    Provides an intelligent way to combine multiple URLs into a single URL, and have Apache Traffic Server combine the components into one response.
 
@@ -159,7 +162,7 @@ directory of the |TS| source tree. Experimental plugins can be compiled by passi
    Multiplexer <multiplexer.en>
    MySQL Remap <mysql_remap.en>
    Signed URLs <url_sig.en>
-   Slicer <slice.en>
+   Slice <slice.en>
    SSL Headers <sslheaders.en>
    SSL Session Reuse <ssl_session_reuse.en>
    System Statistics <system_stats.en>
@@ -223,10 +226,10 @@ directory of the |TS| source tree. Experimental plugins can be compiled by passi
 :doc:`Signed URLs <url_sig.en>`
    Adds support for verifying URL signatures for incoming requests to either deny or redirect access.
 
-:doc:`Slicer <slice.en>`
+:doc:`Slice <slice.en>`
    Slice full file or range based requests into deterministic chunks, allowing large files to be
-   spread across multiple cache stripes. Allows range requests to be satisfied by stitching these
-   chunks together.
+   spread across multiple cache stripes. Allows arbitrary range requests to be satisfied by stitching
+   these chunks together.
 
 :doc:`SSL Session Reuse <ssl_session_reuse.en>`
    Coordinates Session ID and ticket based TLS session resumption between a group of ATS machines.

plugins/cache_range_requests/README.md

@@ -0,0 +1,80 @@
+
+Thousands of range requests for a very large object in the traffic server
+cache are likely to increase system load averages due to I/O wait as
+objects are stored on a single stripe or disk drive.
+
+This plugin allows you to remap individual range requests so that they
+are stored as individual objects in the ATS cache when subsequent range
+requests are likely to use the same range.  This spreads range requests
+over multiple stripes thereby reducing I/O wait and system load averages.
+
+This plugin reads the range request header byte range value and then
+creates a new cache key url using the original request url with the range
+value appended to it.  The range header is removed where appropriate
+from the requests and the origin server response code is changed from
+a 206 to a 200 to insure that the object is written to cache using the
+new cache key url.  The response code sent to the client will be changed
+back to a 206 and all requests to the origin server will contain the
+range header so that the correct response is received.
+
+Configuration:
+
+    Add @plugin=cache_range_requests.so to your remap.config rules.
+
+    Or for a global plugin where all range requests are processed,
+    Add cache_range_requests.so to the plugin.config
+
+Parent Selection Mode (consistent-hash only):
+
+    default: Parent selection is based solely on the hash of a URL Path
+             In this mode, all partial content of a URL is requested
+             from the same upstream parent cache listed in parent.config
+
+    Cache_key_url: Parent selection is based on the full cache_key_url
+                   which includes information about the partial content
+                   range.  In this mode, all requests (include partial
+                   content) will use consistent hashing method for
+                   parent selection.
+
+    To enable cache_key_url parent select mode, the following param must be set:
+
+    Global Plugin (plugin.config):
+
+      cache_range_requests.so -p
+      cache_range_requests.so --ps-cachekey
+
+    Remap Plugin (remap.config):
+
+      <from-url> <to-url> @plugin=cache_range_requests.so @pparam=--ps-cachekey
+      <from-url> <to-url> @plugin=cache_range_requests.so @pparam=-p
+
+X-CRR-IMS header support
+
+    To support slice plugin self healing an option to force
+    revalidation after cache lookup complete was added.  This option
+		is triggered by a special header:
+
+    This optional header looks like:
+
+    X-CRR-IMS: Tue, 19 Nov 2019 13:26:45 GMT
+
+		If the cache lookup was a cache hit and the cache header date
+		is *less* than this header value then the cache state is switched
+		from FRESH to STALE which results in If-Modified-Since or
+		If-Match request being passed to the parent.
+
+		In order for this option to be enabled the the following parameter
+		must be set:
+
+    Global Plugin (plugin.config):
+
+      cache_range_requests.so --consider-ims
+      cache_range_requests.so -c
+
+    Remap Plugin (remap.config):
+
+      <from-url> <to-url> @plugin=cache_range_requests.so @pparam=--consider-ims
+      <from-url> <to-url> @plugin=cache_range_requests.so @pparam=-c
+
+    Consider using the header_rewrite plugin to protect the parent
+		from using this option as an attack vector against an origin.