Updated documentation and added test coverage for some caching behavior

doc/admin-guide/configuration/cache-basics.en.rst

@@ -178,14 +178,14 @@ To specify an absolute freshness limit:
 Specifying Header Requirements
 ------------------------------
 
-To further ensure freshness of the objects in the cache, configure
-Traffic Server to cache only objects with specific headers. By default,
-Traffic Server caches all objects (including objects with no headers);
-you should change the default setting only for specialized proxy
-situations. If you configure Traffic Server to cache only HTTP objects
-with ``Expires`` or ``max-age`` headers, then the cache hit rate will be
-noticeably reduced (since very few objects will have explicit expiration
-information).
+To further ensure freshness of the objects in the cache, Traffic Server by
+default caches only HTTP objects with ``Expires`` or ``max-age`` headers. It can
+also be configured to cache all objects (including objects with no headers).
+Different configurations of header requirements would have a noticeable effect
+on cache hit rate. For instance, if Traffic Server is configured to cache only
+HTTP objects with ``Expires`` or ``max-age`` headers (the default setting), then
+the cache hit rate will be noticeably reduced (since very few objects will have
+explicit expiration information).
 
 To configure Traffic Server to cache objects with specific headers:
 
@@ -207,12 +207,12 @@ Traffic Server bases the servability of a cached object on ``Cache-Control``
 headers that appear in both client requests and server responses. The following
 ``Cache-Control`` headers affect whether objects are served from cache:
 
--  The ``no-cache`` header, sent by clients, tells Traffic Server that
-   it should not serve any objects directly from the cache. When present in a
-   client request, Traffic Server will always obtain the object from the
-   origin server. You can configure Traffic Server to ignore client
-   ``no-cache`` headers. Refer to `Configuring Traffic Server to Ignore Client no-cache Headers`_
-   for more information.
+-  The ``no-cache`` header, sent by clients, tells Traffic Server that it should
+   not serve any objects directly from the cache. When present in a client
+   request, Traffic Server will always obtain the object from the origin server.
+   By default, Traffic Server ignores this header. You can configure Traffic
+   Server to honor client ``no-cache`` headers. Refer to `Configuring Traffic
+   Server to Honor Client no-cache Headers`_ for more information.
 
 -  The ``max-age`` header, sent by servers, is compared to the object
    age. If the age is less than ``max-age``, then the object is fresh
@@ -374,17 +374,17 @@ and files.
 Client Directives
 -----------------
 
-By default, Traffic Server does not cache objects with the following
-request headers:
-
--  ``Authorization``
+Configurations can be set to determine whether to cache objects with the
+following client directives (by default, Traffic Server caches objects with
+these request headers):
 
 -  ``Cache-Control: no-store``
 
 -  ``Cache-Control: no-cache``
 
-   To configure Traffic Server to ignore this request header, refer to
-   `Configuring Traffic Server to Ignore Client no-cache Headers`_.
+   To configure Traffic Server to honor the ``Cache-Control: no-store`` and
+   ``Cache-Control: no-cache`` request headers, refer to `Configuring Traffic
+   Server to Honor Client no-cache Headers`_.
 
 -  ``Cookie`` (for text objects)
 
@@ -394,21 +394,26 @@ request headers:
    cache all cookied content, or cache cookied content that is of image
    type only. For more information, refer to `Caching Cookied Objects`_.
 
-Configuring Traffic Server to Ignore Client no-cache Headers
+Configuring Traffic Server to Honor Client no-cache Headers
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-By default, Traffic Server strictly observes client
-``Cache-Control: no-cache`` directives. If a requested object contains a
-``no-cache`` header, then Traffic Server forwards the request to the
-origin server even if it has a fresh copy in cache. You can configure
-Traffic Server to ignore client ``no-cache`` directives such that it
-ignores ``no-cache`` headers from client requests and serves the object
-from its cache.
+By default, Traffic Server ignores client ``Cache-Control: no-cache``
+directives. Even if a requested object contains a ``no-cache`` directive,
+Traffic Server serves the object from its cache. You can configure Traffic
+Server to honor this header and forward the request to the origin server even
+if it has a fresh copy in cache.
+
+Likewise, by default Traffic Server also ignores client ``Cache-Control:
+no-store`` directives. Traffic Server caches response from the server regardless
+of the ``no-store`` headers from client requests.
+
+You can configure Traffic Server to honor both of these client directives with
+the following:
 
 #. Edit :ts:cv:`proxy.config.http.cache.ignore_client_no_cache` in
    :file:`records.config`. ::
 
-        CONFIG proxy.config.http.cache.ignore_client_no_cache INT 1
+        CONFIG proxy.config.http.cache.ignore_client_no_cache INT 0
 
 #. Run the command :option:`traffic_ctl config reload` to apply the configuration changes.
 
@@ -420,19 +425,18 @@ headers:
 
 -  ``Cache-Control: no-store``
 
--  ``Cache-Control: private``
-
--  ``WWW-Authenticate``
+-  ``Cache-Control: no-cache``
 
-   To configure Traffic Server to ignore ``WWW-Authenticate`` headers,
-   refer to `Configuring Traffic Server to Ignore WWW-Authenticate Headers`_.
+   To configure Traffic Server to ignore ``no-cache`` and ``no-store`` headers,
+   refer to `Configuring Traffic Server to Ignore Server no-cache Headers`_.
 
--  ``Set-Cookie``
+-  ``Cache-Control: private``
 
--  ``Cache-Control: no-cache``
+-  ``WWW-Authenticate``
 
-   To configure Traffic Server to ignore ``no-cache`` headers, refer to
-   `Configuring Traffic Server to Ignore Server no-cache Headers`_.
+   To configure Traffic Server to ignore ``WWW-Authenticate`` headers and cache
+   the corresponding objects, refer to `Configuring Traffic Server to Ignore
+   WWW-Authenticate Headers`_.
 
 -  ``Expires`` header with a value of 0 (zero) or a past date.
 
@@ -547,18 +551,23 @@ Caching Cookied Objects
 
 By default, Traffic Server caches objects served in response to requests
 that contain cookies. This is true for all types of objects including
-text. Traffic Server does not cache cookied text content because object
-headers are stored along with the object, and personalized cookie header
-values could be saved with the object. With non-text objects, it is
-unlikely that personalized headers are delivered or used.
+text.
 
 You can reconfigure Traffic Server to:
 
 -  Not cache cookied content of any type.
 
+-  Cache all cookied content regardless of type.
+
 -  Cache cookied content that is of image type only.
 
--  Cache all cookied content regardless of type.
+-  Cache all cookied content except text type.
+
+   It may be appropriate to configure Traffic Server to not cache cookied text
+   content because object headers are stored along with the object, and
+   personalized cookie header values could be saved with the object. With
+   non-text objects, it is unlikely that personalized headers are delivered or
+   used.
 
 To configure how Traffic Server caches cookied content:
 

doc/admin-guide/files/records.config.en.rst

@@ -2173,7 +2173,7 @@ Cache Control
    :reloadable:
    :overridable:
 
-   When enabled (``1``), |TS| ignores client requests to bypass the cache.
+   When enabled (``1``), |TS| ignores client requests to bypass the cache. Specifically, ``Pragma: no-cache``, ``Cache-Control: no-cache`` and ``Cache-Control: no-store`` in requests are ignored.
 
 .. ts:cv:: CONFIG proxy.config.http.cache.ims_on_client_no_cache INT 1
    :reloadable:
@@ -2185,7 +2185,7 @@ Cache Control
    :reloadable:
    :overridable:
 
-   When enabled (``1``), |TS| ignores origin server requests to bypass the cache.
+   When enabled (``1``), |TS| ignores origin server requests to bypass the cache. Specifically, ``Pragma: no-cache``, ``Cache-Control: no-cache`` and ``Cache-Control: no-store`` in responses are ignored.
 
 .. ts:cv:: CONFIG proxy.config.http.cache.cache_responses_to_cookies INT 1
    :reloadable:
@@ -2207,8 +2207,8 @@ Cache Control
 .. ts:cv:: CONFIG proxy.config.http.cache.ignore_authentication INT 0
    :overridable:
 
-   When enabled (``1``), |TS| ignores ``WWW-Authentication`` headers in responses ``WWW-Authentication`` headers are removed and
-   not cached.
+   When enabled (``1``), |TS| ignores ``WWW-Authentication`` headers in
+   responses and the responses are cached.
 
 .. ts:cv:: CONFIG proxy.config.http.cache.cache_urls_that_look_dynamic INT 1
    :reloadable:

tests/gold_tests/cache/cache-auth.test.py

@@ -0,0 +1,123 @@
+'''
+Test authorization-related caching behaviors
+'''
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+
+Test.Summary = '''
+Test authorization-related caching behaviors
+'''
+
+Test.ContinueOnFail = True
+
+# **testname is required**
+testName = ""
+
+
+class AuthDefaultTest:
+    # Verify the proper caching behavior for request/response containing
+    # auth-related fields when ATS is in default configuration
+    authDefaultReplayFile = "replay/auth-default.replay.yaml"
+
+    def __init__(self):
+        self.setupOriginServer()
+        self.setupTS()
+
+    def setupOriginServer(self):
+        self.server = Test.MakeVerifierServerProcess(
+            "auth-default-verifier-server",
+            self.authDefaultReplayFile)
+
+    def setupTS(self):
+        self.ts = Test.MakeATSProcess("ts-auth-default")
+        self.ts.Disk.records_config.update({
+            "proxy.config.diags.debug.enabled": 1,
+            "proxy.config.diags.debug.tags": "http",
+        })
+        self.ts.Disk.remap_config.AddLine(
+            f"map / http://127.0.0.1:{self.server.Variables.http_port}/",
+        )
+
+        # Verify log for skipping the WWW-Authenticate response
+        self.ts.Disk.traffic_out.Content += Testers.ContainsExpression(
+            "response has WWW-Authenticate, response is not cacheable",
+            "Verify ATS doesn't store the response with WWW-Authenticate.")
+
+    def runTraffic(self):
+        tr = Test.AddTestRun(
+            "Verify the proper caching behavior for request/response containing auth-related fields when ATS is in default configuration")
+        tr.AddVerifierClientProcess(
+            "auth-default-client",
+            self.authDefaultReplayFile,
+            http_ports=[self.ts.Variables.port],
+            other_args='--thread-limit 1')
+        tr.Processes.Default.StartBefore(self.server)
+        tr.Processes.Default.StartBefore(self.ts)
+        tr.StillRunningAfter = self.server
+        tr.StillRunningAfter = self.ts
+
+    def run(self):
+        self.runTraffic()
+
+
+AuthDefaultTest().run()
+
+
+class AuthIgnoredTest:
+    # Verify the proper caching behavior for request/response containing
+    # auth-related fields when ATS is configured to bypass caching for those
+    authIgnoredReplayFile = "replay/auth-ignored.replay.yaml"
+
+    def __init__(self):
+        self.setupOriginServer()
+        self.setupTS()
+
+    def setupOriginServer(self):
+        self.server = Test.MakeVerifierServerProcess(
+            "auth-ignored-verifier-server",
+            self.authIgnoredReplayFile)
+
+    def setupTS(self):
+        self.ts = Test.MakeATSProcess("ts-auth-ignored")
+        self.ts.Disk.records_config.update({
+            "proxy.config.diags.debug.enabled": 1,
+            "proxy.config.diags.debug.tags": "http",
+            # Configure ATS to ignore the WWW-Authenticate header in
+            # response(allow caching of such response)
+            "proxy.config.http.cache.ignore_authentication": 1
+        })
+        self.ts.Disk.remap_config.AddLine(
+            f"map / http://127.0.0.1:{self.server.Variables.http_port}/",
+        )
+
+    def runTraffic(self):
+        tr = Test.AddTestRun(
+            "Verify the proper caching behavior for request/response containing auth-related fields when ATS is configured to bypass caching for those")
+        tr.AddVerifierClientProcess(
+            "auth-ignored-client",
+            self.authIgnoredReplayFile,
+            http_ports=[self.ts.Variables.port],
+            other_args='--thread-limit 1')
+        tr.Processes.Default.StartBefore(self.server)
+        tr.Processes.Default.StartBefore(self.ts)
+        tr.StillRunningAfter = self.server
+        tr.StillRunningAfter = self.ts
+
+    def run(self):
+        self.runTraffic()
+
+
+AuthIgnoredTest().run()