diff --git a/doc/admin-guide/plugins/cache_range_requests.en.rst b/doc/admin-guide/plugins/cache_range_requests.en.rst new file mode 100644 index 00000000000..6d11d21121a --- /dev/null +++ b/doc/admin-guide/plugins/cache_range_requests.en.rst @@ -0,0 +1,191 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +.. include:: ../../common.defs + +.. _admin-plugins-cache-range-requests: + + +Cache Range Requests Plugin +*************************** + +Description +=========== + +Most origin servers support HTTP/1.1 range requests (rfc 7233). +ATS internally handles range request caching in one of 2 ways: + +* Don't cache range requests. +* Only server range requests from a wholly cached object. + +This plugin allows you to remap individual range requests so that they +are stored as individual objects in the ATS cache when subsequent range +requests are likely to use the same range. This spreads range requests +over multiple stripes thereby reducing I/O wait and system load averages. + +:program:`cache_range_requests` reads the range request header byte range +value and then creates a new ``cache key URL`` using the original request +url with the range value appended to it. The range header is removed +where appropriate from the requests and the origin server response code +is changed from a 206 to a 200 to insure that the object is written to +cache using the new cache key url. The response code sent to the client +will be changed back to a 206 and all requests to the origin server will +contain the range header so that the correct response is received. + +The :program:`cache_range_requests` plugin by itself has no logic to +efficiently manage overlapping ranges. It is best to use this plugin +in conjunction with a smart client that only requests predetermined +non overlapping cache ranges (request blocking) or as a helper for the +:program:`slice` plugin. + +Only requests which contain the ``Range: =`` GET header +will be served by the :program:`cache_range_requests` plugin. + +If/when ATS implements partial object caching this plugin will +become deprecated. + +*NOTE* Given a multi range request the :program:`cache_range_requests` +only processes the first range and ignores the rest. + +How to run the plugin +===================== + +The plugin can run as a global plugin (a single global instance configured +using :file:`plugin.config`) or as per-remap plugin (a separate instance +configured per remap rule in :file:`remap.config`). + +Global instance +--------------- + +.. code:: + + $ cat plugin.config + cache_range_request.so + + +Per-remap instance +------------------ + +.. code:: + + $cat remap.config + map http://www.example.com http://www.origin.com \ + @plugin=cache_range_requests.so + + +If both global and per-remap instance are used the per-remap configuration +would take precedence (per-remap configuration would be applied and the +global configuration ignored). + +Plugin options +============== + + +Parent Selection as Cache Key +----------------------------- + +.. option:: --ps-cachekey +.. option:: -p + +Without this option parent selection is based solely on the hash of a +URL Path a URL is requested from the same upstream parent cache listed +in parent.config + + +With this option parent selection is based on the full ``cache key URL`` +which includes information about the partial content range. In this mode, +all requests (include partial content) will use consistent hashing method +for parent selection. + + +X-CRR-IMS header support +------------------------ + +.. option:: --consider-ims +.. option:: -c + +To support slice plugin self healing an option to force revalidation +after cache lookup complete was added. This option is triggered by a +special header: + +.. code:: + + X-CRR-IMS: Tue, 19 Nov 2019 13:26:45 GMT + +When this header is provided and a `cache hit fresh` is encoutered the +``Date`` header of the object in cache is compared to this header date +value. If the cache date is *less* than this IMS date then the object +is marked as STALE and an appropriate If-Modified-Since or If-Match +request along with this X-CRR-IMS header is passed up to the parent. + +In order for this to properly work in a CDN each cache in the +chain *SHOULD* also contain a remap rule with the +:program:`cache_range_requests` plugin with this option set. + +Don't modify the Cache Key +-------------------------- + +.. option:: --no-modify-cachekey +.. option:: -n + +With each transaction TSCacheUrlSet may only be called once. When +using the `cache_range_requests` plugin in conjunction with the +`cachekey` plugin the option `--include-headers=Range` should be +added as a `cachekey` parameter with this option. Configuring this +incorrectly *WILL* result in cache poisoning. + +.. code:: + + map http://ats/ http://parent/ \ + @plugin=cachekey.so @pparam=--include-headers=Range \ + @plugin=cache_range_requests.so @pparam=--no-modify-cachekey + +*Without this `cache_range_requests` plugin option* + +*IF* the TSCacheUrlSet call in cache_range_requests fails, an error is +generated in the logs and the cache_range_requests plugin will disable +transaction caching in order to avoid cache poisoning. + +Configuration examples +====================== + +Global plugin +------------- + +.. code:: + + cache_range_requests.so --ps-cachekey --consider-ims --no-modify-cachekey + +or + +.. code:: + + cache_range_requests.so -p -c -n + +Remap plugin +------------ + +.. code:: + + map http://ats http://parent @plugin=cache_range_requests.so @pparam=--ps-cachekey @pparam=--consider-ims @pparam=--no-modify-cachekey + +or + +.. code:: + + map http://ats http://parent @plugin=cache_range_requests.so @pparam=-p @pparam=-c @pparam=-n diff --git a/doc/admin-guide/plugins/xdebug.en.rst b/doc/admin-guide/plugins/xdebug.en.rst index fd92bc06b1e..09c0791a4db 100644 --- a/doc/admin-guide/plugins/xdebug.en.rst +++ b/doc/admin-guide/plugins/xdebug.en.rst @@ -101,3 +101,9 @@ X-Transaction-ID X-Remap If the URL was remapped for a request, this header gives the *to* and *from* field from the line in remap.config that caused the URL to be remapped. + +X-ParentSelection-Key + The ``X-ParentSelection-Key`` header contains the URL that is used to + determine parent selection for an object in the Traffic Server. This + header is particularly useful if a custom parent selection key is + being used. diff --git a/plugins/experimental/cache_range_requests/cache_range_requests.cc b/plugins/experimental/cache_range_requests/cache_range_requests.cc index b0b61ec1cd1..7ed63d2525b 100644 --- a/plugins/experimental/cache_range_requests/cache_range_requests.cc +++ b/plugins/experimental/cache_range_requests/cache_range_requests.cc @@ -41,7 +41,9 @@ typedef enum parent_select_mode { } parent_select_mode_t; struct pluginconfig { - parent_select_mode_t ps_mode; + parent_select_mode_t ps_mode{PS_DEFAULT}; + bool consider_ims_header{false}; + bool modify_cache_key{true}; }; struct txndata { @@ -79,18 +81,32 @@ create_pluginconfig(int argc, const char *argv[]) return nullptr; } - // Plugin uses default ATS selection (hash of URL path) - pc->ps_mode = PS_DEFAULT; - + static const struct option longopts[] = { + {const_cast("ps-cachekey"), no_argument, nullptr, 'p'}, + {const_cast("consider-ims"), no_argument, nullptr, 'c'}, + {const_cast("no-modify-cachekey"), no_argument, nullptr, 'n'}, + {nullptr, 0, nullptr, 0}, + }; + // // Walk through param list. for (int c = 0; c < argc; c++) { - if (strcmp("ps_mode:cache_key_url", argv[c]) == 0) { - pc->ps_mode = PS_CACHEKEY_URL; - break; - } + case 'p': { + pc->ps_mode = PS_CACHEKEY_URL; + } break; + case 'c': { + DEBUG_LOG("Plugin considers the '%.*s' header", (int)X_IMS_HEADER.size(), X_IMS_HEADER.data()); + pc->consider_ims_header = true; + } break; + case 'n': { + DEBUG_LOG("Plugin doesn't modify cache key"); + pc->modify_cache_key = false; + } break; + default: { + } break; } +} - return pc; +return pc; } /** @@ -165,23 +181,41 @@ range_header_check(TSHttpTxn txnp, struct pluginconfig *pc) TSfree(req_url); } - // set the cache key. - if (TS_SUCCESS != TSCacheUrlSet(txnp, cache_key_url, cache_key_url_length)) { - DEBUG_LOG("failed to change the cache url to %s.", cache_key_url); - } + if (nullptr != pc) { + // set the cache key if configured to. + if (pc->modify_cache_key && TS_SUCCESS != TSCacheUrlSet(txnp, cache_key_url, cache_key_url_length)) { + ERROR_LOG("failed to change the cache url to %s.", cache_key_url); + ERROR_LOG("Disabling cache for this transaction to avoid cache poisoning."); + TSHttpTxnServerRespNoStoreSet(txnp, 1); + TSHttpTxnRespCacheableSet(txnp, 0); + TSHttpTxnReqCacheableSet(txnp, 0); + } + + // Optionally set the parent_selection_url to the cache_key url or path + if (PS_DEFAULT != pc->ps_mode) { + TSMLoc ps_loc = nullptr; + + if (PS_CACHEKEY_URL == pc->ps_mode) { + const char *start = cache_key_url; + const char *end = cache_key_url + cache_key_url_length; + if (TS_SUCCESS == TSUrlCreate(hdr_buf, &ps_loc) && + TS_PARSE_DONE == TSUrlParse(hdr_buf, ps_loc, &start, end) && // This should always succeed. + TS_SUCCESS == TSHttpTxnParentSelectionUrlSet(txnp, hdr_buf, ps_loc)) { + DEBUG_LOG("Set Parent Selection URL to cache_key_url: %s", cache_key_url); + TSHandleMLocRelease(hdr_buf, TS_NULL_MLOC, ps_loc); + } + } + } - // Optionally set the parent_selection_url to the cache_key url or path - if (nullptr != pc && PS_DEFAULT != pc->ps_mode) { - TSMLoc ps_loc = nullptr; - - if (PS_CACHEKEY_URL == pc->ps_mode) { - const char *start = cache_key_url; - const char *end = cache_key_url + cache_key_url_length; - if (TS_SUCCESS == TSUrlCreate(hdr_bufp, &ps_loc) && - TS_PARSE_DONE == TSUrlParse(hdr_bufp, ps_loc, &start, end) && // This should always succeed. - TS_SUCCESS == TSHttpTxnParentSelectionUrlSet(txnp, hdr_bufp, ps_loc)) { - DEBUG_LOG("Set Parent Selection URL to cache_key_url: %s", cache_key_url); - TSHandleMLocRelease(hdr_bufp, TS_NULL_MLOC, ps_loc); + // optionally consider an X-CRR-IMS header + if (pc->consider_ims_header) { + TSMLoc const imsloc = TSMimeHdrFieldFind(hdr_buf, hdr_loc, X_IMS_HEADER.data(), X_IMS_HEADER.size()); + if (TS_NULL_MLOC != imsloc) { + time_t const itime = TSMimeHdrFieldValueDateGet(hdr_buf, hdr_loc, imsloc); + TSHandleMLocRelease(hdr_buf, hdr_loc, imsloc); + if (0 < itime) { + txn_state->ims_time = itime; + } } } } diff --git a/plugins/xdebug/xdebug.cc b/plugins/xdebug/xdebug.cc index 42e83d8def6..3a2e7a16998 100644 --- a/plugins/xdebug/xdebug.cc +++ b/plugins/xdebug/xdebug.cc @@ -35,13 +35,15 @@ static struct { } xDebugHeader = {nullptr, 0}; enum { - XHEADER_X_CACHE_KEY = 0x0004u, - XHEADER_X_MILESTONES = 0x0008u, - XHEADER_X_CACHE = 0x0010u, - XHEADER_X_GENERATION = 0x0020u, - XHEADER_X_TRANSACTION_ID = 0x0040u, - XHEADER_X_DUMP_HEADERS = 0x0080u, - XHEADER_X_REMAP = 0x0100u, + XHEADER_X_CACHE_KEY = 1u << 2, + XHEADER_X_MILESTONES = 1u << 3, + XHEADER_X_CACHE = 1u << 4, + XHEADER_X_GENERATION = 1u << 5, + XHEADER_X_TRANSACTION_ID = 1u << 6, + XHEADER_X_DUMP_HEADERS = 1u << 7, + XHEADER_X_REMAP = 1u << 8, + XHEADER_X_PROBE_HEADERS = 1u << 9, + XHEADER_X_PSELECT_KEY = 1u << 10, }; static int XArgIndex = 0; @@ -357,6 +359,53 @@ log_headers(TSHttpTxn txn, TSMBuffer bufp, TSMLoc hdr_loc, const char *msg_type) TSDebug(DEBUG_TAG_LOG_HEADERS, "%s", ss.str().c_str()); } +static void +InjectParentSelectionKeyHeader(TSHttpTxn txn, TSMBuffer buffer, TSMLoc hdr) +{ + TSMLoc url = TS_NULL_MLOC; + TSMLoc dst = TS_NULL_MLOC; + + struct { + char *ptr; + int len; + } strval = {nullptr, 0}; + + TSDebug("xdebug", "attempting to inject X-ParentSelection-Key header"); + + if (TSUrlCreate(buffer, &url) != TS_SUCCESS) { + goto done; + } + + if (TSHttpTxnParentSelectionUrlGet(txn, buffer, url) != TS_SUCCESS) { + goto done; + } + + strval.ptr = TSUrlStringGet(buffer, url, &strval.len); + if (strval.ptr == nullptr || strval.len == 0) { + goto done; + } + + // Create a new response header field. + dst = FindOrMakeHdrField(buffer, hdr, "X-ParentSelection-Key", lengthof("X-ParentSelection-Key")); + if (dst == TS_NULL_MLOC) { + goto done; + } + + // Now copy the parent selection lookup URL into the response header. + TSReleaseAssert(TSMimeHdrFieldValueStringInsert(buffer, hdr, dst, 0 /* idx */, strval.ptr, strval.len) == TS_SUCCESS); + +done: + if (dst != TS_NULL_MLOC) { + TSHandleMLocRelease(buffer, hdr, dst); + } + + if (url != TS_NULL_MLOC) { + TSHandleMLocRelease(buffer, TS_NULL_MLOC, url); + } + + TSfree(strval.ptr); +} + static int XInjectResponseHeaders(TSCont /* contp */, TSEvent event, void *edata) { @@ -404,6 +453,10 @@ XInjectResponseHeaders(TSCont /* contp */, TSEvent event, void *edata) InjectRemapHeader(txn, buffer, hdr); } + if (xheaders & XHEADER_X_PSELECT_KEY) { + InjectParentSelectionKeyHeader(txn, buffer, hdr); + } + done: TSHttpTxnReenable(txn, TS_EVENT_HTTP_CONTINUE); return TS_EVENT_NONE; @@ -486,7 +539,7 @@ XScanRequestHeaders(TSCont /* contp */, TSEvent event, void *edata) TSHttpTxnReenable(txn, TS_EVENT_HTTP_CONTINUE); return TS_EVENT_NONE; }; - TSHttpTxnHookAdd(txn, TS_HTTP_SEND_REQUEST_HDR_HOOK, TSContCreate(send_req_dump, nullptr)); + TSHttpTxnHookAdd(txn, TS_HTTP_SEND_REQUEST_HDR_HOOK, TSContCreate(send_req_dump, nullptr)) // dump on server response auto read_resp_dump = [](TSCont /* contp */, TSEvent event, void *edata) -> int { @@ -501,6 +554,8 @@ XScanRequestHeaders(TSCont /* contp */, TSEvent event, void *edata) }; TSHttpTxnHookAdd(txn, TS_HTTP_READ_RESPONSE_HDR_HOOK, TSContCreate(read_resp_dump, nullptr)); + } else if (header_field_eq("x-parentselection-key", value, vsize)) { + xheaders |= XHEADER_X_PSELECT_KEY; } else { TSDebug("xdebug", "ignoring unrecognized debug tag '%.*s'", vsize, value); } diff --git a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py new file mode 100644 index 00000000000..49aaf6ed22a --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py @@ -0,0 +1,340 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +Basic cache_range_requests plugin test +''' + +## Test description: +# Preload the cache with the entire asset to be range requested. +# Reload remap rule with cache_range_requests plugin +# Request content through the cache_range_requests plugin + +Test.SkipUnless( + Condition.PluginExists('cache_range_requests.so'), + Condition.PluginExists('xdebug.so'), +) +Test.ContinueOnFail = False +Test.testName = "cache_range_requests" + +# Define and configure ATS +ts = Test.MakeATSProcess("ts", command="traffic_server") + +# Define and configure origin server +server = Test.MakeOriginServer("server", lookup_key="{%uuid}") + +# default root +req_chk = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "uuid: none\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_chk = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +server.addResponse("sessionlog.json", req_chk, res_chk) + +body = "lets go surfin now" + +req_full = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "uuid: full\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_full = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Cache-Control: max-age=500\r\n" + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_full, res_full) + +block_bytes = 7 +bodylen = len(body) + +inner_str = "7-15" + +req_inner = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes={}\r\n".format(inner_str) + + "uuid: inner\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_inner = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + "Cache-Control: max-age=500\r\n" + + "Content-Range: bytes {0}/{1}\r\n".format(inner_str, bodylen) + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body[7:15] +} + +server.addResponse("sessionlog.json", req_inner, res_inner) + +frange_str = "0-" + +req_frange = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes={}\r\n".format(frange_str) + + "uuid: frange\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_frange = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + "Cache-Control: max-age=500\r\n" + + "Content-Range: bytes 0-{0}/{0}\r\n".format(bodylen) + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_frange, res_frange) + +last_str = "-5" + +req_last = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes={}\r\n".format(last_str) + + "uuid: last\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_last = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + "Cache-Control: max-age=200\r\n" + + "Content-Range: bytes {0}-{1}/{1}\r\n".format(bodylen - 5, bodylen) + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body[-5:] +} + +server.addResponse("sessionlog.json", req_last, res_last) + +pselect_str = "1-10" + +req_pselect = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: parentselect\r\n" + + "Accept: */*\r\n" + + "Range: bytes={}\r\n".format(pselect_str) + + "uuid: pselect\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_pselect = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + "Cache-Control: max-age=200\r\n" + + "Content-Range: bytes {}/19\r\n".format(pselect_str) + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body[1:10] +} + +server.addResponse("sessionlog.json", req_pselect, res_pselect) + +# cache range requests plugin remap +ts.Disk.remap_config.AddLines([ + 'map http://www.example.com http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cache_range_requests.so', + 'map http://parentselect http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cache_range_requests.so @pparam=ps_mode:cache_key_url', +]) + +# cache debug +ts.Disk.plugin_config.AddLine('xdebug.so') + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'cache_range_requests', + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, +}) + +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x localhost:{} -H "x-debug: x-cache"'.format(ts.Variables.port) + +# 0 Test - Fetch whole asset into cache +tr = Test.AddTestRun("full asset cache miss bypass") +ps = tr.Processes.Default +ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) +ps.StartBefore(Test.Processes.ts, ready=When.PortOpen(ts.Variables.port)) +ps.Command = curl_and_args + ' http://www.example.com/path -H "uuid: full"' +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/full.stdout.gold" +ps.Streams.stderr = "gold/full.stderr.gold" +tr.StillRunningAfter = ts + +# test inner range +# 1 Test - Fetch range into cache +tr = Test.AddTestRun("inner range cache miss") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {} -H "uuid: inner"'.format(inner_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/inner.stdout.gold" +ps.Streams.stderr = "gold/inner.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss") +tr.StillRunningAfter = ts + +# 2 Test - Fetch from cache +tr = Test.AddTestRun("inner range cache hit") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {}'.format(inner_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/inner.stdout.gold" +ps.Streams.stderr = "gold/inner.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit", "expected cache hit") +tr.StillRunningAfter = ts + +# full range + +# 3 Test - 0- request +tr = Test.AddTestRun("0- request miss") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {} -H "uuid: frange"'.format(frange_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/frange.stdout.gold" +ps.Streams.stderr = "gold/frange.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss") +tr.StillRunningAfter = ts + +# 4 Test - 0- request +tr = Test.AddTestRun("0- request hit") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {}'.format(frange_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/frange.stdout.gold" +ps.Streams.stderr = "gold/frange.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit", "expected cache hit") +tr.StillRunningAfter = ts + +# end range + +# 5 Test - -5 request miss +tr = Test.AddTestRun("-5 request miss") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {} -H "uuid: last"'.format(last_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/last.stdout.gold" +ps.Streams.stderr = "gold/last.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss") +tr.StillRunningAfter = ts + +# 6 Test - -5 request hit +tr = Test.AddTestRun("-5 request hit") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {}'.format(last_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/last.stdout.gold" +ps.Streams.stderr = "gold/last.stderr.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit", "expected cache hit") +tr.StillRunningAfter = ts + +# Ensure 404's aren't getting cached + +# 7 Test - 404 +tr = Test.AddTestRun("404 request 1st") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/404 -r 0-' +ps.Streams.stdout = "gold/404.stdout.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss") +tr.StillRunningAfter = ts + +# 8 Test - 404 +tr = Test.AddTestRun("404 request 2nd") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/404 -r 0-' +ps.Streams.stdout = "gold/404.stdout.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss") +tr.StillRunningAfter = ts + +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x localhost:{} -H "x-debug: x-parentselection-key"'.format(ts.Variables.port) + +# 9 Test - cache_key_url request +tr = Test.AddTestRun("cache_key_url request") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://parentselect/path -r {} -H "uuid: pselect"'.format(pselect_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/pselect.stdout.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression( + "X-ParentSelection-Key: .*-bytes=", + "expected bytes in parent selection key", +) +tr.StillRunningAfter = ts +tr.StillRunningAfter = server + +# 10 Test - non cache_key_url request ... no X-ParentSelectionKey +tr = Test.AddTestRun("non cache_key_url request") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r {} -H "uuid: inner"'.format(inner_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/pselect_none.stdout.gold" +tr.StillRunningAfter = ts +tr.StillRunningAfter = server diff --git a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_cachekey.test.py b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_cachekey.test.py new file mode 100644 index 00000000000..2622694a04e --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_cachekey.test.py @@ -0,0 +1,198 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import time + +Test.Summary = ''' +cache_range_requests with cachekey +''' + +## Test description: +# Preload the cache with the entire asset to be range requested. +# Reload remap rule with cache_range_requests plugin +# Request content through the cache_range_requests plugin + +Test.SkipUnless( + Condition.PluginExists('cache_range_requests.so'), + Condition.PluginExists('cachekey.so'), + Condition.PluginExists('xdebug.so'), +) +Test.ContinueOnFail = False +Test.testName = "cache_range_requests_cachekey" + +# Define and configure ATS, enable traffic_ctl config reload +ts = Test.MakeATSProcess("ts", command="traffic_server") + +# Define and configure origin server +server = Test.MakeOriginServer("server", lookup_key="{%uuid}") + +# default root +req_chk = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "uuid: none\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_chk = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +server.addResponse("sessionlog.json", req_chk, res_chk) + +body = "lets go surfin now" +bodylen = len(body) + +# this request should work +req_full = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "uuid: full\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_full = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + 'Etag: "foo"\r\n' + + "Cache-Control: public, max-age=500\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_full, res_full) + +# this request should work +req_good = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes=0-\r\n" + + "uuid: range_full\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_good = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + 'Etag: "foo"\r\n' + + "Cache-Control: public, max-age=500\r\n" + + "Content-Range: bytes 0-{0}/{0}\r\n".format(bodylen) + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_good, res_good) + +# this request should fail with a cache_range_requests asset +req_fail = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.fail.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes=0-\r\n" + + "uuid: range_fail\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_fail = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + 'Etag: "foo"\r\n' + + "Cache-Control: public, max-age=500\r\n" + + "Content-Range: bytes 0-{0}/{0}\r\n".format(bodylen) + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_fail, res_fail) + +# cache range requests plugin remap, working config +ts.Disk.remap_config.AddLine( + 'map http://www.example.com http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cachekey.so @pparam=--include-headers=Range' + + ' @plugin=cache_range_requests.so @pparam=--no-modify-cachekey', +) + +# improperly configured cache_range_requests with cachekey +ts.Disk.remap_config.AddLine( + 'map http://www.fail.com http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cachekey.so @pparam=--static-prefix=foo' + ' @plugin=cache_range_requests.so', +) + +# cache debug +ts.Disk.plugin_config.AddLine('xdebug.so') + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'cache_range_requests', + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, +}) + +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x localhost:{} -H "x-debug: x-cache"'.format(ts.Variables.port) + +# 0 Test - Fetch full asset into cache (ensure cold) +tr = Test.AddTestRun("full asset fetch") +ps = tr.Processes.Default +ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) +ps.StartBefore(Test.Processes.ts, ready=When.PortOpen(ts.Variables.port)) +ps.Command = curl_and_args + ' http://www.example.com/path -H "uuid: full"' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss for load") +tr.StillRunningAfter = ts + +# 1 Test - Fetch whole asset into cache via range request (ensure cold) +tr = Test.AddTestRun("0- asset fetch") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r 0- -H "uuid: range_full"' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss for load") +tr.StillRunningAfter = ts + +# 2 Test - Ensure assert happens instead of possible cache poisoning. +tr = Test.AddTestRun("Attempt poisoning") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.fail.com/path -r 0- -H "uuid: range_fail"' +ps.ReturnCode = 0 +tr.StillRunningAfter = ts + +ts.Disk.diags_log.Content = Testers.ContainsExpression("ERROR", "error condition hit") +ts.Disk.diags_log.Content = Testers.ContainsExpression("failed to change the cache url", "ensure failure for misconfiguration") +ts.Disk.diags_log.Content = Testers.ContainsExpression("Disabling cache for this transaction to avoid cache poisoning", "ensure transaction caching disabled") diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/404.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/404.stdout.gold new file mode 100644 index 00000000000..f188050f072 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/404.stdout.gold @@ -0,0 +1,8 @@ +HTTP/1.1 404 Not Found +Server: `` +Date: `` +Age: 0 +Transfer-Encoding: chunked +Proxy-Connection: `` +X-Cache: miss +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stderr.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stderr.gold new file mode 100644 index 00000000000..24ad29c1a82 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stderr.gold @@ -0,0 +1 @@ +lets go surfin now`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold new file mode 100644 index 00000000000..6dae639a9c3 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold @@ -0,0 +1,11 @@ +HTTP/1.1 206 Partial Content +Accept-Ranges: bytes +Cache-Control: max-age=500 +Content-Range: bytes 0-18/18 +Etag: "772102f4-56f4bc1e6d417" +Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT +Content-Length: 18 +Date: `` +Server: `` +X-Cache: `` +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stderr.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stderr.gold new file mode 100644 index 00000000000..24ad29c1a82 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stderr.gold @@ -0,0 +1 @@ +lets go surfin now`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold new file mode 100644 index 00000000000..a485d263577 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold @@ -0,0 +1,9 @@ +HTTP/1.1 200 OK +Cache-Control: max-age=500 +Etag: "772102f4-56f4bc1e6d417" +Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT +Content-Length: 18 +Date: `` +Server: `` +X-Cache: miss +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stderr.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stderr.gold new file mode 100644 index 00000000000..1b0ef57a044 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stderr.gold @@ -0,0 +1 @@ + surfin`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold new file mode 100644 index 00000000000..0628bfad209 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold @@ -0,0 +1,11 @@ +HTTP/1.1 206 Partial Content +Accept-Ranges: bytes +Cache-Control: max-age=500 +Content-Range: bytes 7-15/18 +Etag: "772102f4-56f4bc1e6d417" +Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT +Content-Length: 7 +Date: `` +Server: `` +X-Cache: `` +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stderr.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stderr.gold new file mode 100644 index 00000000000..9e4c8c43d82 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stderr.gold @@ -0,0 +1 @@ +n now`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold new file mode 100644 index 00000000000..5e2da0005c8 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold @@ -0,0 +1,11 @@ +HTTP/1.1 206 Partial Content +Accept-Ranges: bytes +Cache-Control: max-age=500 +Content-Range: bytes 12-18/18 +Etag: "772102f4-56f4bc1e6d417" +Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT +Content-Length: 5 +Date: `` +Server: `` +X-Cache: `` +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold new file mode 100644 index 00000000000..11902d7ec3d --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold @@ -0,0 +1,11 @@ +HTTP/1.1 206 Partial Content +Accept-Ranges: bytes +Cache-Control: max-age=500 +Content-Range: bytes 1-10/18 +Etag: "772102f4-56f4bc1e6d417" +Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT +Content-Length: 9 +Date: `` +Server: `` +X-ParentSelection: `` +`` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect_none.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect_none.stdout.gold new file mode 100644 index 00000000000..dc9b1559778 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect_none.stdout.gold @@ -0,0 +1,10 @@ +HTTP/1.1 206 Partial Content +Accept-Ranges: bytes +Cache-Control: `` +Content-Range: `` +Etag: `` +Last-Modified: `` +Content-Length: `` +Date: `` +Server: `` +``