diff --git a/doc/admin-guide/plugins/cache_range_requests.en.rst b/doc/admin-guide/plugins/cache_range_requests.en.rst new file mode 100644 index 00000000000..5d33f7c0ac7 --- /dev/null +++ b/doc/admin-guide/plugins/cache_range_requests.en.rst @@ -0,0 +1,168 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + + +.. include:: ../../common.defs + +.. _admin-plugins-cache-range-requests: + + +Cache Range Requests Plugin +*************************** + +Description +=========== + +Most origin servers support HTTP/1.1 range requests (rfc 7233). +ATS internally handles range request caching in one of 2 ways: + +* Don't cache range requests. +* Only server range requests from a wholly cached object. + +This plugin allows you to remap individual range requests so that they +are stored as individual objects in the ATS cache when subsequent range +requests are likely to use the same range. This spreads range requests +over multiple stripes thereby reducing I/O wait and system load averages. + +:program:`cache_range_requests` reads the range request header byte range +value and then creates a new ``cache key URL`` using the original request +url with the range value appended to it. The range header is removed +where appropriate from the requests and the origin server response code +is changed from a 206 to a 200 to insure that the object is written to +cache using the new cache key url. The response code sent to the client +will be changed back to a 206 and all requests to the origin server will +contain the range header so that the correct response is received. + +The :program:`cache_range_requests` plugin by itself has no logic to +efficiently manage overlapping ranges. It is best to use this plugin +in conjunction with a smart client that only requests predetermined +non overlapping cache ranges (request blocking) or as a helper for the +:program:`slice` plugin. + +Only requests which contain the ``Range: =`` GET header +will be served by the :program:`cache_range_requests` plugin. + +If/when ATS implements partial object caching this plugin will +become deprecated. + +*NOTE* Given a multi range request the :program:`cache_range_requests` +only processes the first range and ignores the rest. + +How to run the plugin +===================== + +The plugin can run as a global plugin (a single global instance configured +using :file:`plugin.config`) or as per-remap plugin (a separate instance +configured per remap rule in :file:`remap.config`). + +Global instance +--------------- + +.. code:: + + $ cat plugin.config + cache_range_request.so + + +Per-remap instance +------------------ + +.. code:: + + $cat remap.config + map http://www.example.com http://www.origin.com \ + @plugin=cache_range_requests.so + + +If both global and per-remap instance are used the per-remap configuration +would take precedence (per-remap configuration would be applied and the +global configuration ignored). + +Plugin options +============== + + +Parent Selection as Cache Key +----------------------------- + +.. option:: --ps-cachekey +.. option:: -p + +Without this option parent selection is based solely on the hash of a +URL Path a URL is requested from the same upstream parent cache listed +in parent.config + + +With this option parent selection is based on the full ``cache key URL`` +which includes information about the partial content range. In this mode, +all requests (include partial content) will use consistent hashing method +for parent selection. + + +X-CRR-IMS header support +------------------------ + +.. option:: --consider-ims +.. option:: -c + +To support slice plugin self healing an option to force revalidation +after cache lookup complete was added. This option is triggered by a +special header: + +.. code:: + + X-CRR-IMS: Tue, 19 Nov 2019 13:26:45 GMT + +When this header is provided and a `cache hit fresh` is encoutered the +``Date`` header of the object in cache is compared to this header date +value. If the cache date is *less* than this IMS date then the object +is marked as STALE and an appropriate If-Modified-Since or If-Match +request along with this X-CRR-IMS header is passed up to the parent. + +In order for this to properly work in a CDN each cache in the +chain *SHOULD* also contain a remap rule with the +:program:`cache_range_requests` plugin with this option set. + + +Configuration examples +====================== + +Global plugin +------------- + +.. code:: + + cache_range_requests.so --ps-cachekey --consider-ims + +or + +.. code:: + + cache_range_requests.so -p -c + +Remap plugin +------------ + +.. code:: + + map http://ats http://parent @plugin=cache_range_requests.so @pparam=--ps-cachekey @pparam=--consider-ims + +or + +.. code:: + + map http://ats http://parent @plugin=cache_range_requests.so @pparam=-p @pparam=-c diff --git a/doc/admin-guide/plugins/index.en.rst b/doc/admin-guide/plugins/index.en.rst index 48e9825bd0d..48a3d5bdf64 100644 --- a/doc/admin-guide/plugins/index.en.rst +++ b/doc/admin-guide/plugins/index.en.rst @@ -50,6 +50,7 @@ Plugins that are considered stable are installed by default in |TS| releases. Background Fetch Cache Key Manipulation Cache Promotion Policies + Cache Range Requests Combo Handler Configuration Remap Cookie Remap @@ -63,7 +64,6 @@ Plugins that are considered stable are installed by default in |TS| releases. Regex Remap Regex Revalidate Remap Purge - Slice Stats over HTTP TCPInfo XDebug @@ -83,6 +83,9 @@ Plugins that are considered stable are installed by default in |TS| releases. :doc:`Cache Promotion Policies ` Allows for control over which assets should be written to cache, or not. +:doc:`Cache Range Requests ` + Cache ranges by adding the range request header to the cache key. + :doc:`Combo Handler ` Provides an intelligent way to combine multiple URLs into a single URL, and have Apache Traffic Server combine the components into one response. @@ -159,7 +162,7 @@ directory of the |TS| source tree. Experimental plugins can be compiled by passi Multiplexer MySQL Remap Signed URLs - Slicer + Slice SSL Headers SSL Session Reuse System Statistics @@ -223,10 +226,10 @@ directory of the |TS| source tree. Experimental plugins can be compiled by passi :doc:`Signed URLs ` Adds support for verifying URL signatures for incoming requests to either deny or redirect access. -:doc:`Slicer ` +:doc:`Slice ` Slice full file or range based requests into deterministic chunks, allowing large files to be - spread across multiple cache stripes. Allows range requests to be satisfied by stitching these - chunks together. + spread across multiple cache stripes. Allows arbitrary range requests to be satisfied by stitching + these chunks together. :doc:`SSL Session Reuse ` Coordinates Session ID and ticket based TLS session resumption between a group of ATS machines. diff --git a/plugins/cache_range_requests/README b/plugins/cache_range_requests/README deleted file mode 100644 index 5495f687d0f..00000000000 --- a/plugins/cache_range_requests/README +++ /dev/null @@ -1,45 +0,0 @@ - -Thousands of range requests for a very large object in the traffic server cache -are likely to increase system load averages due to I/O wait as objects are stored -on a single stripe or disk drive. - -This plugin allows you to remap individual range requests so that they are stored -as individual objects in the ATS cache when subsequent range requests are likely -to use the same range. This spreads range requests over multiple stripes thereby -reducing I/O wait and system load averages. - -This plugin reads the range request header byte range value and then creates a -new cache key url using the original request url with the range value appended -to it. The range header is removed where appropriate from the requests and the -origin server response code is changed from a 206 to a 200 to insure that the -object is written to cache using the new cache key url. The response code sent -to the client will be changed back to a 206 and all requests to the origin server -will contain the range header so that the correct response is received. - -Configuration: - - Add @plugin=cache_range_requests.so to your remap.config rules. - - Or for a global plugin where all range requests are processed, - Add cache_range_requests.so to the plugin.config - -Parent Selection Mode (consistent-hash only): - - default: Parent selection is based solely on the hash of a URL Path - In this mode, all partial content of a URL is requested from the same - upstream parent cache listed in parent.config - - cache_key_url: Parent selection is based on the full cache_key_url which - includes information about the partial content range. - In this mode, all requests (include partial content) will use - consistent hashing method for parent selection. - - To enable cache_key_url parent select mode, the following param must be set: - - Global Plugin (plugin.config): - - cache_range_requests.so ps_mode:cache_key_url - - Remap Plugin (remap.config): - - @plugin=cache_range_requests.so @pparam=ps_mode:cache_key_url diff --git a/plugins/cache_range_requests/README.md b/plugins/cache_range_requests/README.md new file mode 100644 index 00000000000..ef567a70944 --- /dev/null +++ b/plugins/cache_range_requests/README.md @@ -0,0 +1,80 @@ + +Thousands of range requests for a very large object in the traffic server +cache are likely to increase system load averages due to I/O wait as +objects are stored on a single stripe or disk drive. + +This plugin allows you to remap individual range requests so that they +are stored as individual objects in the ATS cache when subsequent range +requests are likely to use the same range. This spreads range requests +over multiple stripes thereby reducing I/O wait and system load averages. + +This plugin reads the range request header byte range value and then +creates a new cache key url using the original request url with the range +value appended to it. The range header is removed where appropriate +from the requests and the origin server response code is changed from +a 206 to a 200 to insure that the object is written to cache using the +new cache key url. The response code sent to the client will be changed +back to a 206 and all requests to the origin server will contain the +range header so that the correct response is received. + +Configuration: + + Add @plugin=cache_range_requests.so to your remap.config rules. + + Or for a global plugin where all range requests are processed, + Add cache_range_requests.so to the plugin.config + +Parent Selection Mode (consistent-hash only): + + default: Parent selection is based solely on the hash of a URL Path + In this mode, all partial content of a URL is requested + from the same upstream parent cache listed in parent.config + + Cache_key_url: Parent selection is based on the full cache_key_url + which includes information about the partial content + range. In this mode, all requests (include partial + content) will use consistent hashing method for + parent selection. + + To enable cache_key_url parent select mode, the following param must be set: + + Global Plugin (plugin.config): + + cache_range_requests.so -p + cache_range_requests.so --ps-cachekey + + Remap Plugin (remap.config): + + @plugin=cache_range_requests.so @pparam=--ps-cachekey + @plugin=cache_range_requests.so @pparam=-p + +X-CRR-IMS header support + + To support slice plugin self healing an option to force + revalidation after cache lookup complete was added. This option + is triggered by a special header: + + This optional header looks like: + + X-CRR-IMS: Tue, 19 Nov 2019 13:26:45 GMT + + If the cache lookup was a cache hit and the cache header date + is *less* than this header value then the cache state is switched + from FRESH to STALE which results in If-Modified-Since or + If-Match request being passed to the parent. + + In order for this option to be enabled the the following parameter + must be set: + + Global Plugin (plugin.config): + + cache_range_requests.so --consider-ims + cache_range_requests.so -c + + Remap Plugin (remap.config): + + @plugin=cache_range_requests.so @pparam=--consider-ims + @plugin=cache_range_requests.so @pparam=-c + + Consider using the header_rewrite plugin to protect the parent + from using this option as an attack vector against an origin. diff --git a/plugins/cache_range_requests/cache_range_requests.cc b/plugins/cache_range_requests/cache_range_requests.cc index 60f512caf7c..13381d248e9 100644 --- a/plugins/cache_range_requests/cache_range_requests.cc +++ b/plugins/cache_range_requests/cache_range_requests.cc @@ -26,68 +26,107 @@ * requests. */ -#include -#include #include "ts/ts.h" #include "ts/remap.h" +#include +#include +#include +#include +#include + #define PLUGIN_NAME "cache_range_requests" #define DEBUG_LOG(fmt, ...) TSDebug(PLUGIN_NAME, "[%s:%d] %s(): " fmt, __FILE__, __LINE__, __func__, ##__VA_ARGS__) #define ERROR_LOG(fmt, ...) TSError("[%s:%d] %s(): " fmt, __FILE__, __LINE__, __func__, ##__VA_ARGS__) +namespace +{ typedef enum parent_select_mode { PS_DEFAULT, // Default ATS parent selection mode PS_CACHEKEY_URL, // Set parent selection url to cache_key url } parent_select_mode_t; struct pluginconfig { - parent_select_mode_t ps_mode; + parent_select_mode_t ps_mode{PS_DEFAULT}; + bool consider_ims_header{false}; }; struct txndata { - char *range_value; + std::string range_value; + time_t ims_time{0}; }; -static int handle_read_request_header(TSCont, TSEvent, void *); -static void range_header_check(TSHttpTxn txnp, struct pluginconfig *pc); -static void handle_send_origin_request(TSCont, TSHttpTxn, struct txndata *); -static void handle_client_send_response(TSHttpTxn, struct txndata *); -static void handle_server_read_response(TSHttpTxn, struct txndata *); -static int remove_header(TSMBuffer, TSMLoc, const char *, int); -static bool set_header(TSMBuffer, TSMLoc, const char *, int, const char *, int); -static int transaction_handler(TSCont, TSEvent, void *); -static struct pluginconfig *create_pluginconfig(int argc, const char *argv[]); -static void delete_pluginconfig(struct pluginconfig *); +// Header for optional revalidation +constexpr std::string_view X_IMS_HEADER = {"X-CRR-IMS"}; // pluginconfig struct (global plugin only) -static struct pluginconfig *gPluginConfig = nullptr; +pluginconfig *gPluginConfig = {nullptr}; + +int handle_read_request_header(TSCont, TSEvent, void *); +void range_header_check(TSHttpTxn, pluginconfig *const); +void handle_send_origin_request(TSCont, TSHttpTxn, txndata *const); +void handle_client_send_response(TSHttpTxn, txndata *const); +void handle_server_read_response(TSHttpTxn, txndata *const); +int remove_header(TSMBuffer, TSMLoc, const char *, int); +bool set_header(TSMBuffer, TSMLoc, const char *, int, const char *, int); +int transaction_handler(TSCont, TSEvent, void *); +struct pluginconfig *create_pluginconfig(int argc, char *const argv[]); +void delete_pluginconfig(pluginconfig *const); /** * Creates pluginconfig data structure * Sets default parent url selection mode * Walk plugin argument list and updates config */ -static struct pluginconfig * -create_pluginconfig(int argc, const char *argv[]) +pluginconfig * +create_pluginconfig(int argc, char *const argv[]) { - struct pluginconfig *pc = nullptr; + DEBUG_LOG("Number of arguments: %d", argc); + for (int index = 0; index < argc; ++index) { + DEBUG_LOG("args[%d] = %s", index, argv[index]); + } - pc = static_cast(TSmalloc(sizeof(struct pluginconfig))); + pluginconfig *const pc = new pluginconfig; if (nullptr == pc) { ERROR_LOG("Can't allocate pluginconfig"); return nullptr; } - // Plugin uses default ATS selection (hash of URL path) - pc->ps_mode = PS_DEFAULT; + static const struct option longopts[] = { + {const_cast("ps-cachekey"), no_argument, nullptr, 'p'}, + {const_cast("consider-ims"), no_argument, nullptr, 'c'}, + {nullptr, 0, nullptr, 0}, + }; - // Walk through param list. - for (int c = 0; c < argc; c++) { - if (strcmp("ps_mode:cache_key_url", argv[c]) == 0) { - pc->ps_mode = PS_CACHEKEY_URL; + // getopt assumes args start at '1' + ++argc; + --argv; + + for (;;) { + int const opt = getopt_long(argc, argv, "", longopts, nullptr); + if (-1 == opt) { break; } + + switch (opt) { + case 'p': { + DEBUG_LOG("Plugin modifies parent selection key"); + pc->ps_mode = PS_CACHEKEY_URL; + } break; + case 'c': { + DEBUG_LOG("Plugin considers the '%.*s' header", (int)X_IMS_HEADER.size(), X_IMS_HEADER.data()); + pc->consider_ims_header = true; + } break; + default: { + } break; + } + } + + // Backwards compatibility + if (optind < argc && 0 == strcmp("ps_mode:cache_key_url", argv[optind])) { + DEBUG_LOG("Plugin modifies parent selection key (deprecated)"); + pc->ps_mode = PS_CACHEKEY_URL; } return pc; @@ -96,21 +135,19 @@ create_pluginconfig(int argc, const char *argv[]) /** * Destroy pluginconfig data structure. */ -static void -delete_pluginconfig(struct pluginconfig *pc) +void +delete_pluginconfig(pluginconfig *const pc) { if (nullptr != pc) { DEBUG_LOG("Delete struct pluginconfig"); - TSfree(pc); - pc = nullptr; + delete pc; } } /** * Entry point when used as a global plugin. - * */ -static int +int handle_read_request_header(TSCont txn_contp, TSEvent event, void *edata) { TSHttpTxn txnp = static_cast(edata); @@ -131,35 +168,36 @@ handle_read_request_header(TSCont txn_contp, TSEvent event, void *edata) * 3. Schedules TS_HTTP_SEND_REQUEST_HDR_HOOK, TS_HTTP_SEND_RESPONSE_HDR_HOOK, * and TS_HTTP_TXN_CLOSE_HOOK for further processing. */ -static void -range_header_check(TSHttpTxn txnp, struct pluginconfig *pc) +void +range_header_check(TSHttpTxn txnp, pluginconfig *const pc) { char cache_key_url[8192] = {0}; char *req_url; int length, url_length, cache_key_url_length; - struct txndata *txn_state; - TSMBuffer hdr_bufp; - TSMLoc req_hdrs = nullptr; - TSMLoc loc = nullptr; + txndata *txn_state; + TSMBuffer hdr_buf; + TSMLoc hdr_loc = nullptr; + TSMLoc loc = nullptr; TSCont txn_contp; - if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &hdr_bufp, &req_hdrs)) { - loc = TSMimeHdrFieldFind(hdr_bufp, req_hdrs, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE); + if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &hdr_buf, &hdr_loc)) { + loc = TSMimeHdrFieldFind(hdr_buf, hdr_loc, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE); if (TS_NULL_MLOC != loc) { - const char *hdr_value = TSMimeHdrFieldValueStringGet(hdr_bufp, req_hdrs, loc, 0, &length); + const char *hdr_value = TSMimeHdrFieldValueStringGet(hdr_buf, hdr_loc, loc, 0, &length); + TSHandleMLocRelease(hdr_buf, hdr_loc, loc); + if (!hdr_value || length <= 0) { DEBUG_LOG("Not a range request."); } else { if (nullptr == (txn_contp = TSContCreate(static_cast(transaction_handler), nullptr))) { ERROR_LOG("failed to create the transaction handler continuation."); } else { - txn_state = static_cast(TSmalloc(sizeof(struct txndata))); - txn_state->range_value = TSstrndup(hdr_value, length); - DEBUG_LOG("length: %d, txn_state->range_value: %s", length, txn_state->range_value); - txn_state->range_value[length] = '\0'; // workaround for bug in core - + txn_state = new txndata; + std::string &rv = txn_state->range_value; + rv.assign(hdr_value, length); + DEBUG_LOG("length: %d, txn_state->range_value: %s", length, rv.c_str()); req_url = TSHttpTxnEffectiveUrlStringGet(txnp, &url_length); - cache_key_url_length = snprintf(cache_key_url, 8192, "%s-%s", req_url, txn_state->range_value); + cache_key_url_length = snprintf(cache_key_url, 8192, "%s-%s", req_url, rv.c_str()); DEBUG_LOG("Rewriting cache URL for %s to %s", req_url, cache_key_url); if (req_url != nullptr) { TSfree(req_url); @@ -170,24 +208,38 @@ range_header_check(TSHttpTxn txnp, struct pluginconfig *pc) DEBUG_LOG("failed to change the cache url to %s.", cache_key_url); } - // Optionally set the parent_selection_url to the cache_key url or path - if (nullptr != pc && PS_DEFAULT != pc->ps_mode) { - TSMLoc ps_loc = nullptr; - - if (PS_CACHEKEY_URL == pc->ps_mode) { - const char *start = cache_key_url; - const char *end = cache_key_url + cache_key_url_length; - if (TS_SUCCESS == TSUrlCreate(hdr_bufp, &ps_loc) && - TS_PARSE_DONE == TSUrlParse(hdr_bufp, ps_loc, &start, end) && // This should always succeed. - TS_SUCCESS == TSHttpTxnParentSelectionUrlSet(txnp, hdr_bufp, ps_loc)) { - DEBUG_LOG("Set Parent Selection URL to cache_key_url: %s", cache_key_url); - TSHandleMLocRelease(hdr_bufp, TS_NULL_MLOC, ps_loc); + if (nullptr != pc) { + // Optionally set the parent_selection_url to the cache_key url or path + if (PS_DEFAULT != pc->ps_mode) { + TSMLoc ps_loc = nullptr; + + if (PS_CACHEKEY_URL == pc->ps_mode) { + const char *start = cache_key_url; + const char *end = cache_key_url + cache_key_url_length; + if (TS_SUCCESS == TSUrlCreate(hdr_buf, &ps_loc) && + TS_PARSE_DONE == TSUrlParse(hdr_buf, ps_loc, &start, end) && // This should always succeed. + TS_SUCCESS == TSHttpTxnParentSelectionUrlSet(txnp, hdr_buf, ps_loc)) { + DEBUG_LOG("Set Parent Selection URL to cache_key_url: %s", cache_key_url); + TSHandleMLocRelease(hdr_buf, TS_NULL_MLOC, ps_loc); + } + } + } + + // optionally consider an X-CRR-IMS header + if (pc->consider_ims_header) { + TSMLoc const imsloc = TSMimeHdrFieldFind(hdr_buf, hdr_loc, X_IMS_HEADER.data(), X_IMS_HEADER.size()); + if (TS_NULL_MLOC != imsloc) { + time_t const itime = TSMimeHdrFieldValueDateGet(hdr_buf, hdr_loc, imsloc); + TSHandleMLocRelease(hdr_buf, hdr_loc, imsloc); + if (0 < itime) { + txn_state->ims_time = itime; + } } } } // remove the range request header. - if (remove_header(hdr_bufp, req_hdrs, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE) > 0) { + if (remove_header(hdr_buf, hdr_loc, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE) > 0) { DEBUG_LOG("Removed the Range: header from the request."); } @@ -196,13 +248,18 @@ range_header_check(TSHttpTxn txnp, struct pluginconfig *pc) TSHttpTxnHookAdd(txnp, TS_HTTP_SEND_RESPONSE_HDR_HOOK, txn_contp); TSHttpTxnHookAdd(txnp, TS_HTTP_TXN_CLOSE_HOOK, txn_contp); DEBUG_LOG("Added TS_HTTP_SEND_REQUEST_HDR_HOOK, TS_HTTP_SEND_RESPONSE_HDR_HOOK, and TS_HTTP_TXN_CLOSE_HOOK"); + + if (0 < txn_state->ims_time) { + TSHttpTxnHookAdd(txnp, TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK, txn_contp); + DEBUG_LOG("Also Added TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK"); + } } } - TSHandleMLocRelease(hdr_bufp, req_hdrs, loc); + // TSHandleMLocRelease(hdr_buf, hdr_loc, loc); } else { DEBUG_LOG("no range request header."); } - TSHandleMLocRelease(hdr_bufp, TS_NULL_MLOC, req_hdrs); + TSHandleMLocRelease(hdr_buf, TS_NULL_MLOC, hdr_loc); } else { DEBUG_LOG("failed to retrieve the server request"); } @@ -212,41 +269,44 @@ range_header_check(TSHttpTxn txnp, struct pluginconfig *pc) * Restores the range request header if the request must be * satisfied from the origin and schedules the TS_READ_RESPONSE_HDR_HOOK. */ -static void -handle_send_origin_request(TSCont contp, TSHttpTxn txnp, struct txndata *txn_state) +void +handle_send_origin_request(TSCont contp, TSHttpTxn txnp, txndata *const txn_state) { - TSMBuffer hdr_bufp; - TSMLoc req_hdrs = nullptr; + TSMBuffer hdr_buf; + TSMLoc hdr_loc = nullptr; + + std::string const &rv = txn_state->range_value; - if (TS_SUCCESS == TSHttpTxnServerReqGet(txnp, &hdr_bufp, &req_hdrs) && txn_state->range_value != nullptr) { - if (set_header(hdr_bufp, req_hdrs, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, txn_state->range_value, - strlen(txn_state->range_value))) { - DEBUG_LOG("Added range header: %s", txn_state->range_value); + if (TS_SUCCESS == TSHttpTxnServerReqGet(txnp, &hdr_buf, &hdr_loc) && !rv.empty()) { + if (set_header(hdr_buf, hdr_loc, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, rv.data(), rv.length())) { + DEBUG_LOG("Added range header: %s", rv.c_str()); TSHttpTxnHookAdd(txnp, TS_HTTP_READ_RESPONSE_HDR_HOOK, contp); } } - TSHandleMLocRelease(hdr_bufp, TS_NULL_MLOC, req_hdrs); + TSHandleMLocRelease(hdr_buf, TS_NULL_MLOC, hdr_loc); } /** * Changes the response code back to a 206 Partial content before * replying to the client that requested a range. */ -static void -handle_client_send_response(TSHttpTxn txnp, struct txndata *txn_state) +void +handle_client_send_response(TSHttpTxn txnp, txndata *const txn_state) { bool partial_content_reason = false; char *p; int length; - TSMBuffer response, hdr_bufp; - TSMLoc resp_hdr, req_hdrs = nullptr; + TSMBuffer resp_buf = nullptr; + TSMBuffer req_buf = nullptr; + TSMLoc resp_loc = nullptr; + TSMLoc req_loc = nullptr; - TSReturnCode result = TSHttpTxnClientRespGet(txnp, &response, &resp_hdr); + TSReturnCode result = TSHttpTxnClientRespGet(txnp, &resp_buf, &resp_loc); DEBUG_LOG("result: %d", result); if (TS_SUCCESS == result) { - TSHttpStatus status = TSHttpHdrStatusGet(response, resp_hdr); + TSHttpStatus status = TSHttpHdrStatusGet(resp_buf, resp_loc); // a cached result will have a TS_HTTP_OK with a 'Partial Content' reason - if ((p = const_cast(TSHttpHdrReasonGet(response, resp_hdr, &length))) != nullptr) { + if ((p = const_cast(TSHttpHdrReasonGet(resp_buf, resp_loc, &length))) != nullptr) { if ((length == 15) && (0 == strncasecmp(p, "Partial Content", length))) { partial_content_reason = true; } @@ -254,23 +314,23 @@ handle_client_send_response(TSHttpTxn txnp, struct txndata *txn_state) DEBUG_LOG("%d %.*s", status, length, p); if (TS_HTTP_STATUS_OK == status && partial_content_reason) { DEBUG_LOG("Got TS_HTTP_STATUS_OK."); - TSHttpHdrStatusSet(response, resp_hdr, TS_HTTP_STATUS_PARTIAL_CONTENT); + TSHttpHdrStatusSet(resp_buf, resp_loc, TS_HTTP_STATUS_PARTIAL_CONTENT); DEBUG_LOG("Set response header to TS_HTTP_STATUS_PARTIAL_CONTENT."); } } + std::string const &rv = txn_state->range_value; // add the range request header back in so that range requests may be logged. - if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &hdr_bufp, &req_hdrs) && txn_state->range_value != nullptr) { - if (set_header(hdr_bufp, req_hdrs, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, txn_state->range_value, - strlen(txn_state->range_value))) { - DEBUG_LOG("added range header: %s", txn_state->range_value); + if (TS_SUCCESS == TSHttpTxnClientReqGet(txnp, &req_buf, &req_loc) && !rv.empty()) { + if (set_header(req_buf, req_loc, TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, rv.data(), rv.length())) { + DEBUG_LOG("added range header: %s", rv.c_str()); } else { DEBUG_LOG("set_header() failed."); } } else { DEBUG_LOG("failed to get Request Headers"); } - TSHandleMLocRelease(response, TS_NULL_MLOC, resp_hdr); - TSHandleMLocRelease(hdr_bufp, TS_NULL_MLOC, req_hdrs); + TSHandleMLocRelease(resp_buf, TS_NULL_MLOC, resp_loc); + TSHandleMLocRelease(req_buf, TS_NULL_MLOC, req_loc); } /** @@ -278,20 +338,20 @@ handle_client_send_response(TSHttpTxn txnp, struct txndata *txn_state) * the response code from a 206 Partial content to a 200 OK so that * the response will be written to cache. */ -static void -handle_server_read_response(TSHttpTxn txnp, struct txndata *txn_state) +void +handle_server_read_response(TSHttpTxn txnp, txndata *const txn_state) { - TSMBuffer response; - TSMLoc resp_hdr; + TSMBuffer resp_buf = nullptr; + TSMLoc resp_loc = nullptr; TSHttpStatus status; - if (TS_SUCCESS == TSHttpTxnServerRespGet(txnp, &response, &resp_hdr)) { - status = TSHttpHdrStatusGet(response, resp_hdr); + if (TS_SUCCESS == TSHttpTxnServerRespGet(txnp, &resp_buf, &resp_loc)) { + status = TSHttpHdrStatusGet(resp_buf, resp_loc); if (TS_HTTP_STATUS_PARTIAL_CONTENT == status) { DEBUG_LOG("Got TS_HTTP_STATUS_PARTIAL_CONTENT."); - TSHttpHdrStatusSet(response, resp_hdr, TS_HTTP_STATUS_OK); + TSHttpHdrStatusSet(resp_buf, resp_loc, TS_HTTP_STATUS_OK); DEBUG_LOG("Set response header to TS_HTTP_STATUS_OK."); - bool cacheable = TSHttpTxnIsCacheable(txnp, nullptr, response); + bool cacheable = TSHttpTxnIsCacheable(txnp, nullptr, resp_buf); DEBUG_LOG("range is cacheable: %d", cacheable); } else if (TS_HTTP_STATUS_OK == status) { DEBUG_LOG("The origin does not support range requests, attempting to disable cache write."); @@ -302,7 +362,7 @@ handle_server_read_response(TSHttpTxn txnp, struct txndata *txn_state) } } } - TSHandleMLocRelease(response, TS_NULL_MLOC, resp_hdr); + TSHandleMLocRelease(resp_buf, TS_NULL_MLOC, resp_loc); } /** @@ -311,18 +371,18 @@ handle_server_read_response(TSHttpTxn txnp, struct txndata *txn_state) * * From background_fetch.cc */ -static int -remove_header(TSMBuffer bufp, TSMLoc hdr_loc, const char *header, int len) +int +remove_header(TSMBuffer buf, TSMLoc hdr_loc, const char *header, int len) { - TSMLoc field = TSMimeHdrFieldFind(bufp, hdr_loc, header, len); + TSMLoc field = TSMimeHdrFieldFind(buf, hdr_loc, header, len); int cnt = 0; while (field) { - TSMLoc tmp = TSMimeHdrFieldNextDup(bufp, hdr_loc, field); + TSMLoc tmp = TSMimeHdrFieldNextDup(buf, hdr_loc, field); ++cnt; - TSMimeHdrFieldDestroy(bufp, hdr_loc, field); - TSHandleMLocRelease(bufp, hdr_loc, field); + TSMimeHdrFieldDestroy(buf, hdr_loc, field); + TSHandleMLocRelease(buf, hdr_loc, field); field = tmp; } @@ -336,25 +396,25 @@ remove_header(TSMBuffer bufp, TSMLoc hdr_loc, const char *header, int len) * * From background_fetch.cc */ -static bool -set_header(TSMBuffer bufp, TSMLoc hdr_loc, const char *header, int len, const char *val, int val_len) +bool +set_header(TSMBuffer buf, TSMLoc hdr_loc, const char *header, int len, const char *val, int val_len) { - if (!bufp || !hdr_loc || !header || len <= 0 || !val || val_len <= 0) { + if (!buf || !hdr_loc || !header || len <= 0 || !val || val_len <= 0) { return false; } DEBUG_LOG("header: %s, len: %d, val: %s, val_len: %d", header, len, val, val_len); bool ret = false; - TSMLoc field_loc = TSMimeHdrFieldFind(bufp, hdr_loc, header, len); + TSMLoc field_loc = TSMimeHdrFieldFind(buf, hdr_loc, header, len); if (!field_loc) { // No existing header, so create one - if (TS_SUCCESS == TSMimeHdrFieldCreateNamed(bufp, hdr_loc, header, len, &field_loc)) { - if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, val, val_len)) { - TSMimeHdrFieldAppend(bufp, hdr_loc, field_loc); + if (TS_SUCCESS == TSMimeHdrFieldCreateNamed(buf, hdr_loc, header, len, &field_loc)) { + if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(buf, hdr_loc, field_loc, -1, val, val_len)) { + TSMimeHdrFieldAppend(buf, hdr_loc, field_loc); ret = true; } - TSHandleMLocRelease(bufp, hdr_loc, field_loc); + TSHandleMLocRelease(buf, hdr_loc, field_loc); } } else { TSMLoc tmp = nullptr; @@ -363,14 +423,14 @@ set_header(TSMBuffer bufp, TSMLoc hdr_loc, const char *header, int len, const ch while (field_loc) { if (first) { first = false; - if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(bufp, hdr_loc, field_loc, -1, val, val_len)) { + if (TS_SUCCESS == TSMimeHdrFieldValueStringSet(buf, hdr_loc, field_loc, -1, val, val_len)) { ret = true; } } else { - TSMimeHdrFieldDestroy(bufp, hdr_loc, field_loc); + TSMimeHdrFieldDestroy(buf, hdr_loc, field_loc); } - tmp = TSMimeHdrFieldNextDup(bufp, hdr_loc, field_loc); - TSHandleMLocRelease(bufp, hdr_loc, field_loc); + tmp = TSMimeHdrFieldNextDup(buf, hdr_loc, field_loc); + TSHandleMLocRelease(buf, hdr_loc, field_loc); field_loc = tmp; } } @@ -378,6 +438,91 @@ set_header(TSMBuffer bufp, TSMLoc hdr_loc, const char *header, int len, const ch return ret; } +time_t +get_date_from_cached_hdr(TSHttpTxn txn) +{ + TSMBuffer buf; + TSMLoc hdr_loc, date_loc; + time_t date = 0; + + if (TSHttpTxnCachedRespGet(txn, &buf, &hdr_loc) == TS_SUCCESS) { + date_loc = TSMimeHdrFieldFind(buf, hdr_loc, TS_MIME_FIELD_DATE, TS_MIME_LEN_DATE); + if (date_loc != TS_NULL_MLOC) { + date = TSMimeHdrFieldValueDateGet(buf, hdr_loc, date_loc); + TSHandleMLocRelease(buf, hdr_loc, date_loc); + } + TSHandleMLocRelease(buf, TS_NULL_MLOC, hdr_loc); + } + + return date; +} + +/** + * Handle a special IMS request. + */ +void +handle_cache_lookup_complete(TSHttpTxn txnp, txndata *const txn_state) +{ + int cachestat; + if (TS_SUCCESS == TSHttpTxnCacheLookupStatusGet(txnp, &cachestat)) { + if (TS_CACHE_LOOKUP_HIT_FRESH == cachestat) { + time_t const ch_time = get_date_from_cached_hdr(txnp); + if (ch_time < txn_state->ims_time) { + TSHttpTxnCacheLookupStatusSet(txnp, TS_CACHE_LOOKUP_HIT_STALE); + if (TSIsDebugTagSet(PLUGIN_NAME)) { + int url_len = 0; + char *const req_url = TSHttpTxnEffectiveUrlStringGet(txnp, &url_len); + if (nullptr != req_url) { + std::string const &rv = txn_state->range_value; + DEBUG_LOG("Forced revalidate %.*s-%s", url_len, req_url, rv.c_str()); + + TSfree(req_url); + } + } + } + } + } +} + +/** + * Transaction event handler. + */ +int +transaction_handler(TSCont contp, TSEvent event, void *edata) +{ + TSHttpTxn txnp = static_cast(edata); + txndata *const txn_state = static_cast(TSContDataGet(contp)); + + switch (event) { + case TS_EVENT_HTTP_READ_RESPONSE_HDR: + handle_server_read_response(txnp, txn_state); + break; + case TS_EVENT_HTTP_SEND_REQUEST_HDR: + handle_send_origin_request(contp, txnp, txn_state); + break; + case TS_EVENT_HTTP_SEND_RESPONSE_HDR: + handle_client_send_response(txnp, txn_state); + break; + case TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE: + handle_cache_lookup_complete(txnp, txn_state); + break; + case TS_EVENT_HTTP_TXN_CLOSE: + if (txn_state != nullptr) { + TSContDataSet(contp, nullptr); + delete txn_state; + } + TSContDestroy(contp); + break; + default: + TSAssert(!"Unexpected event"); + break; + } + TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); + return 0; +} + +} // namespace + /** * Remap initialization. */ @@ -411,11 +556,10 @@ TSRemapNewInstance(int argc, char *argv[], void **ih, char * /*errbuf */, int /* } // Skip over the Remap params - const char **plugin_argv = const_cast(argv + 2); - argc -= 2; + char *const *plugin_argv = const_cast(argv); // Parse the argument list. - *ih = create_pluginconfig(argc, plugin_argv); + *ih = static_cast(create_pluginconfig(argc - 2, plugin_argv + 2)); if (*ih == nullptr) { ERROR_LOG("Can't create pluginconfig"); @@ -430,7 +574,7 @@ TSRemapNewInstance(int argc, char *argv[], void **ih, char * /*errbuf */, int /* void TSRemapDeleteInstance(void *ih) { - struct pluginconfig *pc = static_cast(ih); + pluginconfig *const pc = static_cast(ih); if (nullptr != pc) { delete_pluginconfig(pc); @@ -443,7 +587,7 @@ TSRemapDeleteInstance(void *ih) TSRemapStatus TSRemapDoRemap(void *ih, TSHttpTxn txnp, TSRemapRequestInfo * /* rri */) { - struct pluginconfig *pc = static_cast(ih); + pluginconfig *const pc = static_cast(ih); range_header_check(txnp, pc); @@ -470,11 +614,9 @@ TSPluginInit(int argc, const char *argv[]) } if (nullptr == gPluginConfig) { - if (argc > 1) { - // Skip ahead of first param (name of traffic server plugin shared object) - const char **plugin_argv = const_cast(argv + 1); - argc -= 1; - gPluginConfig = create_pluginconfig(argc, plugin_argv); + if (1 < argc) { + char *const *plugin_argv = const_cast(argv); + gPluginConfig = create_pluginconfig(argc - 1, plugin_argv + 1); } } @@ -485,39 +627,3 @@ TSPluginInit(int argc, const char *argv[]) TSHttpHookAdd(TS_HTTP_READ_REQUEST_HDR_HOOK, txnp_cont); } } - -/** - * Transaction event handler. - */ -static int -transaction_handler(TSCont contp, TSEvent event, void *edata) -{ - TSHttpTxn txnp = static_cast(edata); - struct txndata *txn_state = static_cast(TSContDataGet(contp)); - - switch (event) { - case TS_EVENT_HTTP_READ_RESPONSE_HDR: - handle_server_read_response(txnp, txn_state); - break; - case TS_EVENT_HTTP_SEND_REQUEST_HDR: - handle_send_origin_request(contp, txnp, txn_state); - break; - case TS_EVENT_HTTP_SEND_RESPONSE_HDR: - handle_client_send_response(txnp, txn_state); - break; - case TS_EVENT_HTTP_TXN_CLOSE: - if (txn_state != nullptr && txn_state->range_value != nullptr) { - TSfree(txn_state->range_value); - } - if (txn_state != nullptr) { - TSfree(txn_state); - } - TSContDestroy(contp); - break; - default: - TSAssert(!"Unexpected event"); - break; - } - TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); - return 0; -} diff --git a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py index 49aaf6ed22a..c46b1279814 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py +++ b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests.test.py @@ -76,7 +76,7 @@ "HTTP/1.1 200 OK\r\n" + "Cache-Control: max-age=500\r\n" + "Connection: close\r\n" + - 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + 'Etag: "path"\r\n' + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + "\r\n", "timestamp": "1469733493.993", @@ -107,7 +107,7 @@ "Cache-Control: max-age=500\r\n" + "Content-Range: bytes {0}/{1}\r\n".format(inner_str, bodylen) + "Connection: close\r\n" + - 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + 'Etag: "path"\r\n' + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + "\r\n", "timestamp": "1469733493.993", @@ -135,7 +135,7 @@ "Cache-Control: max-age=500\r\n" + "Content-Range: bytes 0-{0}/{0}\r\n".format(bodylen) + "Connection: close\r\n" + - 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + 'Etag: "path"\r\n' + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + "\r\n", "timestamp": "1469733493.993", @@ -163,7 +163,7 @@ "Cache-Control: max-age=200\r\n" + "Content-Range: bytes {0}-{1}/{1}\r\n".format(bodylen - 5, bodylen) + "Connection: close\r\n" + - 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + 'Etag: "path"\r\n' + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + "\r\n", "timestamp": "1469733493.993", @@ -191,7 +191,7 @@ "Cache-Control: max-age=200\r\n" + "Content-Range: bytes {}/19\r\n".format(pselect_str) + "Connection: close\r\n" + - 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + 'Etag: "path"\r\n' + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + "\r\n", "timestamp": "1469733493.993", @@ -200,11 +200,30 @@ server.addResponse("sessionlog.json", req_pselect, res_pselect) +req_psd = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: psd\r\n" + + "Accept: */*\r\n" + + "Range: bytes={}\r\n".format(pselect_str) + + "uuid: pselect\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +server.addResponse("sessionlog.json", req_psd, res_pselect) + # cache range requests plugin remap ts.Disk.remap_config.AddLines([ 'map http://www.example.com http://127.0.0.1:{}'.format(server.Variables.Port) + ' @plugin=cache_range_requests.so', + + # parent select cache key option 'map http://parentselect http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cache_range_requests.so @pparam=--ps-cachekey', + + # deprecated + 'map http://psd http://127.0.0.1:{}'.format(server.Variables.Port) + ' @plugin=cache_range_requests.so @pparam=ps_mode:cache_key_url', ]) @@ -338,3 +357,16 @@ ps.Streams.stdout = "gold/pselect_none.stdout.gold" tr.StillRunningAfter = ts tr.StillRunningAfter = server + +# 11 Test - cache_key_url request -- deprecated +tr = Test.AddTestRun("cache_key_url request - dprecated") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://psd/path -r {} -H "uuid: pselect"'.format(pselect_str) +ps.ReturnCode = 0 +ps.Streams.stdout = "gold/pselect.stdout.gold" +ps.Streams.stdout.Content = Testers.ContainsExpression( + "X-ParentSelection-Key: .*-bytes=", + "expected bytes in parent selection key", +) +tr.StillRunningAfter = ts +tr.StillRunningAfter = server diff --git a/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_ims.test.py b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_ims.test.py new file mode 100644 index 00000000000..f249700ff39 --- /dev/null +++ b/tests/gold_tests/pluginTest/cache_range_requests/cache_range_requests_ims.test.py @@ -0,0 +1,141 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import time + +Test.Summary = ''' +cache_range_requests X-CRR-IMS plugin test +''' + +## Test description: +# Preload the cache with the entire asset to be range requested. +# Reload remap rule with cache_range_requests plugin +# Request content through the cache_range_requests plugin + +Test.SkipUnless( + Condition.PluginExists('cache_range_requests.so'), + Condition.PluginExists('xdebug.so'), +) +Test.ContinueOnFail = False +Test.testName = "cache_range_requests" + +# Define and configure ATS +ts = Test.MakeATSProcess("ts", command="traffic_server") + +# Define and configure origin server +server = Test.MakeOriginServer("server") + +# default root +req_chk = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "uuid: none\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_chk = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +server.addResponse("sessionlog.json", req_chk, res_chk) + +body = "lets go surfin now" +bodylen = len(body) + +req_full = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Accept: */*\r\n" + + "Range: bytes=0-\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" +} + +res_full = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Accept-Ranges: bytes\r\n" + + "Cache-Control: max-age=500\r\n" + + "Content-Range: bytes 0-{0}/{0}\r\n".format(bodylen) + + "Connection: close\r\n" + + 'Etag: "772102f4-56f4bc1e6d417"\r\n' + + "Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body +} + +server.addResponse("sessionlog.json", req_full, res_full) + +# cache range requests plugin remap +ts.Disk.remap_config.AddLine( + 'map http://www.example.com http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=cache_range_requests.so @pparam=--consider-ims', +) + +# cache debug +ts.Disk.plugin_config.AddLine('xdebug.so') + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'cache_range_requests', + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, +}) + +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x localhost:{} -H "x-debug: x-cache"'.format(ts.Variables.port) + +# 0 Test - Fetch whole asset into cache +tr = Test.AddTestRun("0- range cache load") +ps = tr.Processes.Default +ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) +ps.StartBefore(Test.Processes.ts, ready=When.PortOpen(ts.Variables.port)) +ps.Command = curl_and_args + ' http://www.example.com/path -r 0-' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss for load") +tr.StillRunningAfter = ts + + +# set up the IMS date field (go in the future) RFC 2616 +futuretime = time.time() + 100 # seconds +futurestr = time.strftime("%a, %d %b %Y %H:%M:%S GMT", time.gmtime(futuretime)) + +# test inner range +# 1 Test - Fetch range into cache +tr = Test.AddTestRun("0- cache hit check") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r 0-'.format(futurestr) +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit", "expected cache hit") +tr.StillRunningAfter = ts + +# 2 Test - Ensure X-CRR-IMS header results in hit-stale +tr = Test.AddTestRun("0- range X-CRR-IMS check") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://www.example.com/path -r 0- -H "X-CRR-IMS: {}"'.format(futurestr) +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-stale", "expected cache hit-stale") +tr.StillRunningAfter = ts diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold index 6dae639a9c3..e108e5c0b3a 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/frange.stdout.gold @@ -2,7 +2,7 @@ HTTP/1.1 206 Partial Content Accept-Ranges: bytes Cache-Control: max-age=500 Content-Range: bytes 0-18/18 -Etag: "772102f4-56f4bc1e6d417" +Etag: "path" Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT Content-Length: 18 Date: `` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold index a485d263577..303f162c331 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/full.stdout.gold @@ -1,6 +1,6 @@ HTTP/1.1 200 OK Cache-Control: max-age=500 -Etag: "772102f4-56f4bc1e6d417" +Etag: "path" Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT Content-Length: 18 Date: `` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold index 0628bfad209..56aa6d1e6ce 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/inner.stdout.gold @@ -2,7 +2,7 @@ HTTP/1.1 206 Partial Content Accept-Ranges: bytes Cache-Control: max-age=500 Content-Range: bytes 7-15/18 -Etag: "772102f4-56f4bc1e6d417" +Etag: "path" Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT Content-Length: 7 Date: `` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold index 5e2da0005c8..45c17fa7120 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/last.stdout.gold @@ -2,7 +2,7 @@ HTTP/1.1 206 Partial Content Accept-Ranges: bytes Cache-Control: max-age=500 Content-Range: bytes 12-18/18 -Etag: "772102f4-56f4bc1e6d417" +Etag: "path" Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT Content-Length: 5 Date: `` diff --git a/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold index 11902d7ec3d..98165d48726 100644 --- a/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold +++ b/tests/gold_tests/pluginTest/cache_range_requests/gold/pselect.stdout.gold @@ -2,7 +2,7 @@ HTTP/1.1 206 Partial Content Accept-Ranges: bytes Cache-Control: max-age=500 Content-Range: bytes 1-10/18 -Etag: "772102f4-56f4bc1e6d417" +Etag: "path" Last-Modified: Sat, 23 Jun 2018 09:27:29 GMT Content-Length: 9 Date: ``