diff --git a/.gitignore b/.gitignore index 10a0d15df99..78dedcee19d 100644 --- a/.gitignore +++ b/.gitignore @@ -117,6 +117,10 @@ plugins/esi/processor_test plugins/esi/utils_test plugins/esi/vars_test +plugins/experimental/slice/test_config +plugins/experimental/slice/test_content_range +plugins/experimental/slice/test_range + mgmt/api/traffic_api_cli_remote mgmt/tools/traffic_mcast_snoop mgmt/tools/traffic_net_config diff --git a/doc/admin-guide/plugins/index.en.rst b/doc/admin-guide/plugins/index.en.rst index c345f52a8f6..6880c6b9ad7 100644 --- a/doc/admin-guide/plugins/index.en.rst +++ b/doc/admin-guide/plugins/index.en.rst @@ -62,6 +62,7 @@ Plugins that are considered stable are installed by default in |TS| releases. Regex Remap Regex Revalidate Remap Purge + Slice Stats over HTTP TCPInfo XDebug @@ -117,6 +118,11 @@ Plugins that are considered stable are installed by default in |TS| releases. :doc:`Regex Revalidate ` Configurable rules for forcing cache object revalidations using regular expressions. +:doc:`Slicer ` + Slice full file or range based requests into deterministic chunks, + allowing large files to be spread across multiple cache stripes. Allows + range requests to be satisfied by stitching these chunks together. + :doc:`Stats over HTTP ` Provide an HTTP interface to all |TS| statistics. diff --git a/doc/admin-guide/plugins/slice.en.rst b/doc/admin-guide/plugins/slice.en.rst new file mode 100644 index 00000000000..b131178eba6 --- /dev/null +++ b/doc/admin-guide/plugins/slice.en.rst @@ -0,0 +1,256 @@ +.. Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + +.. _admin-plugins-slice: + +Slice Plugin +*************** + +This plugin takes client requests and breaks them up into +successive aligned block requests. This supports both +whole asset and single range requests. + +Purpose +======= + +This slice plugin, along with the `cache_range_requests` +plugin allows the following: + +- Fulfill arbitrary range requests by fetching a minimum + number of cacheable aligned blocks to fulfill the request. +- Breaks up very large assets into much smaller cache + blocks that can be spread across multiple storage + devices and within cache groups. + +Configuration +============= + +This plugin is intended for use as a remap plugin and is +configured in :file:`remap.config`. + +Or preferably per remap rule in :file:`remap.config`:: + + map http://ats/ http://parent/ @plugin=slice.so \ + @plugin=cache_range_requests.so + +In this case, the plugin will use the default behaviour: + +- Fulfill whole file or range requests by requesting cacheable + block aligned ranges from the parent and assemble them + into client responses, either 200 or 206 depending on the + client request. +- Default block size is 1mb (1048576 bytes). +- This plugin depends on the cache_range_requests plugin + to perform actual parent fetching and block caching + and If-* conditional header evaluations. + +Plugin Options +-------------- + +The slice plugin supports the following options:: + + --blockbytes= (optional) + Default is 1m or 1048576 bytes + -b for short. + Suffix k,m,g supported + Limited to 32k and 32m inclusive. + + --blockbytes-test= (optional) + Suffix k,m,g supported + -t for short. + Limited to any positive number. + Ignored if --blockbytes provided. + + --remap-host= (optional) + Uses effective url with given hostname for remapping. + Requires setting up an intermediate loopback remap rule. + -r for short + + --pace-errorlog= (optional) + Limit stitching error logs to every 'n' second(s) + -p for short + + --disable-errorlog (optional) + Disable writing block stitch errors to the error log. + -d for short + + +Examples:: + + @plugin=slice.so @pparam=--blockbytes=1000000 @plugin=cache_range_requests.so + +Or alternatively:: + + @plugin=slice.so @pparam=-b @pparam=1000000 @plugin=cache_range_requests.so + +Byte suffix examples:: + + slice.so --blockbytes=5m + slice.so -b 512k + slice.so --blockbytes=32m + +For testing and extreme purposes the parameter ``blockbytes-test`` may +be used instead which is unchecked:: + + slice.so --blockbytes-test=1G + slice.so -t 13 + +Because the slice plugin is susceptible to errors during block stitching +extra logs related to stitching are written to ``diags.log``. Worst case +an error log entry could be generated for every transaction. The +following options are provided to help with log overrun:: + + slice.so --pace-errorlog=5 + slice.so -p 1 + slice.so --disable-errorlog + +After modifying :file:`remap.config`, restart or reload traffic server +(sudo traffic_ctl config reload) or (sudo traffic_ctl server restart) +to activate the new configuration values. + +Debug Options +------------- + +While the current slice plugin is able to detect block consistency +errors during the block stitching process, it can only abort the +client connection. A CDN can only "fix" these by issuing an appropriate +content revalidation. + +Under normal logging these slice block errors tend to show up as:: + + pscl value 0 + crc value ERR_READ_ERROR + +By default more detailed stitching errors are written to ``diags.log``. +Examples are as follows:: + +ERROR: [slice.cc: 288] logSliceError(): 1555705573.639 reason="Non 206 internal block response" uri="http://ats_ep/someasset.mp4" uas="curl" req_range="bytes=1000000-" norm_range="bytes 1000000-52428799/52428800" etag_exp="%221603934496%22" lm_exp="Fri, 19 Apr 2019 18:53:20 GMT" blk_range="21000000-21999999" status_got="206" cr_got="" etag_got="%221603934496%22" lm_got="" cc="no-store" via="" + +ERROR: [server.cc: 288] logSliceError(): 1572370000.219 reason="Mismatch block Etag" uri="http://ats_ep/someasset.mp4" uas="curl" req_range="bytes=1092779033-1096299354" norm_range="bytes 1092779033-1096299354/2147483648" etag_exp="%223719843648%22" lm_exp="Tue, 29 Oct 2019 14:40:00 GMT" blk_range="1095000000-1095999999" status_got="206" cr_got="bytes 1095000000-1095999999/2147483648" etag_got="%223719853648%22" lm_got="Tue, 29 Oct 2019 17:26:40 GMT" cc="max-age=10000" via="" + +Whether or how often these detailed log entries are written are +configurable plugin options. + +Implementation Notes +==================== + +This slice plugin is a stop gap plugin for handling special cases +involving very large assets that may be range requested. Hopefully +the slice plugin is deprecated in the future when partial object +caching is finally implemented. + +Slice *ONLY* handles slicing up requests into blocks, it delegates +actual caching and fetching to the cache_range_requests.so plugin. + +Plugin Function +--------------- + +Below is a quick functional outline of how a request is served +by a remap rule containing the Slice plugin with cache_range_requests: + +For each client request that comes in all remap plugins are run up +until the slice plugin is hit. If the slice plugin *can* be run (ie: +GET request) it will handle the request and STOP any further plugins +from executing. + +At this point the request is sliced into 1 or more blocks by +adding in range request headers ("Range: bytes="). A special +header X-Slicer-Info header is added and the pristine URL is +restored. + +For each of these blocks separate sequential TSHttpConnect(s) are made +back into the front end of ATS and all of the remap plugins are rerun. +Slice skips the remap due to presence of the X-Slicer-Info header and +allows cache_range_requests.so to serve the slice block back to Slice +either via cache OR parent request. + +Slice assembles a header based on the first slice block response and +sends it to the client. If necessary it then skips over bytes in +the first block and starts sending byte content, examining each +block header and sends its bytes to the client until the client +request is satisfied. + +Any extra bytes at the end of the last block are consumed by +the the Slice plugin to allow cache_range_requests to finish +the block fetch to ensure the block is cached. + +Important Notes +=============== + +This plugin assumes that the content requested is cacheable. + +Any first block server response that is not a 206 is passed directly +down to the client. If that response is a '200' only the first +portion of the response is passed back and the transaction is closed. + +Only the first server response block is used to evaluate any "If-" +conditional headers. Subsequent server slice block requests +remove these headers. + +The only 416 response that this plugin handles itself is if the +requested range is inside the last slice block but past the end of +the asset contents. Other 416 responses are handled by the parent. + +If a client aborts mid transaction the current slice block continues to +be read from the server until it is complete to ensure that the block +is cached. + +Slice *always* makes ``blockbytes`` sized requests which are handled +by cache_range_requests. The parent will trim those requests to +account for the asset Content-Length so only the appropriate number +of bytes are actually transferred and cached. + +Effective URL remap +=================== + +By default the plugin restores the Pristine Url which reuses the same +remap rule for each slice block. This is wasteful in that it reruns +the previous remap rules, and those remap rules must be smart enough to +check for the existence of any headers they may have created the first +time they have were visited. + +To get around this the '--remap-host=' or '-r ' option may +be used. This requires an intermediate loopback remap to be defined which +handles each slice block request. + +This works well with any remap rules that use the url_sig or uri_signing +plugins. As the client remap rule is not caching any plugins that +manipulate the cache key would need to go into the loopback to parent +remap rule. + +NOTE: Requests NOT handled by the slice plugin (ie: HEAD requests) are +handled as with a typical remap rule. GET requests intercepted by the +slice plugin are virtually reissued into ATS and are proxied through +another remap rule which must contain the ``cache_range_requests`` plugin + +Examples:: + + map http://ats/ http://parent/ @plugin=slice.so @pparam=--remap-host=loopback + map http://loopback/ http://parent/ @plugin=cache_range_requests.so + +Alternatively:: + + map http://ats/ http://parent/ @plugin=slice.so @pparam=-r @pparam=loopback + map http://loopback/ http://parent/ @plugin=cache_range_requests.so + +Current Limitations +=================== + +Since the Slice plugin is written as an intercept handler it loses the +ability to use normal state machine hooks and transaction states. This +functionality is handled by using the ``cache_range_requests`` plugin +to interact with ATS. diff --git a/plugins/Makefile.am b/plugins/Makefile.am index 802f296af81..65201491636 100644 --- a/plugins/Makefile.am +++ b/plugins/Makefile.am @@ -74,6 +74,7 @@ include experimental/mp4/Makefile.inc include experimental/multiplexer/Makefile.inc include experimental/remap_purge/Makefile.inc include experimental/server_push_preload/Makefile.inc +include experimental/slice/Makefile.inc include experimental/ssl_cert_loader/Makefile.inc include experimental/sslheaders/Makefile.inc include experimental/stale_while_revalidate/Makefile.inc diff --git a/plugins/experimental/slice/Config.cc b/plugins/experimental/slice/Config.cc new file mode 100644 index 00000000000..fb04ebb27b8 --- /dev/null +++ b/plugins/experimental/slice/Config.cc @@ -0,0 +1,206 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "Config.h" + +#include +#include +#include +#include +#include + +#include "ts/experimental.h" + +int64_t +Config::bytesFrom(char const *const valstr) +{ + char *endptr = nullptr; + int64_t blockbytes = strtoll(valstr, &endptr, 10); + constexpr int64_t kib = 1024; + + if (nullptr != endptr && valstr < endptr) { + size_t const dist = endptr - valstr; + if (dist < strlen(valstr) && 0 <= blockbytes) { + switch (tolower(*endptr)) { + case 'g': + blockbytes *= (kib * kib * kib); + break; + case 'm': + blockbytes *= (kib * kib); + break; + case 'k': + blockbytes *= kib; + break; + default: + break; + } + } + } + + if (blockbytes < 0) { + blockbytes = 0; + } + + return blockbytes; +} + +bool +Config::fromArgs(int const argc, char const *const argv[]) +{ + DEBUG_LOG("Number of arguments: %d", argc); + for (int index = 0; index < argc; ++index) { + DEBUG_LOG("args[%d] = %s", index, argv[index]); + } + + // look for lowest priority deprecated blockbytes + int64_t blockbytes = 0; + + // backwards compat: look for blockbytes + for (int index = 0; index < argc; ++index) { + std::string_view const argstr = argv[index]; + + std::size_t const spos = argstr.find_first_of(':'); + if (spos != std::string_view::npos) { + std::string_view const key = argstr.substr(0, spos); + std::string_view const val = argstr.substr(spos + 1); + + if (!key.empty() && !val.empty()) { + char const *const valstr = val.data(); // inherits argv's null + int64_t const bytesread = bytesFrom(valstr); + + if (blockbytesmin <= bytesread && bytesread <= blockbytesmax) { + DEBUG_LOG("Found deprecated blockbytes %" PRId64, bytesread); + blockbytes = bytesread; + } + } + } + } + + // standard parsing + constexpr struct option longopts[] = { + {const_cast("blockbytes"), required_argument, nullptr, 'b'}, + {const_cast("blockbytes-test"), required_argument, nullptr, 't'}, + {const_cast("remap-host"), required_argument, nullptr, 'r'}, + {const_cast("pace-errorlog"), required_argument, nullptr, 'p'}, + {const_cast("disable-errorlog"), no_argument, nullptr, 'd'}, + {const_cast("throttle"), no_argument, nullptr, 'o'}, + {nullptr, 0, nullptr, 0}, + }; + + // getopt assumes args start at '1' so this hack is needed + char *const *argvp = (const_cast(argv) - 1); + for (;;) { + int const opt = getopt_long(argc + 1, argvp, "b:t:r:p:do", longopts, nullptr); + if (-1 == opt) { + break; + } + + DEBUG_LOG("processing '%c' %s", (char)opt, argvp[optind - 1]); + + switch (opt) { + case 'b': { + int64_t const bytesread = bytesFrom(optarg); + if (blockbytesmin <= bytesread && bytesread <= blockbytesmax) { + DEBUG_LOG("Using blockbytes %" PRId64, bytesread); + blockbytes = bytesread; + } else { + ERROR_LOG("Invalid blockbytes: %s", optarg); + } + } break; + case 't': { + if (0 == blockbytes) { + int64_t const bytesread = bytesFrom(optarg); + if (0 < bytesread) { + DEBUG_LOG("Using blockbytes-test %" PRId64, bytesread); + blockbytes = bytesread; + } else { + ERROR_LOG("Invalid blockbytes-test: %s", optarg); + } + } else { + DEBUG_LOG("Skipping blockbytes-test in favor of blockbytes"); + } + } break; + case 'r': + m_remaphost = optarg; + DEBUG_LOG("Using loopback remap host override: %s", m_remaphost.c_str()); + break; + case 'p': { + int const secsread = atoi(optarg); + if (0 < secsread) { + m_paceerrsecs = std::min(secsread, 60); + } else { + ERROR_LOG("Ignoring pace-errlog argument"); + } + } break; + case 'd': + m_paceerrsecs = -1; + break; + case 'o': + m_throttle = true; + DEBUG_LOG("Enabling internal block throttling"); + break; + default: + break; + } + } + + if (0 < blockbytes) { + DEBUG_LOG("Using configured blockbytes %" PRId64, blockbytes); + m_blockbytes = blockbytes; + } else { + DEBUG_LOG("Using default blockbytes %" PRId64, m_blockbytes); + } + + if (m_paceerrsecs < 0) { + DEBUG_LOG("Block stitching error logs disabled"); + } else if (0 == m_paceerrsecs) { + DEBUG_LOG("Block stitching error logs enabled"); + } else { + DEBUG_LOG("Block stitching error logs at most every %d sec(s)", m_paceerrsecs); + } + + return true; +} + +bool +Config::canLogError() +{ + if (m_paceerrsecs < 0) { + return false; + } else if (0 == m_paceerrsecs) { + return true; + } + +#if !defined(UNITTEST) + TSHRTime const timenow = TShrtime(); +#endif + + std::lock_guard const guard(m_mutex); + +#if !defined(UNITTEST) + if (timenow < m_nextlogtime) { + return false; + } + + m_nextlogtime = timenow + TS_HRTIME_SECONDS(m_paceerrsecs); +#else + m_nextlogtime = 0; // needed by clang +#endif + + return true; +} diff --git a/plugins/experimental/slice/Config.h b/plugins/experimental/slice/Config.h new file mode 100644 index 00000000000..42de33d1bef --- /dev/null +++ b/plugins/experimental/slice/Config.h @@ -0,0 +1,48 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "slice.h" + +#include + +// Data Structures and Classes +struct Config { + static constexpr int64_t const blockbytesmin = 1024 * 256; // 256KB + static constexpr int64_t const blockbytesmax = 1024 * 1024 * 32; // 32MB + static constexpr int64_t const blockbytesdefault = 1024 * 1024; // 1MB + + int64_t m_blockbytes{blockbytesdefault}; + std::string m_remaphost; // remap host to use for loopback slice GET + bool m_throttle{false}; // internal block throttling + int m_paceerrsecs{0}; // -1 disable logging, 0 no pacing, max 60s + + // Convert optarg to bytes + static int64_t bytesFrom(char const *const valstr); + + // Parse from args, ast one wins + bool fromArgs(int const argc, char const *const argv[]); + + // Check if the error should can be logged, if sucessful may update m_nexttime + bool canLogError(); + +private: + TSHRTime m_nextlogtime{0}; // next time to log in ns + std::mutex m_mutex; +}; diff --git a/plugins/experimental/slice/ContentRange.cc b/plugins/experimental/slice/ContentRange.cc new file mode 100644 index 00000000000..c9d9edde3de --- /dev/null +++ b/plugins/experimental/slice/ContentRange.cc @@ -0,0 +1,55 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "ContentRange.h" + +#include +#include + +static char const *const format = "bytes %" PRId64 "-%" PRId64 "/%" PRId64; + +bool +ContentRange::fromStringClosed(char const *const valstr) +{ + int const fields = sscanf(valstr, format, &m_beg, &m_end, &m_length); + + if (3 == fields && m_beg <= m_end) { + m_end += 1; + } else { + m_beg = m_end = m_length = -1; + } + + return isValid(); +} + +bool +ContentRange::toStringClosed(char *const rangestr, int *const rangelen) const +{ + if (!isValid()) { + if (0 < *rangelen) { + rangestr[0] = '\0'; + } + *rangelen = 0; + return false; + } + + int const lenin = *rangelen; + *rangelen = snprintf(rangestr, lenin, format, m_beg, (m_end - 1), m_length); + + return (0 < *rangelen && *rangelen < lenin); +} diff --git a/plugins/experimental/slice/ContentRange.h b/plugins/experimental/slice/ContentRange.h new file mode 100644 index 00000000000..2412e668433 --- /dev/null +++ b/plugins/experimental/slice/ContentRange.h @@ -0,0 +1,53 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +/** + represents value parsed from a blocked Content-Range response header field. + Range is converted from closed range into a half open range for. + */ +struct ContentRange { + int64_t m_beg = -1; + int64_t m_end = -1; // half open + int64_t m_length = -1; // full content length + + ContentRange() {} + explicit ContentRange(int64_t const begin, int64_t const end, int64_t const len) : m_beg(begin), m_end(end), m_length(len) {} + bool + isValid() const + { + return 0 <= m_beg && m_beg < m_end && m_end <= m_length; + } + + /** parsed from a Content-Range field + */ + bool fromStringClosed(char const *const valstr); + + /** usable for Content-Range field + */ + bool toStringClosed(char *const rangestr, int *const rangelen) const; + + int64_t + rangeSize() const + { + return m_end - m_beg; + } +}; diff --git a/plugins/experimental/slice/Data.cc b/plugins/experimental/slice/Data.cc new file mode 100644 index 00000000000..aac6943d47b --- /dev/null +++ b/plugins/experimental/slice/Data.cc @@ -0,0 +1,65 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "Data.h" + +#include +#include +#include +#include +#include + +namespace +{ +std::mutex mutex; +int64_t inplay = 0; +std::unique_ptr thread; +} // namespace + +void +monitor() +{ + std::lock_guard guard(mutex); + // while (0 < inplay) + while (true) { + mutex.unlock(); + std::this_thread::sleep_for(std::chrono::seconds(10)); + std::cerr << "Inplay: " << inplay << std::endl; + mutex.lock(); + } + // thread.release(); +} + +void +incrData() +{ + std::lock_guard const guard(mutex); + if (!thread) { + thread.reset(new std::thread(monitor)); + } + + ++inplay; +} + +void +decrData() +{ + std::lock_guard const guard(mutex); + --inplay; + assert(0 <= inplay); +} diff --git a/plugins/experimental/slice/Data.h b/plugins/experimental/slice/Data.h new file mode 100644 index 00000000000..3150bd299b8 --- /dev/null +++ b/plugins/experimental/slice/Data.h @@ -0,0 +1,124 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +#include "HttpHeader.h" +#include "Range.h" +#include "Stage.h" + +#include + +struct Config; + +struct Data { + Data(Data const &) = delete; + Data &operator=(Data const &) = delete; + + Config *const m_config; + + sockaddr_storage m_client_ip; + + // for pristine/effective url coming in + TSMBuffer m_urlbuf{nullptr}; + TSMLoc m_urlloc{nullptr}; + + char m_hostname[8192]; + int m_hostlen; + char m_etag[8192]; + int m_etaglen; + char m_lastmodified[8192]; + int m_lastmodifiedlen; + + TSHttpStatus m_statustype; // 200 or 206 + + bool m_bail; // non 206/200 response + + Range m_req_range; // converted to half open interval + int64_t m_contentlen; + + int64_t m_blocknum; // block number to work on, -1 bad/stop + int64_t m_blockexpected; // body bytes expected + int64_t m_blockskip; // number of bytes to skip in this block + int64_t m_blockconsumed; // body bytes consumed + + enum BlockState { Pending, Active, Done, Fail }; + BlockState m_blockstate; // is there an active slice block + + int64_t m_bytestosend; // header + content bytes to send + int64_t m_bytessent; // number of bytes written to the client + + bool m_server_block_header_parsed; + bool m_server_first_header_parsed; + + Stage m_upstream; + Stage m_dnstream; + + HdrMgr m_req_hdrmgr; // manager for server request + HdrMgr m_resp_hdrmgr; // manager for client response + + TSHttpParser m_http_parser{nullptr}; //!< cached for reuse + + explicit Data(Config *const config) + : m_config(config), + m_client_ip(), + m_urlbuf(nullptr), + m_urlloc(nullptr), + m_hostlen(0), + m_etaglen(0), + m_lastmodifiedlen(0), + m_statustype(TS_HTTP_STATUS_NONE), + m_req_range(-1, -1), + m_contentlen(-1), + m_blocknum(-1), + m_blockexpected(0), + m_blockskip(0), + m_blockconsumed(0), + m_blockstate(Pending), + m_bytestosend(0), + m_bytessent(0), + m_server_block_header_parsed(false), + m_server_first_header_parsed(false), + m_http_parser(nullptr) + { + m_hostname[0] = '\0'; + m_lastmodified[0] = '\0'; + m_etag[0] = '\0'; +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::DataCreate, 1); +#endif + } + + ~Data() + { +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::DataDestroy, 1); +#endif + if (nullptr != m_urlbuf) { + if (nullptr != m_urlloc) { + TSHandleMLocRelease(m_urlbuf, TS_NULL_MLOC, m_urlloc); + } + TSMBufferDestroy(m_urlbuf); + } + if (nullptr != m_http_parser) { + TSHttpParserDestroy(m_http_parser); + } + } +}; diff --git a/plugins/experimental/slice/HttpHeader.cc b/plugins/experimental/slice/HttpHeader.cc new file mode 100644 index 00000000000..99137018066 --- /dev/null +++ b/plugins/experimental/slice/HttpHeader.cc @@ -0,0 +1,363 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "HttpHeader.h" + +#include "slice.h" + +#include +#include + +TSHttpType +HttpHeader::type() const +{ + if (isValid()) { + return TSHttpHdrTypeGet(m_buffer, m_lochdr); + } else { + return TS_HTTP_TYPE_UNKNOWN; + } +} + +TSHttpStatus +HttpHeader::status() const +{ + TSHttpStatus res = TS_HTTP_STATUS_NONE; + if (isValid()) { + res = TSHttpHdrStatusGet(m_buffer, m_lochdr); + } + return res; +} + +bool +HttpHeader::setStatus(TSHttpStatus const newstatus) +{ + if (!isValid()) { + return false; + } + + return TS_SUCCESS == TSHttpHdrStatusSet(m_buffer, m_lochdr, newstatus); +} + +char * +HttpHeader ::urlString(int *const urllen) const +{ + char *urlstr = nullptr; + TSAssert(nullptr != urllen); + + TSMLoc locurl = nullptr; + TSReturnCode const rcode = TSHttpHdrUrlGet(m_buffer, m_lochdr, &locurl); + if (TS_SUCCESS == rcode && nullptr != locurl) { + urlstr = TSUrlStringGet(m_buffer, locurl, urllen); + TSHandleMLocRelease(m_buffer, m_lochdr, locurl); + } else { + *urllen = 0; + } + + return urlstr; +} + +bool +HttpHeader::setUrl(TSMBuffer const bufurl, TSMLoc const locurl) +{ + if (!isValid()) { + return false; + } + + TSMLoc locurlout = nullptr; + TSReturnCode rcode = TSHttpHdrUrlGet(m_buffer, m_lochdr, &locurlout); + if (TS_SUCCESS != rcode) { + return false; + } + + // copy the url + rcode = TSUrlCopy(m_buffer, locurlout, bufurl, locurl); + + // set url active + if (TS_SUCCESS == rcode) { + rcode = TSHttpHdrUrlSet(m_buffer, m_lochdr, locurlout); + } + + TSHandleMLocRelease(m_buffer, m_lochdr, locurlout); + + return TS_SUCCESS == rcode; +} + +bool +HttpHeader::setReason(char const *const valstr, int const vallen) +{ + if (isValid()) { + return TS_SUCCESS == TSHttpHdrReasonSet(m_buffer, m_lochdr, valstr, vallen); + } else { + return false; + } +} + +char const * +HttpHeader::getCharPtr(CharPtrGetFunc func, int *const len) const +{ + char const *res = nullptr; + if (isValid()) { + int reslen = 0; + res = func(m_buffer, m_lochdr, &reslen); + + if (nullptr != len) { + *len = reslen; + } + } + + if (nullptr == res && nullptr != len) { + *len = 0; + } + + return res; +} + +bool +HttpHeader::hasKey(char const *const key, int const keylen) const +{ + if (!isValid()) { + return false; + } + + TSMLoc const locfield(TSMimeHdrFieldFind(m_buffer, m_lochdr, key, keylen)); + if (nullptr != locfield) { + TSHandleMLocRelease(m_buffer, m_lochdr, locfield); + return true; + } + + return false; +} + +bool +HttpHeader::removeKey(char const *const keystr, int const keylen) +{ + if (!isValid()) { + return false; + } + + bool status = true; + + TSMLoc const locfield = TSMimeHdrFieldFind(m_buffer, m_lochdr, keystr, keylen); + if (nullptr != locfield) { + int const rcode = TSMimeHdrFieldRemove(m_buffer, m_lochdr, locfield); + status = (TS_SUCCESS == rcode); + TSHandleMLocRelease(m_buffer, m_lochdr, locfield); + } + + return status; +} + +bool +HttpHeader::valueForKey(char const *const keystr, int const keylen, char *const valstr, int *const vallen, int const index) const +{ + if (!isValid()) { + *vallen = 0; + return false; + } + + bool status = false; + + TSMLoc const locfield = TSMimeHdrFieldFind(m_buffer, m_lochdr, keystr, keylen); + + if (nullptr != locfield) { + int getlen = 0; + char const *const getstr = TSMimeHdrFieldValueStringGet(m_buffer, m_lochdr, locfield, index, &getlen); + + int const valcap = *vallen; + if (nullptr != getstr && 0 < getlen && getlen < (valcap - 1)) { + char *const endp = stpncpy(valstr, getstr, getlen); + + *vallen = endp - valstr; + status = (*vallen < valcap); + + if (status) { + *endp = '\0'; + } + } + TSHandleMLocRelease(m_buffer, m_lochdr, locfield); + } else { + *vallen = 0; + } + + return status; +} + +bool +HttpHeader::setKeyVal(char const *const keystr, int const keylen, char const *const valstr, int const vallen, int const index) +{ + if (!isValid()) { + return false; + } + + bool status(false); + + TSMLoc locfield(TSMimeHdrFieldFind(m_buffer, m_lochdr, keystr, keylen)); + + if (nullptr != locfield) { + status = TS_SUCCESS == TSMimeHdrFieldValueStringSet(m_buffer, m_lochdr, locfield, index, valstr, vallen); + } else { + int rcode = TSMimeHdrFieldCreateNamed(m_buffer, m_lochdr, keystr, keylen, &locfield); + + if (TS_SUCCESS == rcode) { + rcode = TSMimeHdrFieldValueStringSet(m_buffer, m_lochdr, locfield, index, valstr, vallen); + if (TS_SUCCESS == rcode) { + rcode = TSMimeHdrFieldAppend(m_buffer, m_lochdr, locfield); + status = (TS_SUCCESS == rcode); + } + } + } + + if (nullptr != locfield) { + TSHandleMLocRelease(m_buffer, m_lochdr, locfield); + } + + return status; +} + +std::string +HttpHeader::toString() const +{ + std::string res; + + if (!isValid()) { + return ""; + } + + TSHttpType const htype(type()); + + switch (htype) { + case TS_HTTP_TYPE_REQUEST: { + res.append(method()); + + int urllen = 0; + char *const urlstr = urlString(&urllen); + if (nullptr != urlstr) { + res.append(" "); + res.append(urlstr, urllen); + TSfree(urlstr); + } else { + res.append(" UnknownURL"); + } + + res.append(" HTTP/unparsed"); + } break; + + case TS_HTTP_TYPE_RESPONSE: { + char bufstr[1024]; + /* + int const version = TSHttpHdrVersionGet(m_buffer, m_lochdr); + snprintf(bufstr, 1023, "%d ", version); + res.append(bufstr); + */ + res.append("HTTP/unparsed"); + + int const status = TSHttpHdrStatusGet(m_buffer, m_lochdr); + snprintf(bufstr, 1023, " %d ", status); + res.append(bufstr); + + int reasonlen = 0; + char const *const hreason = reason(&reasonlen); + + res.append(hreason, reasonlen); + } break; + + default: + case TS_HTTP_TYPE_UNKNOWN: + res.append("UNKNOWN"); + break; + } + + res.append("\r\n"); + + int const numhdrs = TSMimeHdrFieldsCount(m_buffer, m_lochdr); + + for (int indexhdr = 0; indexhdr < numhdrs; ++indexhdr) { + TSMLoc const locfield = TSMimeHdrFieldGet(m_buffer, m_lochdr, indexhdr); + + int keylen = 0; + char const *const keystr = TSMimeHdrFieldNameGet(m_buffer, m_lochdr, locfield, &keylen); + + res.append(keystr, keylen); + res.append(": "); + int vallen = 0; + char const *const valstr = TSMimeHdrFieldValueStringGet(m_buffer, m_lochdr, locfield, -1, &vallen); + + res.append(valstr, vallen); + res.append("\r\n"); + + TSHandleMLocRelease(m_buffer, m_lochdr, locfield); + } + + res.append("\r\n"); + + return res; +} + +/////// HdrMgr + +TSParseResult +HdrMgr::populateFrom(TSHttpParser const http_parser, TSIOBufferReader const reader, HeaderParseFunc const parsefunc, + int64_t *const bytes) +{ + TSParseResult parse_res = TS_PARSE_CONT; + + if (nullptr == m_buffer) { + m_buffer = TSMBufferCreate(); + } + if (nullptr == m_lochdr) { + m_lochdr = TSHttpHdrCreate(m_buffer); + } + + int64_t avail = TSIOBufferReaderAvail(reader); + if (0 < avail) { + TSIOBufferBlock block = TSIOBufferReaderStart(reader); + int64_t consumed = 0; + + parse_res = TS_PARSE_CONT; + + while (nullptr != block && 0 < avail) { + int64_t blockbytes = 0; + char const *const bstart = TSIOBufferBlockReadStart(block, reader, &blockbytes); + + char const *ptr = bstart; + char const *endptr = ptr + blockbytes; + + parse_res = parsefunc(http_parser, m_buffer, m_lochdr, &ptr, endptr); + + int64_t const bytes_parsed(ptr - bstart); + + consumed += bytes_parsed; + avail -= bytes_parsed; + + if (TS_PARSE_CONT == parse_res) { + block = TSIOBufferBlockNext(block); + } else { + break; + } + } + TSIOBufferReaderConsume(reader, consumed); + + if (nullptr != bytes) { + *bytes = consumed; + } + } else if (nullptr != bytes) { + *bytes = 0; + } + + return parse_res; +} diff --git a/plugins/experimental/slice/HttpHeader.h b/plugins/experimental/slice/HttpHeader.h new file mode 100644 index 00000000000..fa3f6cf0549 --- /dev/null +++ b/plugins/experimental/slice/HttpHeader.h @@ -0,0 +1,226 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +/** + An ATS Http header exists in a marshall buffer at a given location. + Unfortunately how that marshall buffer is created and how that + location is determined depends on where those buffers came from. + + A TSHttpTxn manages the buffer itself and creates a location which + has to be managed. + + A TSHttpParsed populates a created buffer that has had TSHttpHdrCreate + run against it which creates a location against it. End users + need to manage the created buffer, the location and invoke + TSHttpHdrDestroy. +*/ + +#include "ts/ts.h" + +#include + +static char const *const SLICER_MIME_FIELD_INFO = "X-Slicer-Info"; + +/** + Designed to be a cheap throwaway struct which allows a + consumer to make various calls to manipulate headers. +*/ +struct HttpHeader { + TSMBuffer const m_buffer; + TSMLoc const m_lochdr; + + explicit HttpHeader(TSMBuffer buffer, TSMLoc lochdr) : m_buffer(buffer), m_lochdr(lochdr) {} + bool + isValid() const + { + return nullptr != m_buffer && nullptr != m_lochdr; + } + + int + byteSize() const + { + if (isValid()) { + return TSHttpHdrLengthGet(m_buffer, m_lochdr); + } else { + return 0; + } + } + + // TS_HTTP_TYPE_UNKNOWN, TS_HTTP_TYPE_REQUEST, TS_HTTP_TYPE_RESPONSE + TSHttpType type() const; + + TSHttpStatus status() const; + + bool setStatus(TSHttpStatus const newstatus); + + bool setUrl(TSMBuffer const bufurl, TSMLoc const locurl); + + typedef char const *(*CharPtrGetFunc)(TSMBuffer, TSMLoc, int *); + + // request method TS_HTTP_METHOD_* + char const * + method(int *const len = nullptr) const + { + return getCharPtr(TSHttpHdrMethodGet, len); + } + + // request method version + int + version() const + { + return TSHttpHdrVersionGet(m_buffer, m_lochdr); + } + + // Returns string representation of the url. Caller gets ownership! + char *urlString(int *const urllen) const; + + // host + char const * + hostname(int *const len) const + { + return getCharPtr(TSHttpHdrHostGet, len); + } + + // response reason + char const * + reason(int *const len) const + { + return getCharPtr(TSHttpHdrReasonGet, len); + } + + bool setReason(char const *const valstr, int const vallen); + + bool hasKey(char const *const key, int const keylen) const; + + // returns false if header invalid or something went wrong with removal. + bool removeKey(char const *const key, int const keylen); + + bool valueForKey(char const *const keystr, int const keylen, + char *const valstr, // <-- return string value + int *const vallen, // <-- pass in capacity, returns len of string + int const index = -1 // retrieves all values + ) const; + + /** + Sets or adds a key/value + */ + bool setKeyVal(char const *const key, int const keylen, char const *const val, int const vallen, + int const index = -1 // sets all values + ); + + /** dump header into provided char buffer + */ + std::string toString() const; + +private: + /** + To be used with + TSHttpHdrMethodGet + TSHttpHdrHostGet + TSHttpHdrReasonGet + */ + char const *getCharPtr(CharPtrGetFunc func, int *const len) const; +}; + +struct TxnHdrMgr { + TxnHdrMgr(TxnHdrMgr const &) = delete; + TxnHdrMgr &operator=(TxnHdrMgr const &) = delete; + + TSMBuffer m_buffer{nullptr}; + TSMLoc m_lochdr{nullptr}; + + TxnHdrMgr() : m_buffer(nullptr), m_lochdr(nullptr) {} + ~TxnHdrMgr() + { + if (nullptr != m_lochdr) { + TSHandleMLocRelease(m_buffer, TS_NULL_MLOC, m_lochdr); + } + } + + typedef TSReturnCode (*HeaderGetFunc)(TSHttpTxn, TSMBuffer *, TSMLoc *); + /** use one of the following: + TSHttpTxnClientReqGet + TSHttpTxnClientRespGet + TSHttpTxnServerReqGet + TSHttpTxnServerRespGet + TSHttpTxnCachedReqGet + TSHttpTxnCachedRespGet + */ + + bool + populateFrom(TSHttpTxn const &txnp, HeaderGetFunc const &func) + { + return TS_SUCCESS == func(txnp, &m_buffer, &m_lochdr); + } + + bool + isValid() const + { + return nullptr != m_lochdr; + } +}; + +struct HdrMgr { + HdrMgr(HdrMgr const &) = delete; + HdrMgr &operator=(HdrMgr const &) = delete; + + TSMBuffer m_buffer{nullptr}; + TSMLoc m_lochdr{nullptr}; + + HdrMgr() : m_buffer(nullptr), m_lochdr(nullptr) {} + ~HdrMgr() + { + if (nullptr != m_buffer) { + if (nullptr != m_lochdr) { + TSHttpHdrDestroy(m_buffer, m_lochdr); + TSHandleMLocRelease(m_buffer, TS_NULL_MLOC, m_lochdr); + } + TSMBufferDestroy(m_buffer); + } + } + + void + resetHeader() + { + if (nullptr != m_buffer && nullptr != m_lochdr) { + TSHttpHdrDestroy(m_buffer, m_lochdr); + TSHandleMLocRelease(m_buffer, TS_NULL_MLOC, m_lochdr); + m_lochdr = nullptr; + } + } + + typedef TSParseResult (*HeaderParseFunc)(TSHttpParser, TSMBuffer, TSMLoc, char const **, char const *); + + /** Clear/create the parser before calling this and don't + use the parser on another header until done with this one. + use one of the following: + TSHttpHdrParseReq + TSHttpHdrParseResp + Call this multiple times if necessary. + */ + TSParseResult populateFrom(TSHttpParser const http_parser, TSIOBufferReader const reader, HeaderParseFunc const parsefunc, + int64_t *const consumed); + + bool + isValid() const + { + return nullptr != m_lochdr; + } +}; diff --git a/plugins/experimental/slice/Makefile.inc b/plugins/experimental/slice/Makefile.inc new file mode 100644 index 00000000000..21f6166f5aa --- /dev/null +++ b/plugins/experimental/slice/Makefile.inc @@ -0,0 +1,64 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +pkglib_LTLIBRARIES += experimental/slice/slice.la + +experimental_slice_slice_la_SOURCES = \ + experimental/slice/client.cc \ + experimental/slice/client.h \ + experimental/slice/Config.cc \ + experimental/slice/Config.h \ + experimental/slice/ContentRange.cc \ + experimental/slice/ContentRange.h \ + experimental/slice/Data.h \ + experimental/slice/HttpHeader.cc \ + experimental/slice/HttpHeader.h \ + experimental/slice/intercept.cc \ + experimental/slice/intercept.h \ + experimental/slice/Range.cc \ + experimental/slice/Range.h \ + experimental/slice/response.cc \ + experimental/slice/response.h \ + experimental/slice/server.cc \ + experimental/slice/server.h \ + experimental/slice/slice.cc \ + experimental/slice/slice.h \ + experimental/slice/Stage.h \ + experimental/slice/transfer.cc \ + experimental/slice/transfer.h \ + experimental/slice/util.cc \ + experimental/slice/util.h + +check_PROGRAMS += experimental/slice/test_content_range + +experimental_slice_test_content_range_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DUNITTEST +experimental_slice_test_content_range_SOURCES = \ + experimental/slice/unit-tests/test_content_range.cc \ + experimental/slice/ContentRange.cc + +check_PROGRAMS += experimental/slice/test_range + +experimental_slice_test_range_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DUNITTEST +experimental_slice_test_range_SOURCES = \ + experimental/slice/unit-tests/test_range.cc \ + experimental/slice/Range.cc + +check_PROGRAMS += experimental/slice/test_config + +experimental_slice_test_config_CPPFLAGS = $(AM_CPPFLAGS) -I$(abs_top_srcdir)/tests/include -DUNITTEST +experimental_slice_test_config_SOURCES = \ + experimental/slice/unit-tests/test_config.cc \ + experimental/slice/Config.cc diff --git a/plugins/experimental/slice/Makefile.tsxs b/plugins/experimental/slice/Makefile.tsxs new file mode 100644 index 00000000000..f8022a1714e --- /dev/null +++ b/plugins/experimental/slice/Makefile.tsxs @@ -0,0 +1,65 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +TSXS?=tsxs +PLUGIN=slice + +all: $(PLUGIN).so + +SOURCES = \ + Config.cc \ + ContentRange.cc \ + HttpHeader.cc \ + Range.cc \ + client.cc \ + intercept.cc \ + response.cc \ + server.cc \ + slice.cc \ + transfer.cc \ + +HEADERS = \ + Config.h \ + ContentRange.h \ + Data.h \ + HttpHeader.h \ + Range.h \ + Stage.h \ + client.h \ + intercept.h \ + response.h \ + server.h \ + slice.h \ + transfer.h \ + +$(PLUGIN).so: $(SOURCES) $(HEADERS) + $(TSXS) -v -o $(PLUGIN).so $(SOURCES) + +install: all + $(TSXS) -v -o $(PLUGIN).so -i + +TSINCLUDE = $(shell tsxs -q INCLUDEDIR) +TSCXX = $(shell tsxs -q CXX) +TSCXXFLAGS = $(shell tsxs -q CXXFLAGS) +#PREFIX = $(shell tsxs -q PREFIX) +#LIBS = -L$(PREFIX)/lib -latscppapi +#LIBS = $(PREFIX)/lib/libtsutil.la + +slice_test: slice_test.cc ContentRange.cc Range.cc + $(TSCXX) -o $@ $^ $(TSCXXFLAGS) -I$(TSINCLUDE) -DUNITTEST + +clean: + rm -fv *.lo *.so diff --git a/plugins/experimental/slice/README.md b/plugins/experimental/slice/README.md new file mode 100644 index 00000000000..6ae1399037a --- /dev/null +++ b/plugins/experimental/slice/README.md @@ -0,0 +1,165 @@ +### Apache Traffic Server - Slicer Plugin + +The purpose of this plugin is to slice full file or range based requests +into deterministic chunks. This allows a large file to be spread across +multiple cache stripes and allows range requests to be satisfied by +stitching these chunks together. + +Deterministic chunks are requested from a parent cache or origin server +using a preconfigured block byte size. + +The plugin is an example of an intercept handler which takes a single +incoming request (range or whole asset), breaks it into a sequence +of block requests and assembles those blocks into a client response. +The plugin uses TSHttpConnect to delegate each block request to +cache_range_requests.so which handles all cache and parent interaction. + +To enable the plugin, specify the plugin library via @plugin at the end +of a remap line as follows (default 1MB slice in this example): + +``` +map http://ats-cache/ http://parent/ @plugin=slice.so @plugin=cache_range_requests.so +map https://ats-cache/ http://parent/ @plugin=slice.so @plugin=cache_range_requests.so +``` + +alternatively (2MB slice block) + +``` +map http://ats-cache/ http://parent/ @plugin=slice.so @pparam=-b @pparam=2M @plugin=cache_range_requests.so +map https://ats-cache/ http://parent/ @plugin=slice.so @pparam=--blockbytes=2M @plugin=cache_range_requests.so +``` + +Options for the slice plugin (typically last one wins): +``` +--blockbytes= (optional) + Slice block size. + Default is 1m or 1048576 bytes. + also -b + Suffix k,m,g supported. + Limited to 32k and 32m inclusive. + For backwards compatibility blockbytes: is also supported. + +--blockbytes-test= (optional) + Slice block size for testing. + also -t + Suffix k,m,g supported. + Limited to any positive number. + Ignored if --blockbytes is provided. + +--remap-host= (optional) + Uses effective url with given host and port 0 for remapping. + Requires setting up an intermediate loopback remap rule. + -r for short + +--pace-errorlog= (optional) + Limit stitching error logs to every 'n' second(s) + Default is to log all errors (no pacing). + also -e + +--disable-errorlog (optional) + Disable writing stitching errors to the error log. + also -d +``` + +By default the plugin uses the pristine url to loopback call back +into the same rule as each range slice is issued. The effective url +with loopback remap host may be used by adding the '-r ' or +'--remap-host=' plugin option. + +Using the `--remap-host` option splits the plugin chain into 2 remap rules. +One remap rule for all the incoming requests and the other for just the block +range requests. This allows for easier trouble shooting via logs and +also allows for more effecient plugin rules. The default pristine method +runs the remap plugins twice, one for the incoming request and one for +eace slice. Splitting the rules allows for plugins like URI signing to +be done on the client request only. + +NOTE: Requests NOT handled by the slice plugin (ie: HEAD requests) are +handled as with a typical remap rule. GET requests intercepted by the +slice plugin are virtually reissued into ATS and are forward proxied +through the cache_range_requests plugin. + +``` +map http://ats/ http://parent/ @plugin=slice.so @pparam=--blockbytes=512k @pparam=--remap-host=loopback +map https://ats/ https://parent/ @plugin=slice.so @pparam=--blockbytes=512k @pparam=--remap-host=loopback + +# Virtual forward proxy for slice range blocks +map http://loopback/ http://parent/ @plugin=cache_range_requests.so +map https://loopback/ http://parent/ @plugin=cache_range_requests.so +``` + +**Note**: For default pristine behavior cache_range_requests **MUST** +follow slice.so Put these plugins at the end of the plugin list + +**Note**: blockbytes is defined in bytes. Postfix for 'K', 'M' and 'G' +may be used. 1048576 (1MB) is the default. + +For testing purposes an unchecked value of "blockbytes-test" is also available. + +Debug output can be enable by setting the debug tag: **slice**. If debug +is enabled all block stitch errors will log to diags.log + +The slice plugin is susceptible to block stitching errors caused by +mismatched blocks. For these cases special detailed error logs are +provided to help with debugging. Below is a sample error log entry:: + +``` +[Apr 19 20:26:13.639] [ET_NET 17] ERROR: [slice] 1555705573.639 reason="Non 206 internal block response" uri="http://localhost:18080/%7Ep.tex/%7Es.50M/%7Eui.20000/" uas="curl/7.29.0" req_range="bytes=1000000-" norm_range="bytes 1000000-52428799/52428800" etag_exp="%221603934496%22" lm_exp="Fri, 19 Apr 2019 18:53:20 GMT" blk_range="21000000-21999999" status_got="400" cr_got="" etag_got="" lm_got="" cc="no-store" via="" +``` + +Current error types logged: +``` + Mismatch block Etag + Mismatch block Last-Modified + Non 206 internal block response + Mismatch/Bad block Content-Range +``` + + +With slice error logs disabled these type errors can typically be detected +by observing crc=ERR_READ_ERROR and pscl=0 in normal logs. + +At the current time only single range requests or the first part of a +multi part range request of the forms: +``` +Range: bytes=- +Range: bytes=- +Range: bytes=- +``` +are supported as multi part range responses are non trivial to implement. +This matches with the cache_range_requests.so plugin capability. +--- + +Important things to note: + +Any first block server response that is not a 206 is passed down to +the client. + +Only the first server response block is used to evaluate any "If-" +headers. Subsequent server slice block requests remove these headers. + +If a client aborts mid transaction the current slice block is completed +to ensure that the block is written to cache. + +The only 416 case this plugin handles itself is if the requested range +is inside the end slice block but past the content length. Otherwise +parents seem to properly issue 416 responses themselves. + +--- + +To manually build the plugin use the "tsxs" executable that installs with +traffic_server. + +Running the following command will build the plugin + +``` +tsxs -v -o slice.so *.cc +``` + +Running the following command will build and install the plugin. +Beware this may crash a running system if the plugin is loaded +and the OS uses memory paging with plugins. + +``` +tsxs -v -i -o slice.so *.cc +``` diff --git a/plugins/experimental/slice/Range.cc b/plugins/experimental/slice/Range.cc new file mode 100644 index 00000000000..b52e8b66e40 --- /dev/null +++ b/plugins/experimental/slice/Range.cc @@ -0,0 +1,196 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "Range.h" +#include "slice.h" + +#include +#include +#include +#include +#include + +bool +Range::isValid() const +{ + return m_beg < m_end && (0 <= m_beg || 0 == m_end); +} + +int64_t +Range::size() const +{ + return m_end - m_beg; +} + +bool +Range::fromStringClosed(char const *const rangestr) +{ + static char const *const BYTESTR = "bytes="; + static size_t const BYTESTRLEN = strlen(BYTESTR); + + m_beg = m_end = -1; // initialize invalid + + // make sure this is in byte units + if (0 != strncmp(BYTESTR, rangestr, BYTESTRLEN)) { + return false; + } + + // advance past any white space + char const *pstr = rangestr + BYTESTRLEN; + while ('\0' != *pstr && isblank(*pstr)) { + ++pstr; + } + + // rip out any whitespace + char rangebuf[1024]; + int const rangelen = sizeof(rangebuf); + char *pbuf = rangebuf; + while ('\0' != *pstr && (pbuf - rangebuf) < rangelen) { + if (!isblank(*pstr)) { + *pbuf++ = *pstr; + } + ++pstr; + } + *pbuf = '\0'; + + int const rlen = (pbuf - rangebuf); + + int consumed = 0; + + // last 'n' bytes - result in range with negative begin and 0 end + int64_t endbytes = 0; + char const *const fmtend = "-%" PRId64 "%n"; + int const fieldsend = sscanf(rangebuf, fmtend, &endbytes, &consumed); + if (1 == fieldsend) { + if (rlen == consumed) { + m_beg = -endbytes; + m_end = 0; + return true; + } else { + return false; + } + } + + // normal range - + char const *const fmtclosed = "%" PRId64 "-%" PRId64 "%n"; + int64_t front = 0; + int64_t back = 0; + + int const fieldsclosed = sscanf(rangebuf, fmtclosed, &front, &back, &consumed); + if (2 == fieldsclosed) { + if (0 <= front && front <= back && rlen == consumed) { + m_beg = front; + m_end = back + 1; + return true; + } else { + return false; + } + } + + front = 0; + char const *const fmtbeg = "%" PRId64 "-%n"; + int const fieldsbeg = sscanf(rangebuf, fmtbeg, &front, &consumed); + if (1 == fieldsbeg) { + if (rlen == consumed) { + m_beg = front; + m_end = Range::maxval; + return true; + } else { + return false; + } + } + + return false; +} // parseRange + +bool +Range::toStringClosed(char *const bufstr, + int *const buflen // returns actual bytes used + ) const +{ + if (!isValid()) { + if (0 < *buflen) { + bufstr[0] = '\0'; + } + *buflen = 0; + return false; + } + + int const lenin = *buflen; + + if (m_end <= Range::maxval) { + *buflen = snprintf(bufstr, lenin, "bytes=%" PRId64 "-%" PRId64, m_beg, m_end - 1); + } else { + *buflen = snprintf(bufstr, lenin, "bytes=%" PRId64 "-", m_beg); + } + + return (0 < *buflen && *buflen < lenin); +} + +int64_t +Range::firstBlockFor(int64_t const blocksize) const +{ + if (0 < blocksize && isValid()) { + return std::max(static_cast(0), m_beg / blocksize); + } else { + return -1; + } +} + +int64_t +Range::lastBlockFor(int64_t const blocksize) const +{ + if (0 < blocksize && isValid()) { + return std::max((int64_t)0, (m_end - 1) / blocksize); + } else { + return -1; + } +} + +Range +Range::intersectedWith(Range const &other) const +{ + return Range(std::max(m_beg, other.m_beg), std::min(m_end, other.m_end)); +} + +bool +Range::blockIsInside(int64_t const blocksize, int64_t const blocknum) const +{ + Range const blockrange(blocksize * blocknum, blocksize * (blocknum + 1)); + Range const isec(blockrange.intersectedWith(*this)); + + return isec.isValid(); +} + +int64_t +Range::skipBytesForBlock(int64_t const blocksize, int64_t const blocknum) const +{ + int64_t const blockstart(blocksize * blocknum); + + if (m_beg < blockstart) { + return 0; + } else { + return m_beg - blockstart; + } +} + +bool +Range::isEndBytes() const +{ + return m_beg < 0 && 0 == m_end; +} diff --git a/plugins/experimental/slice/Range.h b/plugins/experimental/slice/Range.h new file mode 100644 index 00000000000..8c8826d2091 --- /dev/null +++ b/plugins/experimental/slice/Range.h @@ -0,0 +1,78 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +#include + +/** + represents a value parsed from a Range request header field. + Range is converted from a closed range into a half open. + */ + +struct Range { +public: + static int64_t constexpr maxval = (std::numeric_limits::max() >> 2); + + int64_t m_beg = -1; + int64_t m_end = -1; // half open + + Range() {} + explicit Range(int64_t const begin, int64_t const end) : m_beg(begin), m_end(end) {} + + bool isValid() const; + + int64_t size() const; + + /** parse a from a closed request range into a half open range + * This will only correctly handle the *first* range that is + * parsed via TSMimeHdrFieldValueStringGet with index '0'. + * Range representing last N bytes will be coded as (-N, 0) + */ + bool fromStringClosed(char const *const rangestr); + + /** parse a from a closed request range into a half open range + */ + bool toStringClosed(char *const rangestr, int *const rangelen) const; + + /** block number of first range block + */ + int64_t firstBlockFor(int64_t const blockbytes) const; + + /** block number of last (inclusive) range block + */ + int64_t lastBlockFor(int64_t const blockbytes) const; + + /** block intersection + */ + Range intersectedWith(Range const &other) const; + + /** is the given block inside held range? + */ + bool blockIsInside(int64_t const blocksize, int64_t const blocknum) const; + + /** number of skip bytes for the given block + */ + int64_t skipBytesForBlock(int64_t const blocksize, int64_t const blocknum) const; + + /** is this coded to indicate last N bytes? + */ + bool isEndBytes() const; +}; diff --git a/plugins/experimental/slice/Stage.h b/plugins/experimental/slice/Stage.h new file mode 100644 index 00000000000..f677d649614 --- /dev/null +++ b/plugins/experimental/slice/Stage.h @@ -0,0 +1,190 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +#include "slice.h" +#include "util.h" + +#include + +struct Channel { + TSVIO m_vio{nullptr}; + TSIOBuffer m_iobuf{nullptr}; + TSIOBufferReader m_reader{nullptr}; + + ~Channel() + { + if (nullptr != m_reader) { + TSIOBufferReaderFree(m_reader); +#if defined(COLLECT_STATS) + TSStatIntDecrement(stats::Reader, 1); +#endif + } + if (nullptr != m_iobuf) { + TSIOBufferDestroy(m_iobuf); + } + } + + int64_t + drainReader() + { + int64_t consumed = 0; + + if (nullptr != m_reader && reader_avail_more_than(m_reader, 0)) { + int64_t const avail = TSIOBufferReaderAvail(m_reader); + TSIOBufferReaderConsume(m_reader, avail); + consumed = avail; + TSVIONDoneSet(m_vio, TSVIONDoneGet(m_vio) + consumed); + } + + return consumed; + } + + bool + setForRead(TSVConn vc, TSCont contp, int64_t const bytesin) + { + TSAssert(nullptr != vc); + if (nullptr == m_iobuf) { + m_iobuf = TSIOBufferCreate(); + m_reader = TSIOBufferReaderAlloc(m_iobuf); +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::Reader, 1); +#endif + } else { + int64_t const drained = drainReader(); + if (0 < drained) { + DEBUG_LOG("Drained from reader: %" PRId64, drained); + } + } + m_vio = TSVConnRead(vc, contp, m_iobuf, bytesin); + return nullptr != m_vio; + } + + bool + setForWrite(TSVConn vc, TSCont contp, int64_t const bytesout) + { + TSAssert(nullptr != vc); + if (nullptr == m_iobuf) { + m_iobuf = TSIOBufferCreate(); + m_reader = TSIOBufferReaderAlloc(m_iobuf); +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::Reader, 1); +#endif + } else { + int64_t const drained = drainReader(); + if (0 < drained) { + DEBUG_LOG("Drained from reader: %" PRId64, drained); + } + } + m_vio = TSVConnWrite(vc, contp, m_reader, bytesout); + return nullptr != m_vio; + } + + void + close() + { + if (nullptr != m_reader) { + drainReader(); + } + m_vio = nullptr; + } + + bool + isOpen() const + { + return nullptr != m_vio; + } + + bool + isDrained() const + { + return nullptr == m_reader || !reader_avail_more_than(m_reader, 0); + } +}; + +struct Stage // upstream or downstream (server or client) +{ + Stage(Stage const &) = delete; + Stage &operator=(Stage const &) = delete; + + TSVConn m_vc{nullptr}; + Channel m_read; + Channel m_write; + + Stage() {} + ~Stage() + { + if (nullptr != m_vc) { + TSVConnClose(m_vc); + } + } + + void + setupConnection(TSVConn vc) + { + if (nullptr != m_vc) { + TSVConnClose(m_vc); + } + m_read.close(); + m_write.close(); + m_vc = vc; + } + + void + setupVioRead(TSCont contp, int64_t const bytesin) + { + m_read.setForRead(m_vc, contp, bytesin); + } + + void + setupVioWrite(TSCont contp, int64_t const bytesout) + { + m_write.setForWrite(m_vc, contp, bytesout); + } + + void + abort() + { + if (nullptr != m_vc) { + TSVConnAbort(m_vc, TS_VC_CLOSE_ABORT); + m_vc = nullptr; + } + m_read.close(); + m_write.close(); + } + + void + close() + { + if (nullptr != m_vc) { + TSVConnClose(m_vc); + m_vc = nullptr; + } + m_read.close(); + m_write.close(); + } + + bool + isOpen() const + { + return nullptr != m_vc && (m_read.isOpen() || m_write.isOpen()); + } +}; diff --git a/plugins/experimental/slice/client.cc b/plugins/experimental/slice/client.cc new file mode 100644 index 00000000000..5d4bde5e690 --- /dev/null +++ b/plugins/experimental/slice/client.cc @@ -0,0 +1,165 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "client.h" + +#include "Config.h" +#include "util.h" + +#include + +// this is called once per transaction when the client sends a req header +bool +handle_client_req(TSCont contp, TSEvent event, Data *const data) +{ + switch (event) { + case TS_EVENT_VCONN_READ_READY: + case TS_EVENT_VCONN_READ_COMPLETE: { + if (nullptr == data->m_http_parser) { + data->m_http_parser = TSHttpParserCreate(); + } + + // Read the header from the buffer + int64_t consumed = 0; + if (TS_PARSE_DONE != + data->m_req_hdrmgr.populateFrom(data->m_http_parser, data->m_dnstream.m_read.m_reader, TSHttpHdrParseReq, &consumed)) { + return false; + } + + // update the VIO + TSVIO const input_vio = data->m_dnstream.m_read.m_vio; + TSVIONDoneSet(input_vio, TSVIONDoneGet(input_vio) + consumed); + + // make the header manipulator + HttpHeader header(data->m_req_hdrmgr.m_buffer, data->m_req_hdrmgr.m_lochdr); + + // set the request url back to pristine in case of plugin stacking + header.setUrl(data->m_urlbuf, data->m_urlloc); + + header.setKeyVal(TS_MIME_FIELD_HOST, TS_MIME_LEN_HOST, data->m_hostname, data->m_hostlen); + + // default: whole file (unknown, wait for first server response) + Range rangebe; + + char rangestr[1024]; + int rangelen = sizeof(rangestr); + bool const hasRange = header.valueForKey(TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, rangestr, &rangelen, + 0); // <-- first range only + if (hasRange) { + // write parsed header into slicer meta tag + header.setKeyVal(SLICER_MIME_FIELD_INFO, strlen(SLICER_MIME_FIELD_INFO), rangestr, rangelen); + bool const isRangeGood = rangebe.fromStringClosed(rangestr); + + if (isRangeGood) { + DEBUG_LOG("%p Partial content request", data); + data->m_statustype = TS_HTTP_STATUS_PARTIAL_CONTENT; + } else // signal a 416 needs to be formed and sent + { + DEBUG_LOG("%p Ill formed/unhandled range: %s", data, rangestr); + data->m_statustype = TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE; + + // First block will give Content-Length + rangebe = Range(0, data->m_config->m_blockbytes); + } + } else { + DEBUG_LOG("%p Full content request", data); + static char const *const valstr = "-"; + static size_t const vallen = strlen(valstr); + header.setKeyVal(SLICER_MIME_FIELD_INFO, strlen(SLICER_MIME_FIELD_INFO), valstr, vallen); + data->m_statustype = TS_HTTP_STATUS_OK; + rangebe = Range(0, Range::maxval); + } + + // set to the first block in range + data->m_blocknum = rangebe.firstBlockFor(data->m_config->m_blockbytes); + data->m_req_range = rangebe; + + // remove ATS keys to avoid 404 loop + header.removeKey(TS_MIME_FIELD_VIA, TS_MIME_LEN_VIA); + header.removeKey(TS_MIME_FIELD_X_FORWARDED_FOR, TS_MIME_LEN_X_FORWARDED_FOR); + + // send the first block request to server + if (!request_block(contp, data)) { + abort(contp, data); + return false; + } + + // for subsequent blocks remove any conditionals which may fail + // an optimization would be to wait until the first block succeeds + header.removeKey(TS_MIME_FIELD_IF_MATCH, TS_MIME_LEN_IF_MATCH); + header.removeKey(TS_MIME_FIELD_IF_MODIFIED_SINCE, TS_MIME_LEN_IF_MODIFIED_SINCE); + header.removeKey(TS_MIME_FIELD_IF_NONE_MATCH, TS_MIME_LEN_IF_NONE_MATCH); + header.removeKey(TS_MIME_FIELD_IF_RANGE, TS_MIME_LEN_IF_RANGE); + header.removeKey(TS_MIME_FIELD_IF_UNMODIFIED_SINCE, TS_MIME_LEN_IF_UNMODIFIED_SINCE); + } break; + default: { + DEBUG_LOG("%p handle_client_req unhandled event %d %s", data, event, TSHttpEventNameLookup(event)); + } break; + } + + return true; +} + +// this is when the client starts asking us for more data +void +handle_client_resp(TSCont contp, TSEvent event, Data *const data) +{ + DEBUG_LOG("%p handle_client_resp %s", data, TSHttpEventNameLookup(event)); + +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::Client, 1); +#endif + + switch (event) { + case TS_EVENT_VCONN_WRITE_READY: { + if (Data::BlockState::Pending == data->m_blockstate) { + bool start_next_block = true; + + if (data->m_config->m_throttle) { + TSVIO const output_vio = data->m_dnstream.m_write.m_vio; + int64_t const output_done = TSVIONDoneGet(output_vio); + int64_t const output_sent = data->m_bytessent; + int64_t const threshout = data->m_config->m_blockbytes; + + if (threshout < (output_done - output_sent)) { + start_next_block = false; + DEBUG_LOG("%p handle_client_resp: throttling %" PRId64, data, (output_done - output_sent)); + } + } + + if (start_next_block) { + request_block(contp, data); + } + } + } break; + case TS_EVENT_VCONN_WRITE_COMPLETE: { + if (TSIsDebugTagSet(PLUGIN_NAME) && reader_avail_more_than(data->m_upstream.m_read.m_reader, 0)) { + int64_t const left = TSIOBufferReaderAvail(data->m_upstream.m_read.m_reader); + DEBUG_LOG("%p WRITE_COMPLETE called with %" PRId64 " bytes left", data, left); + } + + data->m_dnstream.close(); + if (!data->m_upstream.m_read.isOpen()) { + shutdown(contp, data); + } + } break; + default: { + DEBUG_LOG("%p handle_client_resp unhandled event %d %s", data, event, TSHttpEventNameLookup(event)); + } break; + } +} diff --git a/plugins/experimental/slice/client.h b/plugins/experimental/slice/client.h new file mode 100644 index 00000000000..b91516165d8 --- /dev/null +++ b/plugins/experimental/slice/client.h @@ -0,0 +1,36 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "Data.h" + +#include "ts/ts.h" + +/** Functions to deal with the connection to the client. + * Body content transfers are handled by the client. + * New block requests are also initiated by the client. + */ + +bool requestBlock(TSCont contp, Data *const data); + +/** returns true if the incoming vio can be turned off + */ +bool handle_client_req(TSCont contp, TSEvent event, Data *const data); + +void handle_client_resp(TSCont contp, TSEvent event, Data *const data); diff --git a/plugins/experimental/slice/intercept.cc b/plugins/experimental/slice/intercept.cc new file mode 100644 index 00000000000..2b6ef3bdfcd --- /dev/null +++ b/plugins/experimental/slice/intercept.cc @@ -0,0 +1,82 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "intercept.h" + +#include "Data.h" +#include "client.h" +#include "server.h" +#include "slice.h" + +int +intercept_hook(TSCont contp, TSEvent event, void *edata) +{ + Data *const data = static_cast(TSContDataGet(contp)); + + if (nullptr == data) { + ERROR_LOG("intercept_hook called without data"); + TSContDestroy(contp); + return TS_EVENT_ERROR; + } + + // After the initial TS_EVENT_NET_ACCEPT + // any "events" will be handled by the vio read or write channel handler + switch (event) { + case TS_EVENT_NET_ACCEPT: { + // set up reader from client + TSVConn const downvc = static_cast(edata); + data->m_dnstream.setupConnection(downvc); + data->m_dnstream.setupVioRead(contp, INT64_MAX); + } break; + + case TS_EVENT_NET_ACCEPT_FAILED: + case TS_EVENT_VCONN_INACTIVITY_TIMEOUT: + case TS_EVENT_VCONN_ACTIVE_TIMEOUT: + case TS_EVENT_ERROR: { + abort(contp, data); + } break; + + default: { + // data from client -- only the initial header + if (data->m_dnstream.m_read.isOpen() && edata == data->m_dnstream.m_read.m_vio) { + if (handle_client_req(contp, event, data)) { + // DEBUG_LOG("shutting down read from client pipe"); + TSVConnShutdown(data->m_dnstream.m_vc, 1, 0); + } + } + // server wants more data from us, should never happen + // every time TSHttpConnect is called this resets + else if (data->m_upstream.m_write.isOpen() && edata == data->m_upstream.m_write.m_vio) { + // DEBUG_LOG("shutting down send to server pipe"); + TSVConnShutdown(data->m_upstream.m_vc, 0, 1); + } + // server has data for us + else if (data->m_upstream.m_read.isOpen() && edata == data->m_upstream.m_read.m_vio) { + handle_server_resp(contp, event, data); + } + // client wants more data from us, only body content + else if (data->m_dnstream.m_write.isOpen() && edata == data->m_dnstream.m_write.m_vio) { + handle_client_resp(contp, event, data); + } else { + ERROR_LOG("Unhandled event: %d", event); + } + } break; + } + + return TS_EVENT_CONTINUE; +} diff --git a/plugins/experimental/slice/intercept.h b/plugins/experimental/slice/intercept.h new file mode 100644 index 00000000000..3b59cef30f9 --- /dev/null +++ b/plugins/experimental/slice/intercept.h @@ -0,0 +1,23 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +int intercept_hook(TSCont contp, TSEvent event, void *edata); diff --git a/plugins/experimental/slice/response.cc b/plugins/experimental/slice/response.cc new file mode 100644 index 00000000000..410b1507dfe --- /dev/null +++ b/plugins/experimental/slice/response.cc @@ -0,0 +1,107 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "response.h" + +#include +#include +#include + +#include "ts/ts.h" + +// canned body string for a 416, stolen from nginx +std::string const & +bodyString416() +{ + static std::string bodystr; + static std::mutex mutex; + std::lock_guard const guard(mutex); + + if (bodystr.empty()) { + bodystr.append("\n"); + bodystr.append("416 Requested Range Not Satisfiable\n"); + bodystr.append("\n"); + bodystr.append("

416 Requested Range Not Satisfiable

"); + bodystr.append("
ATS/"); + bodystr.append(TS_VERSION_STRING); + bodystr.append("
\n"); + bodystr.append("\n"); + bodystr.append("\n"); + } + + return bodystr; +} + +// Form a 502 response, preliminary +std::string const & +string502() +{ + static std::string msg; + static std::mutex mutex; + std::lock_guard const guard(mutex); + + if (msg.empty()) { + std::string bodystr; + bodystr.append("\n"); + bodystr.append("502 Bad Gateway\n"); + bodystr.append("\n"); + bodystr.append("

502 Bad Gateway: Missing/Malformed " + "Content-Range

"); + bodystr.append("
ATS/"); + bodystr.append(TS_VERSION_STRING); + bodystr.append("
\n"); + bodystr.append("\n"); + bodystr.append("\n"); + + char clenstr[1024]; + int const clen = snprintf(clenstr, sizeof(clenstr), "%lu", bodystr.size()); + + msg.append("HTTP/1.1 502 Bad Gateway\r\n"); + msg.append("Content-Length: "); + msg.append(clenstr, clen); + msg.append("\r\n"); + + msg.append("\r\n"); + msg.append(bodystr); + } + + return msg; +} + +void +form416HeaderAndBody(HttpHeader &header, int64_t const contentlen, std::string const &bodystr) +{ + header.removeKey(TS_MIME_FIELD_LAST_MODIFIED, TS_MIME_LEN_LAST_MODIFIED); + header.removeKey(TS_MIME_FIELD_ETAG, TS_MIME_LEN_ETAG); + header.removeKey(TS_MIME_FIELD_ACCEPT_RANGES, TS_MIME_LEN_ACCEPT_RANGES); + + header.setStatus(TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE); + char const *const reason = TSHttpHdrReasonLookup(TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE); + header.setReason(reason, strlen(reason)); + + char bufstr[256]; + int buflen = snprintf(bufstr, sizeof(bufstr), "%lu", bodystr.size()); + header.setKeyVal(TS_MIME_FIELD_CONTENT_LENGTH, TS_MIME_LEN_CONTENT_LENGTH, bufstr, buflen); + + static char const *const ctypestr = "text/html"; + static int const ctypelen = strlen(ctypestr); + header.setKeyVal(TS_MIME_FIELD_CONTENT_TYPE, TS_MIME_LEN_CONTENT_TYPE, ctypestr, ctypelen); + + buflen = snprintf(bufstr, 255, "*/%" PRId64, contentlen); + header.setKeyVal(TS_MIME_FIELD_CONTENT_RANGE, TS_MIME_LEN_CONTENT_RANGE, bufstr, buflen); +} diff --git a/plugins/experimental/slice/response.h b/plugins/experimental/slice/response.h new file mode 100644 index 00000000000..925db65f725 --- /dev/null +++ b/plugins/experimental/slice/response.h @@ -0,0 +1,28 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "HttpHeader.h" +#include + +std::string const &string502(); + +std::string const &bodyString416(); + +void form416HeaderAndBody(HttpHeader &header, int64_t const contentlen, std::string const &bodystr); diff --git a/plugins/experimental/slice/server.cc b/plugins/experimental/slice/server.cc new file mode 100644 index 00000000000..5356806a52f --- /dev/null +++ b/plugins/experimental/slice/server.cc @@ -0,0 +1,467 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "server.h" + +#include "Config.h" +#include "ContentRange.h" +#include "response.h" +#include "transfer.h" +#include "util.h" + +#include "ts/experimental.h" + +#include + +namespace +{ +ContentRange +contentRangeFrom(HttpHeader const &header) +{ + ContentRange bcr; + + /* Pull content length off the response header + and manipulate it into a client response header + */ + char rangestr[1024]; + int rangelen = sizeof(rangestr); + + // look for expected Content-Range field + bool const hasContentRange(header.valueForKey(TS_MIME_FIELD_CONTENT_RANGE, TS_MIME_LEN_CONTENT_RANGE, rangestr, &rangelen)); + + if (!hasContentRange) { + DEBUG_LOG("invalid response header, no Content-Range"); + } else if (!bcr.fromStringClosed(rangestr)) { + DEBUG_LOG("invalid response header, malformed Content-Range, %s", rangestr); + } + + return bcr; +} + +bool +handleFirstServerHeader(Data *const data, TSCont const contp) +{ + HttpHeader header(data->m_resp_hdrmgr.m_buffer, data->m_resp_hdrmgr.m_lochdr); + + // DEBUG_LOG("First header\n%s", header.toString().c_str()); + + data->m_dnstream.setupVioWrite(contp, INT64_MAX); + + TSVIO const output_vio = data->m_dnstream.m_write.m_vio; + TSIOBuffer const output_buf = data->m_dnstream.m_write.m_iobuf; + + // only process a 206, everything else gets a (possibly incomplete) + // pass through + if (TS_HTTP_STATUS_PARTIAL_CONTENT != header.status()) { + DEBUG_LOG("Initial response other than 206: %d", header.status()); + + // Should run TSVIONSetBytes(output_io, hlen + bodybytes); + // int const hlen = TSHttpHdrLengthGet(header.m_buffer, header.m_lochdr); + // TSVIONBytesSet(output_vio, hlen); + TSHttpHdrPrint(header.m_buffer, header.m_lochdr, output_buf); + transfer_all_bytes(data); + + return false; + } + + ContentRange const blockcr = contentRangeFrom(header); + // 206 with bad content range? + if (!blockcr.isValid()) { + static std::string const &msg502 = string502(); + TSVIONBytesSet(output_vio, msg502.size()); + TSIOBufferWrite(output_buf, msg502.data(), msg502.size()); + TSVIOReenable(output_vio); + return false; + } + + // set the resource content length from block response + data->m_contentlen = blockcr.m_length; + + // special case last N bytes + if (data->m_req_range.isEndBytes()) { + data->m_req_range.m_end += data->m_contentlen; + data->m_req_range.m_beg += data->m_contentlen; + data->m_req_range.m_beg = std::max(static_cast(0), data->m_req_range.m_beg); + } else { + // fix up request range end now that we have the content length + data->m_req_range.m_end = std::min(data->m_contentlen, data->m_req_range.m_end); + } + + int64_t const bodybytes = data->m_req_range.size(); + + // range begins past end of data but inside last block, send 416 + bool const send416 = (bodybytes <= 0 || TS_HTTP_STATUS_REQUESTED_RANGE_NOT_SATISFIABLE == data->m_statustype); + if (send416) { + std::string const &bodystr = bodyString416(); + form416HeaderAndBody(header, data->m_contentlen, bodystr); + + int const hlen = TSHttpHdrLengthGet(header.m_buffer, header.m_lochdr); + int64_t const blen = bodystr.size(); + + TSVIONBytesSet(output_vio, int64_t(hlen) + blen); + TSHttpHdrPrint(header.m_buffer, header.m_lochdr, output_buf); + TSIOBufferWrite(output_buf, bodystr.data(), bodystr.size()); + TSVIOReenable(output_vio); + + data->m_upstream.m_read.close(); + + return false; + } + + // save weak cache header identifiers (rfc7232 section 2) + data->m_etaglen = sizeof(data->m_etag) - 1; + header.valueForKey(TS_MIME_FIELD_ETAG, TS_MIME_LEN_ETAG, data->m_etag, &data->m_etaglen); + data->m_lastmodifiedlen = sizeof(data->m_lastmodified) - 1; + header.valueForKey(TS_MIME_FIELD_LAST_MODIFIED, TS_MIME_LEN_LAST_MODIFIED, data->m_lastmodified, &data->m_lastmodifiedlen); + + // size of the first block payload + data->m_blockexpected = blockcr.rangeSize(); + + // Now we can set up the expected client response + if (TS_HTTP_STATUS_PARTIAL_CONTENT == data->m_statustype) { + ContentRange respcr; + respcr.m_beg = data->m_req_range.m_beg; + respcr.m_end = data->m_req_range.m_end; + respcr.m_length = data->m_contentlen; + + char rangestr[1024]; + int rangelen = sizeof(rangestr); + bool const crstat = respcr.toStringClosed(rangestr, &rangelen); + + // corner case, return 500 ?? + if (!crstat) { + data->m_upstream.close(); + data->m_dnstream.close(); + + ERROR_LOG("Bad/invalid response content range"); + return false; + } + + header.setKeyVal(TS_MIME_FIELD_CONTENT_RANGE, TS_MIME_LEN_CONTENT_RANGE, rangestr, rangelen); + } else if (TS_HTTP_STATUS_OK == data->m_statustype) { + header.setStatus(TS_HTTP_STATUS_OK); + static char const *const reason = TSHttpHdrReasonLookup(TS_HTTP_STATUS_OK); + header.setReason(reason, strlen(reason)); + header.removeKey(TS_MIME_FIELD_CONTENT_RANGE, TS_MIME_LEN_CONTENT_RANGE); + } + + char bufstr[1024]; + int const buflen = snprintf(bufstr, sizeof(bufstr), "%" PRId64, bodybytes); + header.setKeyVal(TS_MIME_FIELD_CONTENT_LENGTH, TS_MIME_LEN_CONTENT_LENGTH, bufstr, buflen); + + // add the response header length to the total bytes to send + int const hbytes = TSHttpHdrLengthGet(header.m_buffer, header.m_lochdr); + + TSVIONBytesSet(output_vio, hbytes + bodybytes); + data->m_bytestosend = hbytes + bodybytes; + TSHttpHdrPrint(header.m_buffer, header.m_lochdr, output_buf); + data->m_bytessent = hbytes; + TSVIOReenable(output_vio); + + return true; +} + +void +logSliceError(char const *const message, Data const *const data, HttpHeader const &header_resp) +{ + Config *const config = data->m_config; + + bool const logToError = config->canLogError(); + + // always write block stitch errors while in debug mode + if (!logToError && !TSIsDebugTagSet(PLUGIN_NAME)) { + return; + } + + HttpHeader const header_req(data->m_req_hdrmgr.m_buffer, data->m_req_hdrmgr.m_lochdr); + + TSHRTime const timenowus = TShrtime(); + int64_t const msecs = timenowus / 1000000; + int64_t const secs = msecs / 1000; + int64_t const ms = msecs % 1000; + + // Gather information on the request, must delete urlstr + int urllen = 0; + char *const urlstr = header_req.urlString(&urllen); + + char urlpstr[16384]; + size_t urlplen = sizeof(urlpstr); + TSStringPercentEncode(urlstr, urllen, urlpstr, urlplen, &urlplen, nullptr); + + if (nullptr != urlstr) { + TSfree(urlstr); + } + + // uas + char uasstr[8192]; + int uaslen = sizeof(uasstr); + header_req.valueForKey(TS_MIME_FIELD_USER_AGENT, TS_MIME_LEN_USER_AGENT, uasstr, &uaslen); + + // raw range request + char rangestr[1024]; + int rangelen = sizeof(rangestr); + header_req.valueForKey(SLICER_MIME_FIELD_INFO, strlen(SLICER_MIME_FIELD_INFO), rangestr, &rangelen); + + // Normalized range request + ContentRange const crange(data->m_req_range.m_beg, data->m_req_range.m_end, data->m_contentlen); + char normstr[1024]; + int normlen = sizeof(normstr); + crange.toStringClosed(normstr, &normlen); + + // block range request + int64_t const blockbeg = data->m_blocknum * data->m_config->m_blockbytes; + int64_t const blockend = std::min(blockbeg + data->m_config->m_blockbytes, data->m_contentlen); + + // Block response data + TSHttpStatus const statusgot = header_resp.status(); + + // content range + char crstr[1024]; + int crlen = sizeof(crstr); + header_resp.valueForKey(TS_MIME_FIELD_CONTENT_RANGE, TS_MIME_LEN_CONTENT_RANGE, crstr, &crlen); + + // etag + char etagstr[1024]; + int etaglen = sizeof(etagstr); + header_resp.valueForKey(TS_MIME_FIELD_ETAG, TS_MIME_LEN_ETAG, etagstr, &etaglen); + + // last modified + char lmstr[1024]; + int lmlen = sizeof(lmstr); + header_resp.valueForKey(TS_MIME_FIELD_LAST_MODIFIED, TS_MIME_LEN_LAST_MODIFIED, lmstr, &lmlen); + + // cc + char ccstr[2048]; + int cclen = sizeof(ccstr); + header_resp.valueForKey(TS_MIME_FIELD_CACHE_CONTROL, TS_MIME_LEN_CACHE_CONTROL, ccstr, &cclen); + + // via tag + char viastr[8192]; + int vialen = sizeof(viastr); + header_resp.valueForKey(TS_MIME_FIELD_VIA, TS_MIME_LEN_VIA, viastr, &vialen); + + char etagexpstr[1024]; + size_t etagexplen = sizeof(etagexpstr); + TSStringPercentEncode(data->m_etag, data->m_etaglen, etagexpstr, etagexplen, &etagexplen, nullptr); + + char etaggotstr[1024]; + size_t etaggotlen = sizeof(etaggotstr); + TSStringPercentEncode(etagstr, etaglen, etaggotstr, etaggotlen, &etaggotlen, nullptr); + + DEBUG_LOG("Logging Block Stitch error"); + + ERROR_LOG("%" PRId64 ".%" PRId64 " reason=\"%s\"" + " uri=\"%.*s\"" + " uas=\"%.*s\"" + " req_range=\"%.*s\"" + " norm_range=\"%.*s\"" + + " etag_exp=\"%.*s\"" + " lm_exp=\"%.*s\"" + + " blk_range=\"%" PRId64 "-%" PRId64 "\"" + + " status_got=\"%d\"" + " cr_got=\"%.*s\"" + " etag_got=\"%.*s\"" + " lm_got=\"%.*s\"" + " cc=\"%.*s\"" + " via=\"%.*s\"", + secs, ms, message, (int)urlplen, urlpstr, uaslen, uasstr, rangelen, rangestr, normlen, normstr, (int)etagexplen, + etagexpstr, data->m_lastmodifiedlen, data->m_lastmodified, blockbeg, blockend - 1, statusgot, crlen, crstr, + (int)etaggotlen, etaggotstr, lmlen, lmstr, cclen, ccstr, vialen, viastr); +} + +bool +handleNextServerHeader(Data *const data, TSCont const contp) +{ + // block response header + HttpHeader header(data->m_resp_hdrmgr.m_buffer, data->m_resp_hdrmgr.m_lochdr); + // DEBUG_LOG("Next Header:\n%s", header.toString().c_str()); + + // only process a 206, everything else just aborts + if (TS_HTTP_STATUS_PARTIAL_CONTENT != header.status()) { + logSliceError("Non 206 internal block response", data, header); + return false; + } + + // can't parse the content range header, abort -- might be too strict + ContentRange const blockcr = contentRangeFrom(header); + if (!blockcr.isValid() || blockcr.m_length != data->m_contentlen) { + logSliceError("Mismatch/Bad block Content-Range", data, header); + return false; + } + + bool same = true; + + // prefer the etag but use Last-Modified if we must. + char etag[8192]; + int etaglen = sizeof(etag); + header.valueForKey(TS_MIME_FIELD_ETAG, TS_MIME_LEN_ETAG, etag, &etaglen); + + if (0 < data->m_etaglen || 0 < etaglen) { + same = data->m_etaglen == etaglen && 0 == strncmp(etag, data->m_etag, etaglen); + if (!same) { + logSliceError("Mismatch block Etag", data, header); + } + } else { + char lastmodified[8192]; + int lastmodifiedlen = sizeof(lastmodified); + header.valueForKey(TS_MIME_FIELD_LAST_MODIFIED, TS_MIME_LEN_LAST_MODIFIED, lastmodified, &lastmodifiedlen); + if (0 < data->m_lastmodifiedlen || 0 != lastmodifiedlen) { + same = data->m_lastmodifiedlen == lastmodifiedlen && 0 == strncmp(lastmodified, data->m_lastmodified, lastmodifiedlen); + if (!same) { + logSliceError("Mismatch block Last-Modified", data, header); + } + } + } + + if (!same) { + data->m_upstream.close(); + return false; + } + + data->m_blockexpected = blockcr.rangeSize(); + + return true; +} + +} // namespace + +// this is called every time the server has data for us +void +handle_server_resp(TSCont contp, TSEvent event, Data *const data) +{ + DEBUG_LOG("%p handle_server_resp: %s", data, TSHttpEventNameLookup(event)); + +#if defined(COLLECT_STATS) + TSStatIntIncrement(stats::Server, 1); +#endif + + switch (event) { + case TS_EVENT_VCONN_READ_READY: { + // has block response header been parsed?? + if (!data->m_server_block_header_parsed) { + int64_t consumed = 0; + TSIOBufferReader const reader = data->m_upstream.m_read.m_reader; + TSVIO const input_vio = data->m_upstream.m_read.m_vio; + TSParseResult const res = data->m_resp_hdrmgr.populateFrom(data->m_http_parser, reader, TSHttpHdrParseResp, &consumed); + + TSVIONDoneSet(input_vio, TSVIONDoneGet(input_vio) + consumed); + + // the server response header didn't fit into the input buffer. + if (TS_PARSE_CONT == res) { + return; + } + + // very first server response header + bool headerStat = false; + + if (TS_PARSE_DONE == res) { + if (!data->m_server_first_header_parsed) { + headerStat = handleFirstServerHeader(data, contp); + data->m_server_first_header_parsed = true; + } else { + headerStat = handleNextServerHeader(data, contp); + } + + data->m_server_block_header_parsed = true; + } + + // kill the upstream and allow dnstream to clean up + if (!headerStat) { + data->m_upstream.abort(); + data->m_blockstate = Data::BlockState::Fail; + if (data->m_dnstream.m_write.isOpen()) { + TSVIOReenable(data->m_dnstream.m_write.m_vio); + } else { + shutdown(contp, data); + } + return; + } + + // how much to fast forward into this data block + data->m_blockskip = data->m_req_range.skipBytesForBlock(data->m_config->m_blockbytes, data->m_blocknum); + } + + transfer_content_bytes(data); + } break; + case TS_EVENT_VCONN_READ_COMPLETE: { + // fprintf(stderr, "%p: TS_EVENT_VCONN_READ_COMPLETE\n", data); + } break; + case TS_EVENT_VCONN_EOS: { + data->m_blockstate = Data::BlockState::Pending; + data->m_upstream.close(); + + // check for block truncation + if (data->m_blockconsumed < data->m_blockexpected) { + DEBUG_LOG("%p handle_server_resp truncation: %" PRId64 "\n", data, data->m_blockexpected - data->m_blockconsumed); + data->m_blockstate = Data::BlockState::Fail; + // shutdown(contp, data); + return; + } + + // prepare for the next request block + ++data->m_blocknum; + + // when we get a "bytes=-" last N bytes request the plugin + // issues a speculative request for the first block + // in that case fast forward to the real first in range block + // Btw this isn't implemented yet, to be handled + int64_t const firstblock = data->m_req_range.firstBlockFor(data->m_config->m_blockbytes); + if (data->m_blocknum < firstblock) { + data->m_blocknum = firstblock; + } + + // continue processing blocks? + if (data->m_req_range.blockIsInside(data->m_config->m_blockbytes, data->m_blocknum)) { + // Don't immediately request the next slice if the client + // isn't keeping up + + bool start_next_block = true; + + // throttle condition + if (data->m_config->m_throttle && data->m_dnstream.m_read.isOpen()) { + TSVIO const output_vio = data->m_dnstream.m_write.m_vio; + int64_t const output_done = TSVIONDoneGet(output_vio); + int64_t const output_sent = data->m_bytessent; + int64_t const threshout = data->m_config->m_blockbytes; + + if (threshout < (output_done - output_sent)) { + start_next_block = false; + DEBUG_LOG("%p handle_server_resp: throttling %" PRId64, data, (output_done - output_sent)); + } + } + + if (start_next_block) { + request_block(contp, data); + } + + } else { + data->m_upstream.close(); + data->m_blockstate = Data::BlockState::Done; + if (!data->m_dnstream.m_read.isOpen()) { + shutdown(contp, data); + } + } + } break; + default: { + DEBUG_LOG("%p handle_server_resp uhandled event: %s", data, TSHttpEventNameLookup(event)); + } break; + } +} diff --git a/plugins/experimental/slice/server.h b/plugins/experimental/slice/server.h new file mode 100644 index 00000000000..c0d77e48032 --- /dev/null +++ b/plugins/experimental/slice/server.h @@ -0,0 +1,37 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "Data.h" + +#include "ts/ts.h" + +/** Functions to handle the connection to the server. + * In particular slice block header responses are handled here. + * Data transfers are handled by the client code which pulls + * the data from the server side. + * + * Special case is when the client connection has been closed + * because of client data request being fulfilled or + * when the client aborts. The current slice block will + * continue reading to ensure the whole block is transferred + * to cache. + */ + +void handle_server_resp(TSCont contp, TSEvent event, Data *const data); diff --git a/plugins/experimental/slice/slice.cc b/plugins/experimental/slice/slice.cc new file mode 100644 index 00000000000..0109f20b93b --- /dev/null +++ b/plugins/experimental/slice/slice.cc @@ -0,0 +1,308 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "slice.h" + +#include "Config.h" +#include "Data.h" +#include "HttpHeader.h" +#include "intercept.h" + +#include "ts/remap.h" +#include "ts/ts.h" + +#include +#include +#include + +#if defined(COLLECT_STATS) +namespace stats +{ +int DataCreate = -1; +int DataDestroy = -1; +int Reader = -1; +int Server = -1; +int Client = -1; +} // namespace stats +#endif // COLLECT_STATS + +namespace +{ +Config globalConfig; + +bool +read_request(TSHttpTxn txnp, Config *const config) +{ + DEBUG_LOG("slice read_request"); + TxnHdrMgr hdrmgr; + hdrmgr.populateFrom(txnp, TSHttpTxnClientReqGet); + HttpHeader const header(hdrmgr.m_buffer, hdrmgr.m_lochdr); + + if (TS_HTTP_METHOD_GET == header.method()) { + static int const SLICER_MIME_LEN_INFO = strlen(SLICER_MIME_FIELD_INFO); + if (!header.hasKey(SLICER_MIME_FIELD_INFO, SLICER_MIME_LEN_INFO)) { + // check if any previous plugin has monkeyed with the transaction status + TSHttpStatus const txnstat = TSHttpTxnStatusGet(txnp); + if (0 != (int)txnstat) { + DEBUG_LOG("slice: txn status change detected (%d), skipping plugin\n", (int)txnstat); + return false; + } + + // turn off any and all transaction caching (shouldn't matter) + TSHttpTxnServerRespNoStoreSet(txnp, 1); + TSHttpTxnRespCacheableSet(txnp, 0); + TSHttpTxnReqCacheableSet(txnp, 0); + + DEBUG_LOG("slice accepting and slicing"); + // connection back into ATS + sockaddr const *const ip = TSHttpTxnClientAddrGet(txnp); + if (nullptr == ip) { + return false; + } + + TSAssert(nullptr != config); + Data *const data = new Data(config); + + // set up feedback connect + if (AF_INET == ip->sa_family) { + memcpy(&data->m_client_ip, ip, sizeof(sockaddr_in)); + } else if (AF_INET6 == ip->sa_family) { + memcpy(&data->m_client_ip, ip, sizeof(sockaddr_in6)); + } else { + delete data; + return false; + } + + // need to reset the HOST field for global plugin + data->m_hostlen = sizeof(data->m_hostname) - 1; + if (!header.valueForKey(TS_MIME_FIELD_HOST, TS_MIME_LEN_HOST, data->m_hostname, &data->m_hostlen)) { + DEBUG_LOG("Unable to get hostname from header"); + delete data; + return false; + } + + // is the plugin configured to use a remap host? + std::string const &newhost = config->m_remaphost; + if (newhost.empty()) { + TSMBuffer urlbuf; + TSMLoc urlloc; + TSReturnCode rcode = TSHttpTxnPristineUrlGet(txnp, &urlbuf, &urlloc); + + if (TS_SUCCESS == rcode) { + TSMBuffer const newbuf = TSMBufferCreate(); + TSMLoc newloc = nullptr; + rcode = TSUrlClone(newbuf, urlbuf, urlloc, &newloc); + TSHandleMLocRelease(urlbuf, TS_NULL_MLOC, urlloc); + + if (TS_SUCCESS != rcode) { + ERROR_LOG("Error cloning pristine url"); + TSMBufferDestroy(newbuf); + delete data; + return false; + } + + data->m_urlbuf = newbuf; + data->m_urlloc = newloc; + } + } else { // grab the effective url, swap out the host and zero the port + int len = 0; + char *const effstr = TSHttpTxnEffectiveUrlStringGet(txnp, &len); + + if (nullptr != effstr) { + TSMBuffer const newbuf = TSMBufferCreate(); + TSMLoc newloc = nullptr; + bool okay = false; + + if (TS_SUCCESS == TSUrlCreate(newbuf, &newloc)) { + char const *start = effstr; + if (TS_PARSE_DONE == TSUrlParse(newbuf, newloc, &start, start + len)) { + if (TS_SUCCESS == TSUrlHostSet(newbuf, newloc, newhost.c_str(), newhost.size()) && + TS_SUCCESS == TSUrlPortSet(newbuf, newloc, 0)) { + okay = true; + } + } + } + + TSfree(effstr); + + if (!okay) { + ERROR_LOG("Error cloning effective url"); + if (nullptr != newloc) { + TSHandleMLocRelease(newbuf, nullptr, newloc); + } + TSMBufferDestroy(newbuf); + delete data; + return false; + } + + data->m_urlbuf = newbuf; + data->m_urlloc = newloc; + } + } + + if (TSIsDebugTagSet(PLUGIN_NAME)) { + int len = 0; + char *const urlstr = TSUrlStringGet(data->m_urlbuf, data->m_urlloc, &len); + DEBUG_LOG("slice url: %.*s", len, urlstr); + TSfree(urlstr); + } + + // we'll intercept this GET and do it ourselves + TSMutex const mutex = TSContMutexGet(reinterpret_cast(txnp)); + // TSMutex const mutex = TSMutexCreate(); + TSCont const icontp(TSContCreate(intercept_hook, mutex)); + TSContDataSet(icontp, (void *)data); + TSHttpTxnIntercept(icontp, txnp); + return true; + } else { + DEBUG_LOG("slice passing GET request through to next plugin"); + } + } + + return false; +} + +int +global_read_request_hook(TSCont // contp + , + TSEvent // event + , + void *edata) +{ + TSHttpTxn const txnp = static_cast(edata); + read_request(txnp, &globalConfig); + TSHttpTxnReenable(txnp, TS_EVENT_HTTP_CONTINUE); + return 0; +} + +} // namespace + +///// remap plugin engine + +SLICE_EXPORT +TSRemapStatus +TSRemapDoRemap(void *ih, TSHttpTxn txnp, TSRemapRequestInfo *rri) +{ + Config *const config = static_cast(ih); + + if (read_request(txnp, config)) { + return TSREMAP_DID_REMAP_STOP; + } else { + return TSREMAP_NO_REMAP; + } +} + +///// remap plugin setup and teardown +SLICE_EXPORT +void +TSRemapOSResponse(void *ih, TSHttpTxn rh, int os_response_type) +{ +} + +SLICE_EXPORT +TSReturnCode +TSRemapNewInstance(int argc, char *argv[], void **ih, char * /* errbuf */, int /* errbuf_size */) +{ + Config *const config = new Config; + if (2 < argc) { + config->fromArgs(argc - 2, argv + 2); + } + *ih = static_cast(config); + return TS_SUCCESS; +} + +SLICE_EXPORT +void +TSRemapDeleteInstance(void *ih) +{ + if (nullptr != ih) { + Config *const config = static_cast(ih); + delete config; + } +} + +SLICE_EXPORT +TSReturnCode +TSRemapInit(TSRemapInterface *api_info, char *errbug, int errbuf_size) +{ + DEBUG_LOG("slice remap initializing."); + +#if defined(COLLECT_STATS) + static bool init = false; + static std::mutex mutex; + + std::lock_guard lock(mutex); + + if (!init) { + init = true; + + std::string const namedatacreate = std::string(PLUGIN_NAME) + ".DataCreate"; + stats::DataCreate = TSStatCreate(namedatacreate.c_str(), TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); + + assert(0 <= stats::DataCreate); + + std::string const namedatadestroy = std::string(PLUGIN_NAME) + ".DataDestroy"; + stats::DataDestroy = TSStatCreate(namedatadestroy.c_str(), TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); + + assert(0 <= stats::DataDestroy); + + std::string const namereader = std::string(PLUGIN_NAME) + ".Reader"; + stats::Reader = TSStatCreate(namereader.c_str(), TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); + + assert(0 <= stats::Reader); + + std::string const nameserver = std::string(PLUGIN_NAME) + ".Server"; + stats::Server = TSStatCreate(nameserver.c_str(), TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); + + assert(0 <= stats::Server); + + std::string const nameclient = std::string(PLUGIN_NAME) + ".Client"; + stats::Client = TSStatCreate(nameclient.c_str(), TS_RECORDDATATYPE_INT, TS_STAT_NON_PERSISTENT, TS_STAT_SYNC_SUM); + + assert(0 <= stats::Client); + } +#endif // COLLECT_STATS + + return TS_SUCCESS; +} + +///// global plugin +SLICE_EXPORT +void +TSPluginInit(int argc, char const *argv[]) +{ + TSPluginRegistrationInfo info; + info.plugin_name = (char *)PLUGIN_NAME; + info.vendor_name = (char *)"Apache Software Foundation"; + info.support_email = (char *)"dev@trafficserver.apache.org"; + + if (TS_SUCCESS != TSPluginRegister(&info)) { + ERROR_LOG("Plugin registration failed.\n"); + ERROR_LOG("Unable to initialize plugin (disabled)."); + return; + } + + if (1 < argc) { + globalConfig.fromArgs(argc - 1, argv + 1); + } + + TSCont const contp(TSContCreate(global_read_request_hook, nullptr)); + + // Called immediately after the request header is read from the client + TSHttpHookAdd(TS_HTTP_READ_REQUEST_HDR_HOOK, contp); +} diff --git a/plugins/experimental/slice/slice.h b/plugins/experimental/slice/slice.h new file mode 100644 index 00000000000..d4e040b32f2 --- /dev/null +++ b/plugins/experimental/slice/slice.h @@ -0,0 +1,65 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +#include + +#ifndef SLICE_EXPORT +#define SLICE_EXPORT extern "C" tsapi +#endif + +#ifndef PLUGIN_NAME +#define PLUGIN_NAME "slice" +#endif + +#if !defined(UNITTEST) + +#define __FILENAME__ (strrchr(__FILE__, '/') ? strrchr(__FILE__, '/') + 1 : __FILE__) +#define DEBUG_LOG(fmt, ...) \ + TSDebug(PLUGIN_NAME, "[%s:%04d] %s(): " fmt, __FILENAME__, __LINE__, __func__, \ + ##__VA_ARGS__) /* \ + ; fprintf(stderr, "[%s:%04d]: " fmt "\n" \ + , __FILENAME__ \ + , __LINE__ \ + , ##__VA_ARGS__) \ + */ + +#define ERROR_LOG(fmt, ...) \ + TSError("[%s:%04d] %s(): " fmt, __FILENAME__, __LINE__, __func__, ##__VA_ARGS__); \ + TSDebug(PLUGIN_NAME, "[%s:%04d] %s(): " fmt, __FILENAME__, __LINE__, __func__, ##__VA_ARGS__) + +#else + +#define DEBUG_LOG(fmt, ...) +#define ERROR_LOG(fmt, ...) + +#endif + +#if defined(COLLECT_STATS) +namespace stats +{ +extern int DataCreate; +extern int DataDestroy; +extern int Reader; +extern int Server; +extern int Client; +} // namespace stats +#endif // COLLECT_STATS diff --git a/plugins/experimental/slice/slice_test.cc b/plugins/experimental/slice/slice_test.cc new file mode 100644 index 00000000000..4fcbc5be7bd --- /dev/null +++ b/plugins/experimental/slice/slice_test.cc @@ -0,0 +1,195 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +/* + * These are misc unit tests for slicer + */ + +#include "ContentRange.h" +#include "Range.h" + +#include +#include +#include +#include +#include + +std::string +testContentRange() +{ + std::ostringstream oss; + + ContentRange null; + if (null.isValid()) { + oss << "fail: null isValid test" << std::endl; + } + + ContentRange const exprange(1023, 1048576, 307232768); + + if (!exprange.isValid()) { + oss << "Fail: exprange valid" << std::endl; + oss << exprange.m_beg << ' ' << exprange.m_end << ' ' << exprange.m_length << std::endl; + } + + std::string const expstr("bytes 1023-1048575/307232768"); + + char gotbuf[1024]; + int gotlen = sizeof(gotbuf); + + bool const strstat(exprange.toStringClosed(gotbuf, &gotlen)); + + if (!strstat) { + oss << "failure status toStringClosed" << std::endl; + } else if ((int)expstr.size() != gotlen) { + oss << "Fail: expected toStringClosed length" << std::endl; + oss << "got: " << gotlen << " exp: " << expstr.size() << std::endl; + oss << "Got: " << gotbuf << std::endl; + oss << "Exp: " << expstr << std::endl; + } else if (expstr != gotbuf) { + oss << "Fail: expected toStringClosed value" << std::endl; + oss << "Got: " << gotbuf << std::endl; + oss << "Exp: " << expstr << std::endl; + } + + ContentRange gotrange; + bool const gotstat(gotrange.fromStringClosed(expstr.c_str())); + if (!gotstat) { + oss << "fail: gotstat from string" << std::endl; + } else if (gotrange.m_beg != exprange.m_beg || gotrange.m_end != exprange.m_end || gotrange.m_length != exprange.m_length) { + oss << "fail: value compare gotrange and exprange" << std::endl; + } + + std::string const teststr("bytes 0-1048575/30723276"); + if (!gotrange.fromStringClosed(teststr.c_str())) { + oss << "fail: parse teststr" << std::endl; + } + + return oss.str(); +} + +std::string +testParseRange() +{ + std::ostringstream oss; + + std::vector const teststrings = { + "bytes=0-1023", + "bytes=1-1024", + "bytes=11-11", + "bytes=1-" // 2nd byte to end + , + "Range: bytes=-13" // final 13 bytes + , + "bytes=3-17" // ,23-29" // open + , + "bytes=3 -17 " //,18-29" // adjacent + , + "bytes=3- 17" //, 11-29" // overlapping + , + "bytes=3 - 11" //,13-17 , 23-29" // unsorted triplet + , + "bytes=3-11 " //,13-17, 23-29" // unsorted triplet + , + "bytes=0-0" //,-1" // first and last bytes + , + "bytes=-20" // last 20 bytes of file + + , + "bytes=-60-50" // invalid fully negative + , + "bytes=17-13" // degenerate + , + "bytes 0-1023/146515" // this should be rejected (Content-range) + }; // invalid + + std::vector const exps = {Range{0, 1023 + 1}, Range{1, 1024 + 1}, Range{11, 11 + 1}, Range{1, Range::maxval}, + Range{-1, -1}, Range{3, 17 + 1}, Range{3, 17 + 1}, Range{3, 17 + 1}, + Range{3, 11 + 1}, Range{3, 11 + 1}, Range{0, 1}, Range{-20, 0}, + Range{-1, -1}, Range{-1, -1}, Range{-1, -1}}; + + std::vector const expsres = {true, true, true, true, false, true, true, true, true, true, true, true, false, false, false}; + + assert(exps.size() == teststrings.size()); + + std::vector gots; + gots.reserve(exps.size()); + std::vector gotsres; + + for (std::string const &str : teststrings) { + Range rng; + gotsres.push_back(rng.fromStringClosed(str.c_str())); + gots.push_back(rng); + } + + assert(gots.size() == exps.size()); + + for (size_t index(0); index < gots.size(); ++index) { + if (exps[index] != gots[index] || expsres[index] != gotsres[index]) { + oss << "Error parsing index: " << index << std::endl; + oss << "test: '" << teststrings[index] << "'" << std::endl; + oss << "exp: " << exps[index].m_beg << ' ' << exps[index].m_end << std::endl; + oss << "expsres: " << (int)expsres[index] << std::endl; + oss << "got: " << gots[index].m_beg << ' ' << gots[index].m_end << std::endl; + oss << "gotsres: " << (int)gotsres[index] << std::endl; + } + } + + return oss.str(); +} + +struct Tests { + typedef std::string (*TestFunc)(); + std::vector> funcs; + + void + add(TestFunc const &func, char const *const fname) + { + funcs.push_back(std::make_pair(func, fname)); + } + + int + run() const + { + int numfailed(0); + for (std::pair const &namefunc : funcs) { + TestFunc const &func = namefunc.first; + char const *const name = namefunc.second; + + std::cerr << name << " : "; + + std::string const fres(func()); + if (fres.empty()) { + std::cerr << "pass" << std::endl; + } else { + std::cerr << "FAIL" << std::endl; + std::cerr << fres << std::endl; + ++numfailed; + } + } + return numfailed; + } +}; + +int +main() +{ + Tests tests; + tests.add(testContentRange, "testContentRange"); + tests.add(testParseRange, "testParseRange"); + return tests.run(); +} diff --git a/plugins/experimental/slice/transfer.cc b/plugins/experimental/slice/transfer.cc new file mode 100644 index 00000000000..f83b0882c0b --- /dev/null +++ b/plugins/experimental/slice/transfer.cc @@ -0,0 +1,151 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "transfer.h" + +int64_t +transfer_content_bytes(Data *const data) +{ + // nothing to transfer if there's no source. + if (nullptr == data->m_upstream.m_read.m_reader) { + return 0; + } + + TSIOBufferReader const reader = data->m_upstream.m_read.m_reader; + TSIOBuffer const output_buf = data->m_dnstream.m_write.m_iobuf; + TSVIO const output_vio = data->m_dnstream.m_write.m_vio; + + int64_t consumed = 0; // input vio bytes visited + int64_t copied = 0; // output bytes transferred + + bool const canWrite = data->m_dnstream.m_write.isOpen(); + bool done = false; + + TSIOBufferBlock block = TSIOBufferReaderStart(reader); + + while (!done && nullptr != block) { + int64_t bavail = TSIOBufferBlockReadAvail(block, reader); + + if (0 == bavail) { + block = TSIOBufferBlockNext(block); + } else { + int64_t toconsume = 0; + + if (canWrite) { + int64_t const toskip = std::min(data->m_blockskip, bavail); + if (0 < toskip) { // before bytes + toconsume = toskip; + data->m_blockskip -= toskip; + } else { + int64_t const bytesleft = data->m_bytestosend - data->m_bytessent; + if (0 < bytesleft) { // transfer bytes + int64_t const tocopy = std::min(bavail, bytesleft); + int64_t const nbytes = TSIOBufferCopy(output_buf, reader, tocopy, 0); + + done = (nbytes < tocopy); // output buffer stuffed + + copied += nbytes; + data->m_bytessent += nbytes; + + toconsume = nbytes; + } else { // after bytes + toconsume = bavail; + } + } + } else { // drain + toconsume = bavail; + } + + if (0 < toconsume) { + if (bavail == toconsume) { + block = TSIOBufferBlockNext(block); + } + TSIOBufferReaderConsume(reader, toconsume); + consumed += toconsume; + } + } + } + + // tell output more data is available + if (0 < copied) { + TSVIOReenable(output_vio); + } + + if (0 < consumed) { + data->m_blockconsumed += consumed; + + TSVIO const input_vio = data->m_upstream.m_read.m_vio; + if (nullptr != input_vio) { + TSVIONDoneSet(input_vio, TSVIONDoneGet(input_vio) + consumed); + } + } + + return consumed; +} + +// transfer all bytes from the server (error condition) +int64_t +transfer_all_bytes(Data *const data) +{ + // nothing to transfer if there's no source. + if (nullptr == data->m_upstream.m_read.m_reader || !data->m_dnstream.m_write.isOpen()) { + return 0; + } + + int64_t consumed = 0; // input vio bytes visited + + TSIOBufferReader const reader = data->m_upstream.m_read.m_reader; + TSIOBuffer const output_buf = data->m_dnstream.m_write.m_iobuf; + + bool done = false; + + TSIOBufferBlock block = TSIOBufferReaderStart(reader); + + while (!done && nullptr != block) { + int64_t bavail = TSIOBufferBlockReadAvail(block, reader); + + if (0 == bavail) { + block = TSIOBufferBlockNext(block); + } else { + int64_t const nbytes = TSIOBufferCopy(output_buf, reader, bavail, 0); + done = nbytes < bavail; // output buffer is full + + if (0 < nbytes) { + if (bavail == nbytes) { + block = TSIOBufferBlockNext(block); + } + TSIOBufferReaderConsume(reader, nbytes); + consumed += nbytes; + } + } + } + + if (0 < consumed) { + TSVIO const output_vio = data->m_dnstream.m_write.m_vio; + if (nullptr != output_vio) { + TSVIOReenable(output_vio); + } + + TSVIO const input_vio = data->m_upstream.m_read.m_vio; + if (nullptr != input_vio) { + TSVIONDoneSet(input_vio, TSVIONDoneGet(input_vio) + consumed); + } + } + + return consumed; +} diff --git a/plugins/experimental/slice/transfer.h b/plugins/experimental/slice/transfer.h new file mode 100644 index 00000000000..a63bb243a2d --- /dev/null +++ b/plugins/experimental/slice/transfer.h @@ -0,0 +1,37 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "Data.h" + +/** Functions to deal with the connection to the client. + * Body content transfers are handled by the client. + * New block requests are also initiated by the client. + */ + +/* transfer bytes from the server to the client + * Returns amount of bytes consumed from the reader (<= bytes written to client) + */ +int64_t transfer_content_bytes(Data *const data); // , char const * const fstr); + +// transfer all bytes from the server (error condition) +int64_t transfer_all_bytes(Data *const data); + +// Signal the input about write state +void signal_input(TSVIO const input_vio, int64_t const consumed); diff --git a/plugins/experimental/slice/unit-tests/slice_test.cc b/plugins/experimental/slice/unit-tests/slice_test.cc new file mode 100644 index 00000000000..96ae726a53c --- /dev/null +++ b/plugins/experimental/slice/unit-tests/slice_test.cc @@ -0,0 +1,181 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +/* + * These are misc unit tests for slicer + */ + +#include "ContentRange.h" +#include "Range.h" + +#include +#include +#include +#include +#include + +std::string +testContentRange() +{ + std::ostringstream oss; + + ContentRange null; + if (null.isValid()) { + oss << "fail: null isValid test" << std::endl; + } + + ContentRange const exprange(1023, 1048576, 307232768); + + if (!exprange.isValid()) { + oss << "Fail: exprange valid" << std::endl; + oss << exprange.m_beg << ' ' << exprange.m_end << ' ' << exprange.m_length << std::endl; + } + + std::string const expstr("bytes 1023-1048575/307232768"); + + char gotbuf[1024]; + int gotlen = sizeof(gotbuf); + + bool const strstat(exprange.toStringClosed(gotbuf, &gotlen)); + + if (!strstat) { + oss << "failure status toStringClosed" << std::endl; + } else if ((int)expstr.size() != gotlen) { + oss << "Fail: expected toStringClosed length" << std::endl; + oss << "got: " << gotlen << " exp: " << expstr.size() << std::endl; + oss << "Got: " << gotbuf << std::endl; + oss << "Exp: " << expstr << std::endl; + } else if (expstr != gotbuf) { + oss << "Fail: expected toStringClosed value" << std::endl; + oss << "Got: " << gotbuf << std::endl; + oss << "Exp: " << expstr << std::endl; + } + + ContentRange gotrange; + bool const gotstat(gotrange.fromStringClosed(expstr.c_str())); + if (!gotstat) { + oss << "fail: gotstat from string" << std::endl; + } else if (gotrange.m_beg != exprange.m_beg || gotrange.m_end != exprange.m_end || gotrange.m_length != exprange.m_length) { + oss << "fail: value compare gotrange and exprange" << std::endl; + } + + std::string const teststr("bytes 0-1048575/30723276"); + if (!gotrange.fromStringClosed(teststr.c_str())) { + oss << "fail: parse teststr" << std::endl; + } + + return oss.str(); +} + +std::string +testParseRange() +{ + std::ostringstream oss; + + std::vector const teststrings = { + "bytes=0-1023", "bytes=1-1024", "bytes=11-11", + "bytes=1-", // 2nd byte to end + "Range: bytes=-13", // final 13 bytes + "bytes=3-17", // ,23-29" // open + "bytes=3 -17 ", //,18-29" // adjacent + "bytes=3- 17", //, 11-29" // overlapping + "bytes=3 - 11", //,13-17 , 23-29" // unsorted triplet + "bytes=3-11 ", //,13-17, 23-29" // unsorted triplet + "bytes=0-0", //,-1" // first and last bytes + "bytes=-20", // last 20 bytes of file + "bytes=-60-50", // invalid fully negative + "bytes=17-13", // degenerate + "bytes 0-1023/146515" // this should be rejected (Content-range) + }; // invalid + + std::vector const exps = {Range{0, 1023 + 1}, Range{1, 1024 + 1}, Range{11, 11 + 1}, Range{1, Range::maxval}, + Range{-1, -1}, Range{3, 17 + 1}, Range{3, 17 + 1}, Range{3, 17 + 1}, + Range{3, 11 + 1}, Range{3, 11 + 1}, Range{0, 1}, Range{-20, 0}, + Range{-1, -1}, Range{-1, -1}, Range{-1, -1}}; + + std::vector const expsres = {true, true, true, true, false, true, true, true, true, true, true, true, false, false, false}; + + assert(exps.size() == teststrings.size()); + + std::vector gots; + gots.reserve(exps.size()); + std::vector gotsres; + + for (std::string const &str : teststrings) { + Range rng; + gotsres.push_back(rng.fromStringClosed(str.c_str())); + gots.push_back(rng); + } + + assert(gots.size() == exps.size()); + + for (size_t index(0); index < gots.size(); ++index) { + if (exps[index] != gots[index] || expsres[index] != gotsres[index]) { + oss << "Eror parsing index: " << index << std::endl; + oss << "test: '" << teststrings[index] << "'" << std::endl; + oss << "exp: " << exps[index].m_beg << ' ' << exps[index].m_end << std::endl; + oss << "expsres: " << (int)expsres[index] << std::endl; + oss << "got: " << gots[index].m_beg << ' ' << gots[index].m_end << std::endl; + oss << "gotsres: " << (int)gotsres[index] << std::endl; + } + } + + return oss.str(); +} + +struct Tests { + typedef std::string (*TestFunc)(); + std::vector> funcs; + + void + add(TestFunc const &func, char const *const fname) + { + funcs.push_back(std::make_pair(func, fname)); + } + + int + run() const + { + int numfailed(0); + for (std::pair const &namefunc : funcs) { + TestFunc const &func = namefunc.first; + char const *const name = namefunc.second; + + std::cerr << name << " : "; + + std::string const fres(func()); + if (fres.empty()) { + std::cerr << "pass" << std::endl; + } else { + std::cerr << "FAIL" << std::endl; + std::cerr << fres << std::endl; + ++numfailed; + } + } + return numfailed; + } +}; + +int +main() +{ + Tests tests; + tests.add(testContentRange, "testContentRange"); + tests.add(testParseRange, "testParseRange"); + return tests.run(); +} diff --git a/plugins/experimental/slice/unit-tests/test_config.cc b/plugins/experimental/slice/unit-tests/test_config.cc new file mode 100644 index 00000000000..b85f381e959 --- /dev/null +++ b/plugins/experimental/slice/unit-tests/test_config.cc @@ -0,0 +1,86 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +/** + * @file test_content_range.cc + * @brief Unit test for slice ContentRange + */ + +#define CATCH_CONFIG_MAIN /* include main function */ +#include "../Config.h" +#include "catch.hpp" /* catch unit-test framework */ + +#include + +TEST_CASE("config default", "[AWS][slice][utility]") +{ + Config const config; + int64_t const defval = Config::blockbytesdefault; + CHECK(defval == config.m_blockbytes); +} + +TEST_CASE("config bytesfrom valid parsing", "[AWS][slice][utility]") +{ + static std::array const teststrings = {{ + "1000", + "1m", + "5g", + "2k", + "3kb", + "1z", + }}; + + constexpr std::array const expvals = {{ + 1000, + 1024 * 1024, + int64_t(1024) * 1024 * 1024 * 5, + 1024 * 2, + 1024 * 3, + 1, + }}; + + for (size_t index = 0; index < teststrings.size(); ++index) { + std::string const &teststr = teststrings[index]; + int64_t const &exp = expvals[index]; + int64_t const got = Config::bytesFrom(teststr.c_str()); + + CHECK(got == exp); + if (got != exp) { + INFO(teststr.c_str()); + } + } +} + +TEST_CASE("config bytesfrom invalid parsing", "[AWS][slice][utility]") +{ + static std::array const badstrings = {{ + "abc", // alpha + "g00", // giga + "M00", // mega + "k00", // kilo + "-500", // negative + }}; + + for (std::string const &badstr : badstrings) { + int64_t const val = Config::bytesFrom(badstr.c_str()); + CHECK(0 == val); + if (0 != val) { + INFO(badstr.c_str()); + } + } +} diff --git a/plugins/experimental/slice/unit-tests/test_content_range.cc b/plugins/experimental/slice/unit-tests/test_content_range.cc new file mode 100644 index 00000000000..a9876297d62 --- /dev/null +++ b/plugins/experimental/slice/unit-tests/test_content_range.cc @@ -0,0 +1,80 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +/** + * @file test_content_range.cc + * @brief Unit test for slice ContentRange + */ + +#define CATCH_CONFIG_MAIN /* include main function */ +#include "catch.hpp" /* catch unit-test framework */ +#include "../ContentRange.h" + +TEST_CASE("content_range invalid state", "[AWS][slice][utility]") +{ + CHECK_FALSE(ContentRange().isValid()); // null range + CHECK_FALSE(ContentRange(1024, 1024, 4000).isValid()); // zero range + CHECK_FALSE(ContentRange(0, 1024, 1023).isValid()); // past end + CHECK_FALSE(ContentRange(-5, 13, 40).isValid()); // negative start +} + +TEST_CASE("content_range to/from string - valid", "[AWS][slice][utility]") +{ + ContentRange const exprange(1023, 1048576, 307232768); + + CHECK(exprange.isValid()); + + std::string const expstr("bytes 1023-1048575/307232768"); + + char gotbuf[1024]; + int gotlen = sizeof(gotbuf); + + bool const strstat(exprange.toStringClosed(gotbuf, &gotlen)); + + CHECK(strstat); + CHECK(gotlen == expstr.size()); + CHECK(expstr == std::string(gotbuf)); + + ContentRange gotrange; + bool const gotstat(gotrange.fromStringClosed(expstr.c_str())); + + CHECK(gotstat); + CHECK(gotrange.m_beg == exprange.m_beg); + CHECK(gotrange.m_end == exprange.m_end); + CHECK(gotrange.m_length == exprange.m_length); +} + +TEST_CASE("content_range from string - invalid", "[AWS][slice][utility]") +{ + std::vector const badstrings = { + "bytes=1024-1692", // malformed + "bytes=1023-1048575/307232768", // malformed + "bytes 1023-1022/5000", // zero size + "bytes -40-12/50", // negative start + "bytes 5-13/11" // past end + }; + + ContentRange cr; + + for (std::string const &badstr : badstrings) { + if (!cr.fromStringClosed(badstr.c_str())) { + CHECK_FALSE(cr.isValid()); + INFO(badstr.c_str()); + } + } +} diff --git a/plugins/experimental/slice/unit-tests/test_range.cc b/plugins/experimental/slice/unit-tests/test_range.cc new file mode 100644 index 00000000000..d2b1813fe3f --- /dev/null +++ b/plugins/experimental/slice/unit-tests/test_range.cc @@ -0,0 +1,99 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +/** + * @file test_content_range.cc + * @brief Unit test for slice ContentRange + */ + +#define CATCH_CONFIG_MAIN /* include main function */ +#include "catch.hpp" /* catch unit-test framework */ +#include "../Range.h" + +TEST_CASE("range invalid state", "[AWS][slice][utility]") +{ + CHECK_FALSE(Range().isValid()); // null range + CHECK_FALSE(Range(1024, 1024).isValid()); // zero range + CHECK_FALSE(Range(-5, 13).isValid()); // negative start +} + +TEST_CASE("range to/from string - valid", "[AWS][slice][utility]") +{ + std::vector const teststrings = { + "bytes=0-1023", // start at zero + "bytes=1-1024", // start from non zero + "bytes=11-11", // single byte + "bytes=1-", // 2nd byte to end + "bytes=3-17", // ,23-29" // open + "bytes=3 -17 ", //,18-29" // adjacent + "bytes=3- 17", //, 11-29" // overlapping + "bytes=3 - 11", //,13-17 , 23-29" // unsorted triplet + "bytes=3-11 ", //,13-17, 23-29" // unsorted triplet + "bytes=0-0", //,-1" // first and last bytes + "bytes=-20", // last 20 bytes of file + }; + + std::vector const exps = { + Range{0, 1023 + 1}, // + Range{1, 1024 + 1}, // + Range{11, 11 + 1}, // + Range{1, Range::maxval}, // + Range{3, 17 + 1}, // + Range{3, 17 + 1}, // + Range{3, 17 + 1}, // + Range{3, 11 + 1}, // + Range{3, 11 + 1}, // + Range{0, 1}, // + Range{-20, 0} // + }; + + for (size_t index = 0; index < teststrings.size(); ++index) { + std::string const &str = teststrings[index]; + + Range got; + CHECK(got.fromStringClosed(str.c_str())); + CHECK(got.isValid()); + + if (!got.isValid()) { + INFO(str.c_str()); + } + + Range const &exp = exps[index]; + CHECK(got.m_beg == exp.m_beg); + CHECK(got.m_end == exp.m_end); + } +} + +TEST_CASE("range from string - invalid") +{ + std::vector const badstrings = { + "Range: bytes=-13", // malformed + "bytes=-60-50", // first negative, second nonzero + "bytes=17-13", // degenerate + "bytes 0-1023/146515" // malformed + }; + + Range range; + for (std::string const &badstr : badstrings) { + CHECK_FALSE(range.fromStringClosed(badstr.c_str())); + CHECK_FALSE(range.isValid()); + if (range.isValid()) { + INFO(badstr.c_str()); + } + } +} diff --git a/plugins/experimental/slice/util.cc b/plugins/experimental/slice/util.cc new file mode 100644 index 00000000000..f3ae7d2bcf1 --- /dev/null +++ b/plugins/experimental/slice/util.cc @@ -0,0 +1,134 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "util.h" + +#include "Config.h" +#include "Data.h" + +void +shutdown(TSCont const contp, Data *const data) +{ + DEBUG_LOG("shutting down transaction"); + TSContDataSet(contp, nullptr); + delete data; + TSContDestroy(contp); +} + +void +abort(TSCont const contp, Data *const data) +{ + DEBUG_LOG("aborting transaction"); + TSContDataSet(contp, nullptr); + data->m_upstream.abort(); + data->m_dnstream.abort(); + delete data; + TSContDestroy(contp); +} + +// create and issue a block request +bool +request_block(TSCont contp, Data *const data) +{ + // ensure no upstream connection + if (data->m_upstream.m_read.isOpen()) { + ERROR_LOG("Block request already in flight!"); + return false; + } + + if (Data::BlockState::Pending != data->m_blockstate) { + ERROR_LOG("request_block called with non Pending state!"); + return false; + } + + int64_t const blockbeg = (data->m_config->m_blockbytes * data->m_blocknum); + Range blockbe(blockbeg, blockbeg + data->m_config->m_blockbytes); + + char rangestr[1024]; + int rangelen = sizeof(rangestr); + bool const rpstat = blockbe.toStringClosed(rangestr, &rangelen); + TSAssert(rpstat); + + DEBUG_LOG("requestBlock: %s", rangestr); + + // reuse the incoming client header, just change the range + HttpHeader header(data->m_req_hdrmgr.m_buffer, data->m_req_hdrmgr.m_lochdr); + + // add/set sub range key and add slicer tag + bool const rangestat = header.setKeyVal(TS_MIME_FIELD_RANGE, TS_MIME_LEN_RANGE, rangestr, rangelen); + + if (!rangestat) { + ERROR_LOG("Error trying to set range request header %s", rangestr); + return false; + } + + // create virtual connection back into ATS + TSVConn const upvc = TSHttpConnectWithPluginId((sockaddr *)&data->m_client_ip, PLUGIN_NAME, 0); + + int const hlen = TSHttpHdrLengthGet(header.m_buffer, header.m_lochdr); + + // set up connection with the HttpConnect server + data->m_upstream.setupConnection(upvc); + data->m_upstream.setupVioWrite(contp, hlen); + + // Send full request + TSHttpHdrPrint(header.m_buffer, header.m_lochdr, data->m_upstream.m_write.m_iobuf); + TSVIOReenable(data->m_upstream.m_write.m_vio); + + /* + std::string const headerstr(header.toString()); + DEBUG_LOG("Headers\n%s", headerstr.c_str()); + */ + + // get ready for data back from the server + data->m_upstream.setupVioRead(contp, INT64_MAX); + + // anticipate the next server response header + TSHttpParserClear(data->m_http_parser); + data->m_resp_hdrmgr.resetHeader(); + + data->m_blockexpected = 0; + data->m_blockconsumed = 0; + data->m_blockstate = Data::BlockState::Active; + data->m_server_block_header_parsed = false; + + return true; +} + +bool +reader_avail_more_than(TSIOBufferReader const reader, int64_t bytes) +{ + TSIOBufferBlock block = TSIOBufferReaderStart(reader); + + if (nullptr == block) { + return false; + } + + while (nullptr != block) { + int64_t const blockbytes = TSIOBufferBlockReadAvail(block, reader); + if (bytes < blockbytes) { + return true; + } else { + bytes -= blockbytes; + } + + block = TSIOBufferBlockNext(block); + } + + return false; +} diff --git a/plugins/experimental/slice/util.h b/plugins/experimental/slice/util.h new file mode 100644 index 00000000000..9da6f368a3b --- /dev/null +++ b/plugins/experimental/slice/util.h @@ -0,0 +1,36 @@ +/** @file + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include "ts/ts.h" + +struct Data; + +/** Functions to deal with the connection to the client. + * Body content transfers are handled by the client. + * New block requests are also initiated by the client. + */ + +void shutdown(TSCont const contp, Data *const data); + +void abort(TSCont const contp, Data *const data); + +bool request_block(TSCont contp, Data *const data); + +bool reader_avail_more_than(TSIOBufferReader const reader, int64_t bytes); diff --git a/tests/gold_tests/pluginTest/slice/curlsort.sh b/tests/gold_tests/pluginTest/slice/curlsort.sh new file mode 100755 index 00000000000..eabf5618ff2 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/curlsort.sh @@ -0,0 +1,33 @@ +#!/bin/sh + +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cmd='curl' +for arg in "$@"; do + case "$arg" in + *\'*) +# arg=`printf '%s' "$arg" | sed s/'/'\"'\"'/g"` + arg=`printf '%s' "$arg"` + ;; + *) : ;; + esac + cmd="$cmd '$arg'" +done + +cmd="$cmd -s -D /dev/stdout -o /dev/stderr" + +eval " $cmd" | sort diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_200.stderr.gold b/tests/gold_tests/pluginTest/slice/gold/slice_200.stderr.gold new file mode 100644 index 00000000000..24ad29c1a82 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_200.stderr.gold @@ -0,0 +1 @@ +lets go surfin now`` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_200.stdout.gold b/tests/gold_tests/pluginTest/slice/gold/slice_200.stdout.gold new file mode 100644 index 00000000000..50eeb38afb7 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_200.stdout.gold @@ -0,0 +1,8 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 18 +Date: `` +Etag: "path" +HTTP/1.1 200 OK +Server: `` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_206.stderr.gold b/tests/gold_tests/pluginTest/slice/gold/slice_206.stderr.gold new file mode 100644 index 00000000000..24ad29c1a82 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_206.stderr.gold @@ -0,0 +1 @@ +lets go surfin now`` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_206.stdout.gold b/tests/gold_tests/pluginTest/slice/gold/slice_206.stdout.gold new file mode 100644 index 00000000000..0ff73438579 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_206.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 18 +Content-Range: bytes 0-17/18 +Date: `` +Etag: "path" +HTTP/1.1 206 Partial Content +Server: `` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_first.stderr.gold b/tests/gold_tests/pluginTest/slice/gold/slice_first.stderr.gold new file mode 100644 index 00000000000..10893bb0688 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_first.stderr.gold @@ -0,0 +1 @@ +lets go`` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_first.stdout.gold b/tests/gold_tests/pluginTest/slice/gold/slice_first.stdout.gold new file mode 100644 index 00000000000..c12d4c194da --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_first.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: max-age=`` +Connection: `` +Content-Length: 7 +Content-Range: bytes 0-6/18 +Date: `` +Etag: `` +HTTP/1.1 206 Partial Content +Server: `` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_last.stderr.gold b/tests/gold_tests/pluginTest/slice/gold/slice_last.stderr.gold new file mode 100644 index 00000000000..30d9d270514 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_last.stderr.gold @@ -0,0 +1 @@ + now`` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_last.stdout.gold b/tests/gold_tests/pluginTest/slice/gold/slice_last.stdout.gold new file mode 100644 index 00000000000..8457c952f01 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_last.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: max-age=`` +Connection: `` +Content-Length: 4 +Content-Range: bytes 14-17/18 +Date: `` +Etag: `` +HTTP/1.1 206 Partial Content +Server: `` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_mid.stderr.gold b/tests/gold_tests/pluginTest/slice/gold/slice_mid.stderr.gold new file mode 100644 index 00000000000..ef40e3253f5 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_mid.stderr.gold @@ -0,0 +1 @@ +go surfin no`` diff --git a/tests/gold_tests/pluginTest/slice/gold/slice_mid.stdout.gold b/tests/gold_tests/pluginTest/slice/gold/slice_mid.stdout.gold new file mode 100644 index 00000000000..e078833d768 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold/slice_mid.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: max-age=`` +Connection: `` +Content-Length: 12 +Content-Range: bytes 5-16/18 +Date: `` +Etag: `` +HTTP/1.1 206 Partial Content +Server: `` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/crr.stderr.gold b/tests/gold_tests/pluginTest/slice/gold_error/crr.stderr.gold new file mode 100644 index 00000000000..fad05754105 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/crr.stderr.gold @@ -0,0 +1 @@ +the quick`` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/crr.stdout.gold b/tests/gold_tests/pluginTest/slice/gold_error/crr.stdout.gold new file mode 100644 index 00000000000..7109c04a13f --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/crr.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 19 +Date: `` +Etag: `` +HTTP/1.1 200 OK +Server: `` +Via: `` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/etag.stderr.gold b/tests/gold_tests/pluginTest/slice/gold_error/etag.stderr.gold new file mode 100644 index 00000000000..fad05754105 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/etag.stderr.gold @@ -0,0 +1 @@ +the quick`` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/etag.stdout.gold b/tests/gold_tests/pluginTest/slice/gold_error/etag.stdout.gold new file mode 100644 index 00000000000..7109c04a13f --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/etag.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 19 +Date: `` +Etag: `` +HTTP/1.1 200 OK +Server: `` +Via: `` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/lm.stderr.gold b/tests/gold_tests/pluginTest/slice/gold_error/lm.stderr.gold new file mode 100644 index 00000000000..fad05754105 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/lm.stderr.gold @@ -0,0 +1 @@ +the quick`` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/lm.stdout.gold b/tests/gold_tests/pluginTest/slice/gold_error/lm.stdout.gold new file mode 100644 index 00000000000..20bd1a54127 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/lm.stdout.gold @@ -0,0 +1,9 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 19 +Date: `` +HTTP/1.1 200 OK +Last-Modified: `` +Server: `` +Via: `` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/non206.stderr.gold b/tests/gold_tests/pluginTest/slice/gold_error/non206.stderr.gold new file mode 100644 index 00000000000..fad05754105 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/non206.stderr.gold @@ -0,0 +1 @@ +the quick`` diff --git a/tests/gold_tests/pluginTest/slice/gold_error/non206.stdout.gold b/tests/gold_tests/pluginTest/slice/gold_error/non206.stdout.gold new file mode 100644 index 00000000000..f381661d6a6 --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/gold_error/non206.stdout.gold @@ -0,0 +1,10 @@ +`` +Cache-Control: `` +Connection: `` +Content-Length: 19 +Date: `` +Etag: `` +HTTP/1.1 200 OK +Last-Modified: `` +Server: `` +Via: `` diff --git a/tests/gold_tests/pluginTest/slice/slice.test.py b/tests/gold_tests/pluginTest/slice/slice.test.py new file mode 100644 index 00000000000..81e1f563f5b --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/slice.test.py @@ -0,0 +1,183 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Basic slice plugin test +''' + +## Test description: +# Preload the cache with the entire asset to be range requested. +# Reload remap rule with slice plugin +# Request content through the slice plugin + +Test.SkipUnless( + Condition.PluginExists('slice.so'), +) +Test.ContinueOnFail = False + +# configure origin server +server = Test.MakeOriginServer("server") + +# Define ATS and configure +ts = Test.MakeATSProcess("ts", command="traffic_manager", select_ports=True) + +# default root +request_header_chk = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_chk = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +server.addResponse("sessionlog.json", request_header_chk, response_header_chk) + +#block_bytes = 7 +body = "lets go surfin now" + +request_header = {"headers": + "GET /path HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + 'Etag: "path"\r\n' + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body, +} + +server.addResponse("sessionlog.json", request_header, response_header) + +ts.Setup.CopyAs('curlsort.sh', Test.RunDirectory) +curl_and_args = 'sh curlsort.sh -H "Host: www.example.com"' + +# set up whole asset fetch into cache +ts.Disk.remap_config.AddLine( + 'map / http://127.0.0.1:{}'.format(server.Variables.Port) +) + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'slice', + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, + 'proxy.config.http.insert_age_in_response': 0, + 'proxy.config.http.response_via_str': 3, +}) + +# 0 Test - Prefetch entire asset into cache +tr = Test.AddTestRun("Fetch first slice range") +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_200.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_200.stderr.gold" +tr.StillRunningAfter = ts + +block_bytes = 7 + +# 1 - Reconfigure remap.config with slice plugin +tr = Test.AddTestRun("Load Slice plugin") +remap_config_path = ts.Disk.remap_config.Name +tr.Disk.File(remap_config_path, typename="ats:config").AddLines([ + 'map / http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=slice.so @pparam=--blockbytes-test={}'.format(block_bytes) +]) + +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +tr.Processes.Default.Command = 'traffic_ctl config reload' +# Need to copy over the environment so traffic_ctl knows where to find the unix domain socket +tr.Processes.Default.Env = ts.Env +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.TimeOut = 5 +tr.TimeOut = 5 + +# 2 Test - First complete slice +tr = Test.AddTestRun("Fetch first slice range") +tr.DelayStart = 5 +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 0-6' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_first.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_first.stderr.gold" +tr.StillRunningAfter = ts + +# 3 Test - Last slice auto +tr = Test.AddTestRun("Last slice -- 14-") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 14-' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_last.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_last.stderr.gold" +tr.StillRunningAfter = ts + +# 4 Test - Last slice exact +tr = Test.AddTestRun("Last slice 14-17") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 14-17' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_last.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_last.stderr.gold" +tr.StillRunningAfter = ts + +# 5 Test - Last slice truncated +tr = Test.AddTestRun("Last truncated slice 14-20") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 14-20' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_last.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_last.stderr.gold" +tr.StillRunningAfter = ts + +# 6 Test - Whole asset via slices +tr = Test.AddTestRun("Whole asset via slices") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_200.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_200.stderr.gold" +tr.StillRunningAfter = ts + +# 7 Test - Whole asset via range +tr = Test.AddTestRun("Whole asset via range") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 0-' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_206.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_206.stderr.gold" +tr.StillRunningAfter = ts + +# 8 Test - Non aligned slice request +tr = Test.AddTestRun("Non aligned slice request") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path'.format(ts.Variables.port) + ' -r 5-16' +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold/slice_mid.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold/slice_mid.stderr.gold" +tr.StillRunningAfter = ts diff --git a/tests/gold_tests/pluginTest/slice/slice_error.test.py b/tests/gold_tests/pluginTest/slice/slice_error.test.py new file mode 100644 index 00000000000..0ae06f1f23c --- /dev/null +++ b/tests/gold_tests/pluginTest/slice/slice_error.test.py @@ -0,0 +1,322 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Slice plugin error.log test +''' + +## Test description: +# Preload the cache with the entire asset to be range requested. +# Reload remap rule with slice plugin +# Request content through the slice plugin + +Test.SkipUnless( + Condition.PluginExists('slice.so'), +) +Test.ContinueOnFail = False + +# configure origin server +server = Test.MakeOriginServer("server", lookup_key="{%Range}{PATH}") + +# Define ATS and configure +ts = Test.MakeATSProcess("ts", command="traffic_manager", select_ports=True) + +body = "the quick brown fox" # len 19 + +# default root +request_header_chk = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes=0-\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_chk = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body, +} + +server.addResponse("sessionlog.json", request_header_chk, response_header_chk) + +blockbytes = 9 + +range0 = "{}-{}".format(0, blockbytes - 1) +range1 = "{}-{}".format(blockbytes, (2 * blockbytes) - 1) + +body0 = body[0:blockbytes] +body1 = body[blockbytes:2 * blockbytes] + +# Mismatch etag + +request_header_etag0 = {"headers": + "GET /etag HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range0) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_etag0 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + 'Etag: "etag0"\r\n' + + "Content-Range: bytes {}/{}\r\n".format(range0, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body0, +} + +server.addResponse("sessionlog.json", request_header_etag0, response_header_etag0) + +request_header_etag1 = {"headers": + "GET /etag HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range1) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_etag1 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + 'Etag: "etag1"\r\n' + + "Content-Range: bytes {}/{}\r\n".format(range1, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body1, +} + +server.addResponse("sessionlog.json", request_header_etag1, response_header_etag1) + +# mismatch Last-Modified + +request_header_lm0 = {"headers": + "GET /lastmodified HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range0) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_lm0 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + "Last-Modified: Tue, 08 May 2018 15:49:41 GMT\r\n" + + "Content-Range: bytes {}/{}\r\n".format(range0, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body0, +} + +server.addResponse("sessionlog.json", request_header_lm0, response_header_lm0) + +request_header_lm1 = {"headers": + "GET /lastmodified HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range1) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_lm1 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + "Last-Modified: Tue, 08 Apr 2019 18:00:00 GMT\r\n" + + "Content-Range: bytes {}/{}\r\n".format(range1, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body1, +} + +server.addResponse("sessionlog.json", request_header_lm1, response_header_lm1) + +# non 206 slice block + +request_header_n206_0 = {"headers": + "GET /non206 HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range0) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_n206_0 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + 'Etag: "etag"\r\n' + + "Last-Modified: Tue, 08 May 2018 15:49:41 GMT\r\n" + + "Content-Range: bytes {}/{}\r\n".format(range0, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body0, +} + +server.addResponse("sessionlog.json", request_header_n206_0, response_header_n206_0) + +# mismatch content-range + +request_header_crr0 = {"headers": + "GET /crr HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range0) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_crr0 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + "Etag: crr\r\n" + + "Content-Range: bytes {}/{}\r\n".format(range0, len(body)) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body0, +} + +server.addResponse("sessionlog.json", request_header_crr0, response_header_crr0) + +request_header_crr1 = {"headers": + "GET /crr HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "Range: bytes={}\r\n".format(range1) + + "X-Slicer-Info: full content request\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", +} + +response_header_crr1 = {"headers": + "HTTP/1.1 206 Partial Content\r\n" + + "Connection: close\r\n" + + "Etag: crr\r\n" + + "Content-Range: bytes {}/{}\r\n".format(range1, len(body) - 1) + + "Cache-Control: max-age=500\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": body1, +} + +server.addResponse("sessionlog.json", request_header_crr1, response_header_crr1) + +ts.Setup.CopyAs('curlsort.sh', Test.RunDirectory) +curl_and_args = 'sh curlsort.sh -H "Host: www.example.com"' + +# set up whole asset fetch into cache +ts.Disk.remap_config.AddLine( + 'map / http://127.0.0.1:{}'.format(server.Variables.Port) + + ' @plugin=slice.so @pparam=--blockbytes-test={}'.format(blockbytes) +) + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 0, + 'proxy.config.diags.debug.tags': 'slice', + 'proxy.config.http.cache.http': 0, + 'proxy.config.http.wait_for_cache': 0, + 'proxy.config.http.insert_age_in_response': 0, + 'proxy.config.http.insert_request_via_str': 0, + 'proxy.config.http.insert_response_via_str': 3, +}) + +# Override builtin error check as these cases will fail +# taken from the slice plug code +ts.Disk.diags_log.Content = Testers.ContainsExpression('reason="Mismatch block Etag"', "Mismatch block etag") +ts.Disk.diags_log.Content += Testers.ContainsExpression('reason="Mismatch block Last-Modified"', "Mismatch block Last-Modified") +ts.Disk.diags_log.Content += Testers.ContainsExpression('reason="Non 206 internal block response"', "Non 206 internal block response") +ts.Disk.diags_log.Content += Testers.ContainsExpression('reason="Mismatch/Bad block Content-Range"', "Mismatch/Bad block Content-Range") + +# 0 Test - Etag mismatch test +tr = Test.AddTestRun("Etag test") +tr.Processes.Default.StartBefore(server) +tr.Processes.Default.StartBefore(Test.Processes.ts) +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/etag'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold_error/etag.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold_error/etag.stderr.gold" +tr.StillRunningAfter = ts + +# 1 Check - diags.log message +tr = Test.AddTestRun("Etag error check") +tr.Processes.Default.Command = "grep 'Mismatch block Etag' {}".format(ts.Disk.diags_log.Name) +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts + +# 2 Test - Last Modified mismatch test +tr = Test.AddTestRun("Last-Modified test") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/lastmodified'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold_error/lm.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold_error/lm.stderr.gold" +tr.StillRunningAfter = ts + +# 3 Check - diags.log message +tr = Test.AddTestRun("Last-Modified error check") +tr.Processes.Default.Command = "grep 'Mismatch block Last-Modified' {}".format(ts.Disk.diags_log.Name) +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts + +# 4 Test - Non 206 mismatch test +tr = Test.AddTestRun("Non 206 test") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/non206'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold_error/non206.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold_error/non206.stderr.gold" +tr.StillRunningAfter = ts + +# 3 Check - diags.log message +tr = Test.AddTestRun("Non 206 error check") +tr.Processes.Default.Command = "grep 'Non 206 internal block response' {}".format(ts.Disk.diags_log.Name) +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts + +# 4 Test - Block content-range +tr = Test.AddTestRun("Content-Range test") +tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/crr'.format(ts.Variables.port) +tr.Processes.Default.ReturnCode = 0 +tr.Processes.Default.Streams.stdout = "gold_error/crr.stdout.gold" +tr.Processes.Default.Streams.stderr = "gold_error/crr.stderr.gold" +tr.StillRunningAfter = ts + +# 3 Check - diags.log message +tr = Test.AddTestRun("Content-Range error check") +tr.Processes.Default.Command = "grep 'Mismatch/Bad block Content-Range' {}".format(ts.Disk.diags_log.Name) +tr.Processes.Default.ReturnCode = 0 +tr.StillRunningAfter = ts