diff --git a/doc/admin-guide/plugins/regex_revalidate.en.rst b/doc/admin-guide/plugins/regex_revalidate.en.rst index 18a582350f5..30ddccfa841 100644 --- a/doc/admin-guide/plugins/regex_revalidate.en.rst +++ b/doc/admin-guide/plugins/regex_revalidate.en.rst @@ -40,36 +40,34 @@ regular expression against your origin URLs permits. Thus, individual cache objects may have rules created for them, or entire path prefixes, or even any cache objects with a particular file extension. -Installation -============ - -To make this plugin available, you must enable experimental plugins when -building |TS|:: - - ./configure --enable-experimental-plugins - Configuration ============= +.. program:: regex_revalidate.so -This plugin is enabled via the :file:`plugin.config` configuration file, with -two required arguments: the path to a rules file, and the path to a log file:: +``Regex Revalidate`` is a global plugin and is configured via :file:`plugin.config`. + .. option:: --config (also -c) - regex_revalidate.so -c -l + (`required`) - specifies the file which contains the revalidation rules. + The rule configuration file format is described below in + `Revalidation Rules`_. These rules are always reloaded when + ``traffic_ctl config reload`` is invoked. -The rule configuration file format is described below in `Revalidation Rules`_. + .. option:: --log (also -l) -By default The plugin regularly (every 60 seconds) checks its rules configuration -file for changes and it will also check for changes when ``traffic_ctl config reload`` -is run. If the file has been modified since its last scan, the contents -are read and the in-memory rules list is updated. Thus, new rules may be added and -existing ones modified without requiring a service restart. + (`optional`) - specifies path to rule logging. This log is written to + after rule changes and contains the current active ruleset. If missing + no log file is generated. -The configuration parameter `--disable-timed-updates` or `-d` may be used to configure -the plugin to disable timed config file change checks. With timed checks disabled, -config file changes are checked are only when ``traffic_ctl config reload`` is run.:: + .. option:: --disable-timed-reload (also -d) - regex_revalidate.so -d -c -l + (`optional`) - default plugin behaviour is to check the revalidate + rules file for changes every 60 seconds. This option disables the + checking. +``traffic_ctl`` + * ``traffic_ctl config reload`` - triggers a reload of the rules file. If there are no changes then the loaded rules are discarded. + * ``traffic_ctl plugin msg regex_revalidate config_reload`` - triggers a reload of the rules file apart from a full config reload. + * ``traffic_ctl plugin msg regex_revalidate config_print`` - writes the current active ruleset to :file:`traffic.out` Revalidation Rules ================== @@ -78,7 +76,7 @@ Inside your revalidation rules configuration, each rule line is defined as a regular expression followed by an integer which expresses the epoch time at which the rule will expire:: - + [type MISS or default STALE] Blank lines and lines beginning with a ``#`` character are ignored. @@ -97,6 +95,16 @@ expressed as an integer of seconds since epoch (equivalent to the return value of :manpage:`time(2)`), after which the forced revalidation will no longer occur. +Type +---- + +By default any matching asset will have its cache lookup status changed +from HIT_FRESH to HIT_STALE. By adding an extra keyword MISS at the end +of a line the asset will be marked MISS instead, forcing a refetch from +the parent. *Use with care* as this will increase bandwidth to the parent. +During configuration reload, any rule which changes it type will be +reloaded and treated as a new rule. + Caveats ======= @@ -111,11 +119,11 @@ the fact that the plugin uses :c:data:`TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK`. Removing Rules -------------- -While new rules are added dynamically (the configuration file is checked every -60 seconds for changes), rule lines removed from the configuration file do not -currently lead to that rule being removed from the running plugin. In these -cases, if the rule must be taken out of service, a service restart may be -necessary. +While new rules are added dynamically (the configuration file is checked +every 60 seconds for changes), rule lines removed from the configuration +file do not currently lead to that rule being removed from the running +plugin. To take these rules out of service the rule should be assigned a +new time in the past which will cause it to be pruned during reload phase. Examples ======== @@ -128,3 +136,6 @@ in |TS| until 6:47:27 AM on Saturday, November 14th, 2015 (UTC):: Note the escaping of the ``.`` metacharacter in the rule's regular expression. +Alternatively the following rule would case a refetch from the parent:: + + http://origin\.tld/images/foo\.jpg 1447483647 MISS diff --git a/plugins/regex_revalidate/Makefile.inc b/plugins/regex_revalidate/Makefile.inc index 30dce81389e..ec8dcbc5865 100644 --- a/plugins/regex_revalidate/Makefile.inc +++ b/plugins/regex_revalidate/Makefile.inc @@ -15,4 +15,4 @@ # limitations under the License. pkglib_LTLIBRARIES += regex_revalidate/regex_revalidate.la -regex_revalidate_regex_revalidate_la_SOURCES = regex_revalidate/regex_revalidate.c +regex_revalidate_regex_revalidate_la_SOURCES = regex_revalidate/regex_revalidate.cc regex_revalidate/regex.cc diff --git a/plugins/regex_revalidate/regex.cc b/plugins/regex_revalidate/regex.cc new file mode 100644 index 00000000000..0fbbb80fff5 --- /dev/null +++ b/plugins/regex_revalidate/regex.cc @@ -0,0 +1,115 @@ +/** @file + + PCRE support wrapper. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "regex.h" + +#ifdef PCRE_CONFIG_JIT +#include + +struct RegexThreadKey { + RegexThreadKey() { pthread_key_create(&this->key, reinterpret_cast(&pcre_jit_stack_free)); } + pthread_key_t key; +}; + +static RegexThreadKey k; + +static pcre_jit_stack * +get_jit_stack(void *) +{ + pcre_jit_stack *jit_stack; + + if ((jit_stack = static_cast(pthread_getspecific(k.key))) == nullptr) { + jit_stack = pcre_jit_stack_alloc(8192, 1024 * 1024); // 1 page min and 1MB max + pthread_setspecific(k.key, (void *)jit_stack); + } + + return jit_stack; +} +#endif + +bool +Regex::compile(const char *pattern, const unsigned flags) +{ + const char *error; + int erroffset; + int options = 0; + int study_opts = 0; + + if (nullptr != this->regex) { + return false; + } + + if (flags & CASE_INSENSITIVE) { + options |= PCRE_CASELESS; + } + + if (flags & ANCHORED) { + options |= PCRE_ANCHORED; + } + + this->regex = pcre_compile(pattern, options, &error, &erroffset, nullptr); + if (error) { + this->regex = nullptr; + return false; + } + +#ifdef PCRE_CONFIG_JIT + study_opts |= PCRE_STUDY_JIT_COMPILE; +#endif + + this->regex_extra = pcre_study(this->regex, study_opts, &error); + +#ifdef PCRE_CONFIG_JIT + if (nullptr != this->regex_extra) { + pcre_assign_jit_stack(this->regex_extra, &get_jit_stack, nullptr); + } +#endif + + return true; +} + +bool +Regex::matches(std::string_view const &src) const +{ + return 0 <= exec(src, nullptr, 0); +} + +int +Regex::exec(std::string_view const &src, int *ovector, int const ovecsize) const +{ + return pcre_exec(this->regex, this->regex_extra, src.data(), src.size(), 0, 0, ovector, ovecsize); +} + +Regex::~Regex() +{ + if (regex_extra) { +#ifdef PCRE_CONFIG_JIT + pcre_free_study(regex_extra); +#else + pcre_free(regex_extra); +#endif + } + if (regex) { + pcre_free(regex); + } +} diff --git a/plugins/regex_revalidate/regex.h b/plugins/regex_revalidate/regex.h new file mode 100644 index 00000000000..d6f07136163 --- /dev/null +++ b/plugins/regex_revalidate/regex.h @@ -0,0 +1,83 @@ +/** @file + + Wrapper to make PCRE handling easier. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#pragma once + +#include +#include +#include + +class Regex +{ +public: + enum Flag { + CASE_INSENSITIVE = 0x0001, // default is case sensitive + UNANCHORED = 0x0002, // default (for DFA) is to anchor at the first matching position + ANCHORED = 0x0004, // default (for Regex) is unanchored + }; + + Regex() = default; + Regex(Regex const &) = delete; + Regex &operator=(Regex const &) = delete; + Regex(Regex &&that); + Regex &operator=(Regex &&that); + + bool compile(const char *pattern, const unsigned flags = 0); + + // check if valid regex + bool is_valid() const; + + // check for simple match + bool matches(std::string_view const &src) const; + + // match returning substring positions. + int exec(std::string_view const &src, int *ovector, int const ovecsize) const; + + ~Regex(); + +private: + pcre *regex = nullptr; + pcre_extra *regex_extra = nullptr; +}; + +inline Regex::Regex(Regex &&that) +{ + std::swap(regex, that.regex); + std::swap(regex_extra, that.regex_extra); +} + +inline Regex & +Regex::operator=(Regex &&that) +{ + if (&that != this) { + std::swap(regex, that.regex); + std::swap(regex_extra, that.regex_extra); + } + return *this; +} + +inline bool +Regex::is_valid() const +{ + return nullptr != regex; +} diff --git a/plugins/regex_revalidate/regex_revalidate.c b/plugins/regex_revalidate/regex_revalidate.c deleted file mode 100644 index 2fa32c5e5e4..00000000000 --- a/plugins/regex_revalidate/regex_revalidate.c +++ /dev/null @@ -1,524 +0,0 @@ -/** @file - - @section license License - - Licensed to the Apache Software Foundation (ASF) under one - or more contributor license agreements. See the NOTICE file - distributed with this work for additional information - regarding copyright ownership. The ASF licenses this file - to you under the Apache License, Version 2.0 (the - "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at - - http://www.apache.org/licenses/LICENSE-2.0 - - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. - */ - -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#ifdef HAVE_PCRE_PCRE_H -#include -#else -#include -#endif - -#define LOG_PREFIX "regex_revalidate" -#define CONFIG_TMOUT 60000 -#define FREE_TMOUT 300000 -#define OVECTOR_SIZE 30 -#define LOG_ROLL_INTERVAL 86400 -#define LOG_ROLL_OFFSET 0 - -typedef struct invalidate_t { - const char *regex_text; - pcre *regex; - pcre_extra *regex_extra; - time_t epoch; - time_t expiry; - struct invalidate_t *next; -} invalidate_t; - -typedef struct { - invalidate_t *invalidate_list; - char *config_file; - time_t last_load; - TSTextLogObject log; -} plugin_state_t; - -static invalidate_t * -init_invalidate_t(invalidate_t *i) -{ - i->regex_text = NULL; - i->regex = NULL; - i->regex_extra = NULL; - i->epoch = 0; - i->expiry = 0; - i->next = NULL; - return i; -} - -static void -free_invalidate_t(invalidate_t *i) -{ - if (i->regex_extra) { -#ifndef PCRE_STUDY_JIT_COMPILE - pcre_free(i->regex_extra); -#else - pcre_free_study(i->regex_extra); -#endif - } - if (i->regex) { - pcre_free(i->regex); - } - if (i->regex_text) { - pcre_free_substring(i->regex_text); - } - TSfree(i); -} - -static void -free_invalidate_t_list(invalidate_t *i) -{ - if (i->next) { - free_invalidate_t_list(i->next); - } - free_invalidate_t(i); -} - -static plugin_state_t * -init_plugin_state_t(plugin_state_t *pstate) -{ - pstate->invalidate_list = NULL; - pstate->config_file = NULL; - pstate->last_load = 0; - pstate->log = NULL; - return pstate; -} - -static void -free_plugin_state_t(plugin_state_t *pstate) -{ - if (pstate->invalidate_list) { - free_invalidate_t_list(pstate->invalidate_list); - } - if (pstate->config_file) { - TSfree(pstate->config_file); - } - if (pstate->log) { - TSTextLogObjectDestroy(pstate->log); - } - TSfree(pstate); -} - -static invalidate_t * -copy_invalidate_t(invalidate_t *i) -{ - invalidate_t *iptr; - const char *errptr; - int erroffset; - - iptr = (invalidate_t *)TSmalloc(sizeof(invalidate_t)); - iptr->regex_text = TSstrdup(i->regex_text); - iptr->regex = pcre_compile(iptr->regex_text, 0, &errptr, &erroffset, NULL); // There is no pcre_copy :-( - iptr->regex_extra = pcre_study(iptr->regex, 0, &errptr); // Assuming no errors since this worked before :-/ - iptr->epoch = i->epoch; - iptr->expiry = i->expiry; - iptr->next = NULL; - return iptr; -} - -static invalidate_t * -copy_config(invalidate_t *old_list) -{ - invalidate_t *new_list = NULL; - invalidate_t *iptr_old, *iptr_new; - - if (old_list) { - new_list = copy_invalidate_t(old_list); - iptr_old = old_list->next; - iptr_new = new_list; - while (iptr_old) { - iptr_new->next = copy_invalidate_t(iptr_old); - iptr_new = iptr_new->next; - iptr_old = iptr_old->next; - } - } - - return new_list; -} - -static bool -prune_config(invalidate_t **i) -{ - invalidate_t *iptr, *ilast; - time_t now; - bool pruned = false; - - now = time(NULL); - - if (*i) { - iptr = *i; - ilast = NULL; - while (iptr) { - if (difftime(iptr->expiry, now) < 0) { - TSDebug(LOG_PREFIX, "Removing %s expiry: %d now: %d", iptr->regex_text, (int)iptr->expiry, (int)now); - if (ilast) { - ilast->next = iptr->next; - free_invalidate_t(iptr); - iptr = ilast->next; - } else { - *i = iptr->next; - free_invalidate_t(iptr); - iptr = *i; - } - pruned = true; - } else { - ilast = iptr; - iptr = iptr->next; - } - } - } - return pruned; -} - -static bool -load_config(plugin_state_t *pstate, invalidate_t **ilist) -{ - FILE *fs; - struct stat s; - size_t path_len; - char *path; - char line[LINE_MAX]; - time_t now; - pcre *config_re; - const char *errptr; - int erroffset, ovector[OVECTOR_SIZE], rc; - int ln = 0; - invalidate_t *iptr, *i; - - if (pstate->config_file[0] != '/') { - path_len = strlen(TSConfigDirGet()) + strlen(pstate->config_file) + 2; - path = alloca(path_len); - snprintf(path, path_len, "%s/%s", TSConfigDirGet(), pstate->config_file); - } else { - path = pstate->config_file; - } - if (stat(path, &s) < 0) { - TSDebug(LOG_PREFIX, "Could not stat %s", path); - return false; - } - if (s.st_mtime > pstate->last_load) { - now = time(NULL); - if (!(fs = fopen(path, "r"))) { - TSDebug(LOG_PREFIX, "Could not open %s for reading", path); - return false; - } - config_re = pcre_compile("^([^#].+?)\\s+(\\d+)\\s*$", 0, &errptr, &erroffset, NULL); - while (fgets(line, LINE_MAX, fs) != NULL) { - ln++; - TSDebug(LOG_PREFIX, "Processing: %d %s", ln, line); - rc = pcre_exec(config_re, NULL, line, strlen(line), 0, 0, ovector, OVECTOR_SIZE); - if (rc == 3) { - i = (invalidate_t *)TSmalloc(sizeof(invalidate_t)); - init_invalidate_t(i); - pcre_get_substring(line, ovector, rc, 1, &i->regex_text); - i->epoch = now; - i->expiry = atoi(line + ovector[4]); - i->regex = pcre_compile(i->regex_text, 0, &errptr, &erroffset, NULL); - if (i->expiry <= i->epoch) { - TSDebug(LOG_PREFIX, "Rule is already expired!"); - free_invalidate_t(i); - } else if (i->regex == NULL) { - TSDebug(LOG_PREFIX, "%s did not compile", i->regex_text); - free_invalidate_t(i); - } else { - i->regex_extra = pcre_study(i->regex, 0, &errptr); - if (!*ilist) { - *ilist = i; - TSDebug(LOG_PREFIX, "Created new list and Loaded %s %d %d", i->regex_text, (int)i->epoch, (int)i->expiry); - } else { - iptr = *ilist; - while (1) { - if (strcmp(i->regex_text, iptr->regex_text) == 0) { - if (iptr->expiry != i->expiry) { - TSDebug(LOG_PREFIX, "Updating duplicate %s", i->regex_text); - iptr->epoch = i->epoch; - iptr->expiry = i->expiry; - } - free_invalidate_t(i); - i = NULL; - break; - } else if (!iptr->next) { - break; - } else { - iptr = iptr->next; - } - } - if (i) { - iptr->next = i; - TSDebug(LOG_PREFIX, "Loaded %s %d %d", i->regex_text, (int)i->epoch, (int)i->expiry); - } - } - } - } else { - TSDebug(LOG_PREFIX, "Skipping line %d", ln); - } - } - pcre_free(config_re); - fclose(fs); - pstate->last_load = s.st_mtime; - return true; - } else { - TSDebug(LOG_PREFIX, "File mod time is not newer: %d >= %d", (int)pstate->last_load, (int)s.st_mtime); - } - return false; -} - -static void -list_config(plugin_state_t *pstate, invalidate_t *i) -{ - invalidate_t *iptr; - - TSDebug(LOG_PREFIX, "Current config:"); - if (pstate->log) { - TSTextLogObjectWrite(pstate->log, "Current config:"); - } - if (i) { - iptr = i; - while (iptr) { - TSDebug(LOG_PREFIX, "%s epoch: %d expiry: %d", iptr->regex_text, (int)iptr->epoch, (int)iptr->expiry); - if (pstate->log) { - TSTextLogObjectWrite(pstate->log, "%s epoch: %d expiry: %d", iptr->regex_text, (int)iptr->epoch, (int)iptr->expiry); - } - iptr = iptr->next; - } - } else { - TSDebug(LOG_PREFIX, "EMPTY"); - if (pstate->log) { - TSTextLogObjectWrite(pstate->log, "EMPTY"); - } - } -} - -static int -free_handler(TSCont cont, TSEvent event, void *edata) -{ - invalidate_t *iptr; - - TSDebug(LOG_PREFIX, "Freeing old config"); - iptr = (invalidate_t *)TSContDataGet(cont); - free_invalidate_t_list(iptr); - TSContDestroy(cont); - return 0; -} - -static int -config_handler(TSCont cont, TSEvent event, void *edata) -{ - plugin_state_t *pstate; - invalidate_t *i, *iptr; - TSCont free_cont; - bool updated; - TSMutex mutex; - - mutex = TSContMutexGet(cont); - TSMutexLock(mutex); - - TSDebug(LOG_PREFIX, "In config Handler"); - pstate = (plugin_state_t *)TSContDataGet(cont); - i = copy_config(pstate->invalidate_list); - - updated = prune_config(&i); - updated = load_config(pstate, &i) || updated; - - if (updated) { - list_config(pstate, i); - iptr = __sync_val_compare_and_swap(&(pstate->invalidate_list), pstate->invalidate_list, i); - - if (iptr) { - free_cont = TSContCreate(free_handler, TSMutexCreate()); - TSContDataSet(free_cont, (void *)iptr); - TSContScheduleOnPool(free_cont, FREE_TMOUT, TS_THREAD_POOL_TASK); - } - } else { - TSDebug(LOG_PREFIX, "No Changes"); - if (i) { - free_invalidate_t_list(i); - } - } - - TSMutexUnlock(mutex); - - // Don't reschedule for TS_EVENT_MGMT_UPDATE - if (event == TS_EVENT_TIMEOUT) { - TSContScheduleOnPool(cont, CONFIG_TMOUT, TS_THREAD_POOL_TASK); - } - return 0; -} - -static time_t -get_date_from_cached_hdr(TSHttpTxn txn) -{ - TSMBuffer buf; - TSMLoc hdr_loc, date_loc; - time_t date = 0; - - if (TSHttpTxnCachedRespGet(txn, &buf, &hdr_loc) == TS_SUCCESS) { - date_loc = TSMimeHdrFieldFind(buf, hdr_loc, TS_MIME_FIELD_DATE, TS_MIME_LEN_DATE); - if (date_loc != TS_NULL_MLOC) { - date = TSMimeHdrFieldValueDateGet(buf, hdr_loc, date_loc); - TSHandleMLocRelease(buf, hdr_loc, date_loc); - } - TSHandleMLocRelease(buf, TS_NULL_MLOC, hdr_loc); - } - - return date; -} - -static int -main_handler(TSCont cont, TSEvent event, void *edata) -{ - TSHttpTxn txn = (TSHttpTxn)edata; - int status; - invalidate_t *iptr; - plugin_state_t *pstate; - - time_t date = 0, now = 0; - char *url = NULL; - int url_len = 0; - - switch (event) { - case TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE: - if (TSHttpTxnCacheLookupStatusGet(txn, &status) == TS_SUCCESS) { - if (status == TS_CACHE_LOOKUP_HIT_FRESH) { - pstate = (plugin_state_t *)TSContDataGet(cont); - iptr = pstate->invalidate_list; - while (iptr) { - if (!date) { - date = get_date_from_cached_hdr(txn); - now = time(NULL); - } - if ((difftime(iptr->epoch, date) >= 0) && (difftime(iptr->expiry, now) >= 0)) { - if (!url) { - url = TSHttpTxnEffectiveUrlStringGet(txn, &url_len); - } - if (pcre_exec(iptr->regex, iptr->regex_extra, url, url_len, 0, 0, NULL, 0) >= 0) { - TSHttpTxnCacheLookupStatusSet(txn, TS_CACHE_LOOKUP_HIT_STALE); - iptr = NULL; - TSDebug(LOG_PREFIX, "Forced revalidate - %.*s", url_len, url); - } - } - if (iptr) { - iptr = iptr->next; - } - } - if (url) { - TSfree(url); - } - } - } - break; - default: - break; - } - - TSHttpTxnReenable(txn, TS_EVENT_HTTP_CONTINUE); - return 0; -} - -void -TSPluginInit(int argc, const char *argv[]) -{ - TSPluginRegistrationInfo info; - TSCont main_cont, config_cont; - plugin_state_t *pstate; - invalidate_t *iptr = NULL; - bool disable_timed_reload = false; - - TSDebug(LOG_PREFIX, "Starting plugin init"); - - pstate = (plugin_state_t *)TSmalloc(sizeof(plugin_state_t)); - init_plugin_state_t(pstate); - - int c; - static const struct option longopts[] = {{"config", required_argument, NULL, 'c'}, - {"log", required_argument, NULL, 'l'}, - {"disable-timed-reload", no_argument, NULL, 'd'}, - {NULL, 0, NULL, 0}}; - - while ((c = getopt_long(argc, (char *const *)argv, "c:l:", longopts, NULL)) != -1) { - switch (c) { - case 'c': - pstate->config_file = TSstrdup(optarg); - break; - case 'l': - if (TS_SUCCESS == TSTextLogObjectCreate(optarg, TS_LOG_MODE_ADD_TIMESTAMP, &pstate->log)) { - TSTextLogObjectRollingIntervalSecSet(pstate->log, LOG_ROLL_INTERVAL); - TSTextLogObjectRollingOffsetHrSet(pstate->log, LOG_ROLL_OFFSET); - } - break; - case 'd': - disable_timed_reload = true; - break; - default: - break; - } - } - - if (!pstate->config_file) { - TSError("[regex_revalidate] Plugin requires a --config option along with a config file name"); - free_plugin_state_t(pstate); - return; - } - - if (!load_config(pstate, &iptr)) { - TSDebug(LOG_PREFIX, "Problem loading config from file %s", pstate->config_file); - } else { - pstate->invalidate_list = iptr; - list_config(pstate, iptr); - } - - info.plugin_name = LOG_PREFIX; - info.vendor_name = "Apache Software Foundation"; - info.support_email = "dev@trafficserver.apache.org"; - - if (TSPluginRegister(&info) != TS_SUCCESS) { - TSError("[regex_revalidate] Plugin registration failed"); - - free_plugin_state_t(pstate); - return; - } else { - TSDebug(LOG_PREFIX, "Plugin registration succeeded"); - } - - main_cont = TSContCreate(main_handler, NULL); - TSContDataSet(main_cont, (void *)pstate); - TSHttpHookAdd(TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK, main_cont); - - config_cont = TSContCreate(config_handler, TSMutexCreate()); - TSContDataSet(config_cont, (void *)pstate); - - TSMgmtUpdateRegister(config_cont, LOG_PREFIX); - - if (!disable_timed_reload) { - TSContScheduleOnPool(config_cont, CONFIG_TMOUT, TS_THREAD_POOL_TASK); - } - - TSDebug(LOG_PREFIX, "Plugin Init Complete"); -} diff --git a/plugins/regex_revalidate/regex_revalidate.cc b/plugins/regex_revalidate/regex_revalidate.cc new file mode 100644 index 00000000000..73e5ca99147 --- /dev/null +++ b/plugins/regex_revalidate/regex_revalidate.cc @@ -0,0 +1,747 @@ +/** @file + + ATS plugin to do (simple) regular expression remap rules + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "ts/ts.h" +#include "ts/remap.h" +#include "ts/experimental.h" +#include "regex.h" + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef HAVE_PCRE_PCRE_H +#include +#else +#include +#endif + +namespace +{ +constexpr char const *const PLUGIN_NAME = "regex_revalidate"; +constexpr char const *const RELOAD_TAG = "config_reload"; +constexpr char const *const PRINT_TAG = "config_print"; + +constexpr char const *const RESULT_MISS = "MISS"; +constexpr char const *const RESULT_STALE = "STALE"; + +#define DEBUG_LOG(fmt, ...) TSDebug(PLUGIN_NAME, "%s:%d " fmt, __func__, __LINE__, ##__VA_ARGS__) + +#define ERROR_LOG(fmt, ...) \ + TSError("(%s) " fmt, PLUGIN_NAME, ##__VA_ARGS__); \ + DEBUG_LOG(fmt, ##__VA_ARGS__) + +constexpr TSHRTime const CONFIG_TMOUT = 60000; // ms, 60s +constexpr int const OVECTOR_SIZE = 30; +constexpr int const LOG_ROLL_INTERVAL = 86400; +constexpr int const LOG_ROLL_OFFSET = 0; + +TSCont config_cont{nullptr}; + +struct Invalidate { + static Regex config_re; + + std::string regex_text{}; + Regex regex{}; + time_t epoch{0}; + time_t expiry{0}; + TSCacheLookupResult new_result{TS_CACHE_LOOKUP_HIT_STALE}; + + Invalidate() = default; + + Invalidate(Invalidate const &orig) + { + regex_text = orig.regex_text; + regex.compile(regex_text.c_str()); + epoch = orig.epoch; + expiry = orig.expiry; + new_result = orig.new_result; + } + + Invalidate(Invalidate &&orig) + { + std::swap(regex_text, orig.regex_text); + std::swap(regex, orig.regex); + std::swap(epoch, orig.epoch); + std::swap(expiry, orig.expiry); + std::swap(new_result, orig.new_result); + } + + Invalidate & + operator=(Invalidate &&rhs) + { + if (&rhs != this) { + this->~Invalidate(); + new (this) Invalidate(std::move(rhs)); + } + return *this; + } + + Invalidate & + operator=(Invalidate const &rhs) + { + if (&rhs != this) { + this->~Invalidate(); + new (this) Invalidate(rhs); + } + return *this; + } + + ~Invalidate() = default; + + inline bool + is_valid() const + { + return regex.is_valid(); + } + + inline bool + matches(char const *const url, int const url_len) const + { + return regex.matches(std::string_view{url, (unsigned)url_len}); + } + + static Invalidate fromLine(time_t const epoch, char *const line); +}; + +Regex Invalidate::config_re; + +struct PluginState { + std::string config_file{}; + std::shared_ptr> invalidate_vec; // sorted by regex_text + bool timed_reload{true}; + time_t config_file_mtime{0}; + time_t min_expiry{0}; + TSTextLogObject log{nullptr}; + + ~PluginState() + { + if (nullptr != log) { + TSTextLogObjectDestroy(log); + } + } + + bool fromArgs(int argc, char const **argv); + bool loadConfig(time_t const timenow, std::vector *const newrules) const; + + void printConfig() const; + void logConfig() const; +}; + +Invalidate +Invalidate::fromLine(time_t const epoch, char *const line) +{ + Invalidate rule; + + int ovector[OVECTOR_SIZE]; + DEBUG_LOG("'%s'", line); + int const rc = config_re.exec(line, ovector, OVECTOR_SIZE); + + if (3 <= rc) { + int const regbeg = ovector[2]; + int const regend = ovector[3]; + + char const *const regstr = line + regbeg; + char const oldch = line[regend]; + line[regend] = '\0'; // temporarily inject null termination + + if (rule.regex.compile(regstr)) { + rule.regex_text = regstr; + rule.expiry = (time_t)atoll(line + ovector[4]); + rule.epoch = epoch; + } else { + DEBUG_LOG("Invalid regex in line: '%s'", regstr); + } + + if (5 == rc) { + char const *const type = line + ovector[8]; + if (0 == strcasecmp(type, RESULT_MISS)) { + DEBUG_LOG("Regex line set to result type %s: '%s'", RESULT_MISS, regstr); + rule.new_result = TS_CACHE_LOOKUP_MISS; + } else if (0 != strcasecmp(type, RESULT_STALE)) { + DEBUG_LOG("Unknown regex line result type '%s', using %s '%s'", type, RESULT_STALE, regstr); + } + } + + // restore the line char* + line[regend] = oldch; + } + + return rule; +} + +void +get_time_now_str(char *const buf, size_t const buflen) +{ + TSHRTime const timenowusec = TShrtime(); + int64_t const timemsec = static_cast(timenowusec / 1000000); + time_t const timesec = static_cast(timemsec / 1000); + int const ms = static_cast(timemsec % 1000); + + struct tm tm; + gmtime_r(×ec, &tm); + size_t const dtlen = strftime(buf, buflen, "%b %e %H:%M:%S", &tm); + + // tack on the ms + snprintf(buf + dtlen, buflen - dtlen, ".%03d", ms); +} + +void +PluginState::printConfig() const +{ + char timebuf[64] = ""; + get_time_now_str(timebuf, sizeof(timebuf)); + + fprintf(stderr, "[%s] %s config file: %s\n", timebuf, PLUGIN_NAME, config_file.c_str()); + + if (invalidate_vec) { + for (Invalidate const &iv : *invalidate_vec) { + char const *const typestr = (iv.new_result == TS_CACHE_LOOKUP_MISS ? RESULT_MISS : RESULT_STALE); + fprintf(stderr, "[%s] %s line: '%s' epoch: %ju expiry: %ju result: '%s'\n", timebuf, PLUGIN_NAME, iv.regex_text.c_str(), + (uintmax_t)iv.epoch, (uintmax_t)iv.expiry, typestr); + } + } else { + fprintf(stderr, "[%s] %s config: EMPTY\n", timebuf, PLUGIN_NAME); + } + + fflush(stderr); +} + +void +PluginState::logConfig() const +{ + TSDebug(PLUGIN_NAME, "Current config: %s", config_file.c_str()); + if (nullptr != log) { + TSTextLogObjectWrite(log, "Current config: %s", config_file.c_str()); + } + + if (invalidate_vec) { + for (Invalidate const &iv : *invalidate_vec) { + char const *const typestr = (iv.new_result == TS_CACHE_LOOKUP_MISS ? RESULT_MISS : RESULT_STALE); + TSDebug(PLUGIN_NAME, "line: '%s' epoch: %ju expiry: %ju result: '%s'", iv.regex_text.c_str(), (uintmax_t)iv.epoch, + (uintmax_t)iv.expiry, typestr); + if (nullptr != log) { + TSTextLogObjectWrite(log, "line: '%s' epoch: %ju expiry: %ju result: '%s'", iv.regex_text.c_str(), (uintmax_t)iv.epoch, + (uintmax_t)iv.expiry, typestr); + } + } + } else { + TSDebug(PLUGIN_NAME, "Configuration EMPTY"); + if (nullptr != log) { + TSTextLogObjectWrite(log, "EMPTY"); + } + } +} + +bool +PluginState::fromArgs(int argc, char const **argv) +{ + int c; + constexpr option const longopts[] = { + {"config", required_argument, nullptr, 'c'}, + {"disable-timed-reload", no_argument, nullptr, 'd'}, + {"log", required_argument, nullptr, 'l'}, + {nullptr, 0, nullptr, 0}, + }; + + while ((c = getopt_long(argc, (char *const *)argv, "c:l:d", longopts, nullptr)) != -1) { + switch (c) { + case 'c': + config_file = optarg; + + // path is relative to config dir + if ('/' != config_file[0]) { + config_file = std::string(TSConfigDirGet()) + "/" + config_file; + } + + DEBUG_LOG("Config File: %s", config_file.c_str()); + break; + case 'l': + if (TS_SUCCESS == TSTextLogObjectCreate(optarg, TS_LOG_MODE_ADD_TIMESTAMP, &log)) { + TSTextLogObjectRollingEnabledSet(log, 1); + TSTextLogObjectRollingIntervalSecSet(log, LOG_ROLL_INTERVAL); + TSTextLogObjectRollingOffsetHrSet(log, LOG_ROLL_OFFSET); + DEBUG_LOG("Logging Mode enabled"); + } + break; + case 'd': + timed_reload = false; + DEBUG_LOG("Timed reload disabled (disable-timed-reload)"); + break; + default: + break; + } + } + + if (config_file.empty()) { + ERROR_LOG("Plugin requires a --config option along with a config file name"); + return false; + } + + return true; +} + +time_t +timeForFile(std::string const &filepath) +{ + time_t mtime{0}; + struct stat fstat; + if (0 == stat(filepath.c_str(), &fstat)) { + mtime = fstat.st_mtime; + } else { + DEBUG_LOG("Could not stat %s", filepath.c_str()); + } + return mtime; +} + +// load config, true if rules changed +bool +PluginState::loadConfig(time_t const timenow, std::vector *const rules) const +{ + TSAssert(nullptr != rules); + + FILE *const fs = fopen(config_file.c_str(), "r"); + if (nullptr == fs) { + DEBUG_LOG("Could not open %s for reading", config_file.c_str()); + return false; + } + + // load from file + std::vector loaded; + int lineno = 0; + char line[LINE_MAX]; + while (nullptr != fgets(line, LINE_MAX, fs)) { + ++lineno; + line[strcspn(line, "\r\n")] = '\0'; + if (0 < strlen(line) && '#' != line[0]) { + Invalidate rnew = Invalidate::fromLine(timenow, line); + if (rnew.is_valid()) { + loaded.push_back(std::move(rnew)); + } else { + DEBUG_LOG("Invalid rule '%s' from line: '%d'", line, lineno); + } + } + } + + fclose(fs); + + if (loaded.empty()) { + DEBUG_LOG("No rules loaded from file '%s'", config_file.c_str()); + return false; + } + + // stable sort to make clearing duplicates easy + std::stable_sort(loaded.begin(), loaded.end(), + [](Invalidate const &lhs, Invalidate const &rhs) { return lhs.regex_text < rhs.regex_text; }); + + // sweep to clear duplicates, last one wins + for (size_t index = 0; index < (loaded.size() - 1); ++index) { + if (loaded[index].regex_text == loaded[index + 1].regex_text) { + loaded[index] = Invalidate{}; + } + } + + if (!this->invalidate_vec || this->invalidate_vec->empty()) { + DEBUG_LOG("Installing fresh rules"); + *rules = std::move(loaded); + return true; + } + + DEBUG_LOG("Merging new config"); + + // merge loaded and current rule set + auto const &cur = *(this->invalidate_vec); + auto itload = loaded.begin(); + auto itcur = cur.cbegin(); + + bool changed = false; + + // reimplementation of std::set_union + while (cur.cend() != itcur) { + if (loaded.cend() == itload) { + std::copy(itcur, cur.cend(), rules->end()); + break; + } + + // fast forward over cleared items + while (!itload->is_valid()) { + DEBUG_LOG("Skipping cleared duplicate rule"); + ++itload; + TSAssert(loaded.end() != itload); // last item will always be valid. + } + + int const cmp = itload->regex_text.compare(itcur->regex_text); + if (cmp < 0) { + if (timenow < itload->expiry) { + DEBUG_LOG("Adding new rule: '%s'", itload->regex_text.c_str()); + rules->push_back(std::move(*itload)); + changed = true; + } else { + DEBUG_LOG("Not adding new expired rule: '%s'", itload->regex_text.c_str()); + } + ++itload; + } else if (0 < cmp) { + DEBUG_LOG("Retaining old rule: '%s'", itcur->regex_text.c_str()); + rules->push_back(*itcur); + ++itcur; + } else { + if (itload->expiry != itcur->expiry || itload->new_result != itcur->new_result) { + DEBUG_LOG("Updating rule: '%s'", itload->regex_text.c_str()); + rules->push_back(std::move(*itload)); + changed = true; + } else { + DEBUG_LOG("Using old rule: '%s'", itcur->regex_text.c_str()); + rules->push_back(*itcur); + } + ++itcur; + ++itload; + } + } + + // any leftover loaded rules get tacked on + while (loaded.end() != itload) { + DEBUG_LOG("Adding new rule: '%s'", itload->regex_text.c_str()); + rules->push_back(std::move(*itload)); + ++itload; + changed = true; + } + + DEBUG_LOG("Rules have been changed: '%s'", changed ? "true" : "false"); + + return changed; +} + +// remove expired rules +bool +pruneConfig(time_t const timenow, std::vector *const rules, time_t *const min_expiry) +{ + TSAssert(nullptr != rules); + TSAssert(nullptr != min_expiry); + + bool pruned = false; + if (timenow < *min_expiry || nullptr == rules) { + return pruned; + } + + // recalculate min_expiry + *min_expiry = std::numeric_limits::max(); + + auto iter = rules->begin(); + while (rules->end() != iter) { + Invalidate const &inv = *iter; + + if (inv.expiry < timenow) { + DEBUG_LOG("Removing rule: %s", inv.regex_text.c_str()); + iter = rules->erase(iter); // retain sort order + pruned = true; + } else { + *min_expiry = std::min(*min_expiry, inv.expiry); + ++iter; + } + } + + std::vector::iterator itvec(rules->begin()); + + while (rules->end() != itvec) { + Invalidate const &inv = *itvec; + + if (inv.expiry < timenow) { + DEBUG_LOG("Removing rule: %s", inv.regex_text.c_str()); + itvec = rules->erase(itvec); // retain sort order + pruned = true; + } else { + *min_expiry = std::min(*min_expiry, inv.expiry); + ++itvec; + } + } + + if (pruned && rules->empty()) { + DEBUG_LOG("All rules pruned"); + } + + return pruned; +} + +int +config_handler(TSCont config_cont, TSEvent event, void *edata) +{ + DEBUG_LOG("config_handler, event: %d", event); + + // config_cont's mutex is already locked + PluginState *const pstate = static_cast(TSContDataGet(config_cont)); + + bool should_reload = false; + + switch (event) { + case TS_EVENT_TIMEOUT: { + should_reload = pstate->timed_reload; + } break; + case TS_EVENT_MGMT_UPDATE: { + should_reload = true; + } break; + case TS_EVENT_LIFECYCLE_MSG: { + // ensure the message is for regex_revalidate + TSPluginMsg *const msgp = (TSPluginMsg *)edata; + if (0 == strncasecmp(msgp->tag, PLUGIN_NAME, strlen(msgp->tag))) { + char const *const msgstr = static_cast(msgp->data); + int const msglen = static_cast(msgp->data_size); + DEBUG_LOG("Lifecycle plugin message received: %.*s", msglen, msgstr); + if (0 == strncasecmp(msgstr, RELOAD_TAG, msglen)) { + should_reload = true; + } else if (0 == strncasecmp(msgstr, PRINT_TAG, msglen)) { + pstate->printConfig(); + } else { + ERROR_LOG("Unrecognized lifecycle message %.*s", msglen, msgstr); + } + } + } break; + default: // unknown message + return 0; + break; + } + + time_t const timenow = time(nullptr); + + bool rules_changed = false; + std::vector newrules; + + if (should_reload) { + time_t const mtime = timeForFile(pstate->config_file); + if (mtime != pstate->config_file_mtime) { + rules_changed = pstate->loadConfig(timenow, &newrules); + pstate->config_file_mtime = mtime; + + // this sets off a new min expiry scan + if (rules_changed) { + pstate->min_expiry = 0; + } + } + } + + // prune + if (pstate->min_expiry < timenow) { + if (newrules.empty()) { + newrules = *(pstate->invalidate_vec); + } + + time_t new_expiry = 0; + bool const pruned = pruneConfig(timenow, &newrules, &new_expiry); + pstate->min_expiry = new_expiry; + + if (pruned) { + rules_changed = true; + } + } + + if (rules_changed) { + auto vecnew = std::make_shared>(std::move(newrules)); + + std::atomic_store(&(pstate->invalidate_vec), vecnew); + DEBUG_LOG("New configuation installed"); + pstate->logConfig(); + } + + // Reschedule the continuous pruning/reload job + if (TS_EVENT_TIMEOUT == event) { + TSContScheduleOnPool(config_cont, CONFIG_TMOUT, TS_THREAD_POOL_TASK); + } + + return 0; +} + +time_t +get_date_from_cached_hdr(TSHttpTxn txn) +{ + TSMBuffer buf; + TSMLoc hdr_loc, date_loc; + time_t date = 0; + + if (TSHttpTxnCachedRespGet(txn, &buf, &hdr_loc) == TS_SUCCESS) { + date_loc = TSMimeHdrFieldFind(buf, hdr_loc, TS_MIME_FIELD_DATE, TS_MIME_LEN_DATE); + if (nullptr != date_loc) { + date = TSMimeHdrFieldValueDateGet(buf, hdr_loc, date_loc); + TSHandleMLocRelease(buf, hdr_loc, date_loc); + } + TSHandleMLocRelease(buf, nullptr, hdr_loc); + } + + return date; +} + +int +main_handler(TSCont cont, TSEvent event, void *edata) +{ + TSHttpTxn txn = (TSHttpTxn)edata; + int status = TS_ERROR; + + switch (event) { + case TS_EVENT_HTTP_CACHE_LOOKUP_COMPLETE: + if (TSHttpTxnCacheLookupStatusGet(txn, &status) == TS_SUCCESS) { + if (status == TS_CACHE_LOOKUP_HIT_FRESH) { + PluginState *const pstate = static_cast(TSContDataGet(cont)); + // atomcally grab a handle + using InvHandle = std::shared_ptr>; + InvHandle const vecinv = std::atomic_load(&(pstate->invalidate_vec)); + + if (vecinv) { + time_t const date = get_date_from_cached_hdr(txn); + time_t const timenow = time(nullptr); + char *url = nullptr; + int url_len = 0; + + for (Invalidate const &inv : *vecinv) { + if (date <= inv.epoch && timenow <= inv.expiry) { + if (nullptr == url) { + url = TSHttpTxnEffectiveUrlStringGet(txn, &url_len); + } + + if (nullptr == url || 0 == url_len) { + break; + } else if (inv.matches(url, url_len)) { + TSHttpTxnCacheLookupStatusSet(txn, inv.new_result); + DEBUG_LOG("Forced revalidate - %.*s", url_len, url); + break; + } + } + } + + if (nullptr != url) { + TSfree(url); + } + } + } + } + break; + default: + break; + } + + TSHttpTxnReenable(txn, TS_EVENT_HTTP_CONTINUE); + return 0; +} + +} // namespace + +namespace +{ +void +setup_config_cont(PluginState *const pstate) +{ + if (nullptr == config_cont) { + DEBUG_LOG("creating config continuation"); + + // create reusable configuration file regex + constexpr char const *const expr = "^([^#].+?)\\s+(\\d+)(\\s+(\\w+))?\\s*$"; + bool const cstat = Invalidate::config_re.compile(expr); + + if (!cstat) { + ERROR_LOG("setup_config_cont: Unable to compile config_re, disabling plugin"); + return; + } + + // set up the config continuation + config_cont = TSContCreate(config_handler, TSMutexCreate()); + TSContDataSet(config_cont, static_cast(pstate)); + + // constantly run this for rule grooming + TSContScheduleOnPool(config_cont, CONFIG_TMOUT, TS_THREAD_POOL_TASK); + + // also register this continuation with any config reloads + TSMgmtUpdateRegister(config_cont, PLUGIN_NAME); + + // and also register as a lifecycle hook + TSLifecycleHookAdd(TS_LIFECYCLE_MSG_HOOK, config_cont); + } +} + +} // namespace + +void +TSPluginInit(int argc, char const *argv[]) +{ + DEBUG_LOG("Starting plugin init"); + + TSPluginRegistrationInfo info; + info.plugin_name = PLUGIN_NAME; + info.vendor_name = "Apache Software Foundation"; + info.support_email = "dev@trafficserver.apache.org"; + + if (TSPluginRegister(&info) != TS_SUCCESS) { + ERROR_LOG("Global plugin registration failed"); + return; + } else { + DEBUG_LOG("Global plugin registration succeeded"); + } + + if (TS_VERSION_MAJOR != TSTrafficServerVersionGetMajor()) { + ERROR_LOG("Plugin requires Traffic Server %d", TS_VERSION_MAJOR); + return; + } + + PluginState *const pstate = new PluginState; + if (!pstate->fromArgs(argc, argv)) { + ERROR_LOG("Remap plugin registration failed"); + delete pstate; + return; + } + + setup_config_cont(pstate); + + TSCont const main_cont = TSContCreate(main_handler, nullptr); + TSContDataSet(main_cont, static_cast(pstate)); + TSHttpHookAdd(TS_HTTP_CACHE_LOOKUP_COMPLETE_HOOK, main_cont); + + TSAssert(nullptr != config_cont); + + time_t const timenow = time(nullptr); + + std::vector newrules; + pstate->loadConfig(timenow, &newrules); + pstate->config_file_mtime = timeForFile(pstate->config_file); + + time_t new_expiry = 0; + pruneConfig(timenow, &newrules, &new_expiry); + pstate->min_expiry = new_expiry; + + auto vecnew = std::make_shared>(std::move(newrules)); + pstate->invalidate_vec = vecnew; + + DEBUG_LOG("Configuation installed"); + + pstate->logConfig(); + + DEBUG_LOG("Global Plugin Init Complete"); +} diff --git a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-hit.gold b/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-hit.gold deleted file mode 100644 index 68e9596f565..00000000000 --- a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-hit.gold +++ /dev/null @@ -1,10 +0,0 @@ -HTTP/1.1 200 OK -Etag: `` -Cache-Control: max-age=``,public -Content-Length: 3 -Date: `` -Connection: `` -Server: `` -X-Cache: hit-fresh -`` -`` diff --git a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-miss.gold b/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-miss.gold deleted file mode 100644 index 43a2e4a6e3f..00000000000 --- a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-miss.gold +++ /dev/null @@ -1,10 +0,0 @@ -HTTP/1.1 200 OK -Etag: `` -Cache-Control: max-age=``,public -Content-Length: 3 -Date: `` -Connection: `` -Server: `` -X-Cache: miss -`` -`` diff --git a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-stale.gold b/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-stale.gold deleted file mode 100644 index 40a8044a989..00000000000 --- a/tests/gold_tests/pluginTest/regex_revalidate/gold/regex_reval-stale.gold +++ /dev/null @@ -1,10 +0,0 @@ -HTTP/1.1 200 OK -Etag: `` -Cache-Control: max-age=``,public -Content-Length: 3 -Date: `` -Connection: `` -Server: `` -X-Cache: hit-stale -`` -`` diff --git a/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate.test.py b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate.test.py index 3db076e623c..3c19071363e 100644 --- a/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate.test.py +++ b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate.test.py @@ -19,7 +19,7 @@ import os import time Test.Summary = ''' -Test a basic regex_revalidate +Basic regex_revalidate plugin test ''' # Test description: @@ -42,7 +42,7 @@ server = Test.MakeOriginServer("server") # Define ATS and configure -ts = Test.MakeATSProcess("ts", command="traffic_manager", select_ports=True) +ts = Test.MakeATSProcess("ts", command="traffic_manager") # default root request_header_0 = {"headers": @@ -128,139 +128,184 @@ ) regex_revalidate_conf_path = os.path.join(ts.Variables.CONFIGDIR, 'regex_revalidate.conf') -curl_and_args = 'curl -s -D - -v -H "x-debug: x-cache" -H "Host: www.example.com"' +#curl_and_args = 'curl -s -D - -v -H "x-debug: x-cache" -H "Host: www.example.com"' path1_rule = 'path1 {}\n'.format(int(time.time()) + 600) -# Define first revistion for when trafficserver starts -ts.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ - "# Empty\n" -]) +# Define first revision for when trafficserver starts +ts.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine( + "# Empty" +) ts.Disk.remap_config.AddLine( - 'map / http://127.0.0.1:{}'.format(server.Variables.Port) + 'map http://ats/ http://127.0.0.1:{}'.format(server.Variables.Port) ) # minimal configuration ts.Disk.records_config.update({ 'proxy.config.diags.debug.enabled': 1, 'proxy.config.diags.debug.tags': 'regex_revalidate', - # 'proxy.config.diags.debug.enabled': 0, 'proxy.config.http.insert_age_in_response': 0, 'proxy.config.http.response_via_str': 3, + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, }) +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x http://127.0.0.1:{}'.format(ts.Variables.port) + ' -H "x-debug: x-cache"' + # 0 Test - Load cache (miss) (path1) tr = Test.AddTestRun("Cache miss path1") -tr.Processes.Default.StartBefore(server) -tr.Processes.Default.StartBefore(Test.Processes.ts) -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-miss.gold" +ps = tr.Processes.Default +ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) +ps.StartBefore(Test.Processes.ts) +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") tr.StillRunningAfter = ts # 1 Test - Load cache (miss) for later test (path1a) tr = Test.AddTestRun("Cache miss path1a") -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1a'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-miss.gold" +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1a' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") tr.StillRunningAfter = ts # 2 Test - Load cache (miss) for later test (path2a) tr = Test.AddTestRun("Cache miss path2a") -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path2a'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-miss.gold" +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path2a' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") tr.StillRunningAfter = ts # 3 Test - Cache hit path1 tr = Test.AddTestRun("Cache hit fresh path1") -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-hit.gold" +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") tr.StillRunningAfter = ts # 4 Stage - Reload new regex_revalidate tr = Test.AddTestRun("Reload config add path1") -tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ - path1_rule -]) +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule) +tr.Disk.File(regex_revalidate_conf_path + "_tr4", typename="ats:config").AddLine(path1_rule) tr.StillRunningAfter = ts tr.StillRunningAfter = server -tr.Processes.Default.Command = 'traffic_ctl config reload' +ps.Command = 'traffic_ctl config reload' # Need to copy over the environment so traffic_ctl knows where to find the unix domain socket -tr.Processes.Default.Env = ts.Env -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.TimeOut = 5 +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 tr.TimeOut = 5 # 5 Test - Revalidate path1 tr = Test.AddTestRun("Revalidate stale path1") +ps = tr.Processes.Default tr.DelayStart = 5 -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-stale.gold" +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-stale", "expected cache hit stale response") tr.StillRunningAfter = ts # 6 Test - Cache hit (path1) tr = Test.AddTestRun("Cache hit fresh path1") -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-hit.gold" +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") tr.StillRunningAfter = ts # 7 Stage - Reload new regex_revalidate tr = Test.AddTestRun("Reload config add path2") +ps = tr.Processes.Default tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ path1_rule, 'path2 {}\n'.format(int(time.time()) + 700) ]) +tr.Disk.File(regex_revalidate_conf_path + "_tr7", typename="ats:config").AddLines([ + path1_rule, + 'path2 {}\n'.format(int(time.time()) + 700) +]) tr.StillRunningAfter = ts tr.StillRunningAfter = server -tr.Processes.Default.Command = 'traffic_ctl config reload' -# Need to copy over the environment so traffic_ctl knows where to find the unix domain socket -tr.Processes.Default.Env = ts.Env -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.TimeOut = 5 +ps.Command = 'traffic_ctl config reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 tr.TimeOut = 5 # 8 Test - Cache hit (path1) tr = Test.AddTestRun("Cache hit fresh path1") +ps = tr.Processes.Default tr.DelayStart = 5 -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-hit.gold" +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") tr.StillRunningAfter = ts # 9 Test - Cache stale (check rule is still loaded) (path1a) tr = Test.AddTestRun("Revalidate stale path1a") -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path1a'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-stale.gold" +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1a' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-stale", "expected cache hit stale response") tr.StillRunningAfter = ts -# The C version of regex_revalidate doesn't allow an existing rule to -# be changed by a reload. - # 10 Stage - regex_revalidate rewrite rule early expire tr = Test.AddTestRun("Reload config change path2") +ps = tr.Processes.Default tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ path1_rule, 'path2 {}\n'.format(int(time.time()) - 100), ]) +tr.Disk.File(regex_revalidate_conf_path + "_tr10", typename="ats:config").AddLines([ + path1_rule, + 'path2 {}\n'.format(int(time.time()) - 100), +]) tr.StillRunningAfter = ts tr.StillRunningAfter = server -tr.Processes.Default.Command = 'traffic_ctl config reload' -# Need to copy over the environment so traffic_ctl knows where to find the unix domain socket -tr.Processes.Default.Env = ts.Env -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.TimeOut = 5 +ps.Command = 'traffic_ctl config reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 11 Test - Cache hit fresh (path2a) -- path2 rule expired! +tr = Test.AddTestRun("Cache hit fresh path2a") +ps = tr.Processes.Default +tr.DelayStart = 5 +ps.Command = curl_and_args + ' http://ats/path2a' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") +tr.StillRunningAfter = ts + +# 12 Test - Lifecycle plugin reload +tr = Test.AddTestRun("Reload config reenable path2") +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ + path1_rule, + 'path2 {}\n'.format(int(time.time()) + 500) +]) +tr.Disk.File(regex_revalidate_conf_path + "_tr12", typename="ats:config").AddLines([ + path1_rule, + 'path2 {}\n'.format(int(time.time()) + 500) +]) +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 tr.TimeOut = 5 -# 11 Test - Cache hit (path2a) +# 13 Test - Cache hit stale (path2a) -- path2 rule re-instated tr = Test.AddTestRun("Cache hit stale path2a") +ps = tr.Processes.Default tr.DelayStart = 5 -tr.Processes.Default.Command = curl_and_args + ' http://127.0.0.1:{}/path2a'.format(ts.Variables.port) -tr.Processes.Default.ReturnCode = 0 -tr.Processes.Default.Streams.stdout = "gold/regex_reval-stale.gold" +ps.Command = curl_and_args + ' http://ats/path2a' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-stale", "expected cache hit stale response") tr.StillRunningAfter = ts diff --git a/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_miss.test.py b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_miss.test.py new file mode 100644 index 00000000000..539cf8e8f68 --- /dev/null +++ b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_miss.test.py @@ -0,0 +1,233 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import time +Test.Summary = ''' +regex_revalidate plugin test, MISS (refetch) functionality +''' + +# Test description: +# If MISS tag encountered, should load rule as refetch instead of IMS. +# If rule switched from MISS to IMS or vice versa, rule should reset. + +Test.SkipUnless( + Condition.PluginExists('regex_revalidate.so'), + Condition.PluginExists('xdebug.so') +) +Test.ContinueOnFail = False + +# configure origin server +server = Test.MakeOriginServer("server") + +# Define ATS and configure +ts = Test.MakeATSProcess("ts", command="traffic_manager") + +# **testname is required** +#testName = "regex_reval" + +# default root +request_header_0 = {"headers": + "GET / HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "", + } + +response_header_0 = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + "Cache-Control: max-age=300\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "xxx", + } + +# cache item path1 +request_header_1 = {"headers": + "GET /path1 HTTP/1.1\r\n" + + "Host: www.example.com\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "" + } +response_header_1 = {"headers": + "HTTP/1.1 200 OK\r\n" + + "Connection: close\r\n" + + 'Etag: "path1"\r\n' + + "Cache-Control: max-age=600,public\r\n" + + "\r\n", + "timestamp": "1469733493.993", + "body": "abc" + } + + +server.addResponse("sessionlog.json", request_header_0, response_header_0) +server.addResponse("sessionlog.json", request_header_1, response_header_1) + +# Configure ATS server +ts.Disk.plugin_config.AddLine('xdebug.so') +ts.Disk.plugin_config.AddLine( + 'regex_revalidate.so -d -c regex_revalidate.conf' +) + +regex_revalidate_conf_path = os.path.join(ts.Variables.CONFIGDIR, 'regex_revalidate.conf') +#curl_and_args = 'curl -s -D - -v -H "x-debug: x-cache" -H "Host: www.example.com"' + +path1_rule = 'path1 {}'.format(int(time.time()) + 600) + +# Define first revision for when trafficserver starts +ts.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine( + "# Empty" +) + +ts.Disk.remap_config.AddLine( + 'map http://ats/ http://127.0.0.1:{}'.format(server.Variables.Port) +) + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'regex_revalidate', + 'proxy.config.http.insert_age_in_response': 0, + 'proxy.config.http.response_via_str': 3, + 'proxy.config.http.cache.http': 1, + 'proxy.config.http.wait_for_cache': 1, +}) + +curl_and_args = 'curl -s -D /dev/stdout -o /dev/stderr -x http://127.0.0.1:{}'.format(ts.Variables.port) + ' -H "x-debug: x-cache"' + +# 0 Test - Load cache (miss) (path1) +tr = Test.AddTestRun("Cache miss path1") +ps = tr.Processes.Default +ps.StartBefore(server, ready=When.PortOpen(server.Variables.Port)) +ps.StartBefore(Test.Processes.ts) +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") +tr.StillRunningAfter = ts + +# 1 Test - Cache hit path1 +tr = Test.AddTestRun("Cache hit fresh path1") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") +tr.StillRunningAfter = ts + +# 2 Stage - Load new regex_revalidate +tr = Test.AddTestRun("Reload config add path1") +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule + ' MISS') +tr.Disk.File(regex_revalidate_conf_path + "_tr2", typename="ats:config").AddLine(path1_rule + ' MISS') +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +#ps.Command = 'traffic_ctl config reload' +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +# Need to copy over the environment so traffic_ctl knows where to find the unix domain socket +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 3 Test - Revalidate path1 +tr = Test.AddTestRun("Revalidate MISS path1") +ps = tr.Processes.Default +tr.DelayStart = 5 +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") +tr.StillRunningAfter = ts + +# 4 Test - Cache hit (path1) +tr = Test.AddTestRun("Cache hit fresh path1") +ps = tr.Processes.Default +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit fresh response") +tr.StillRunningAfter = ts + +# 5 Stage - Change from MISS to STALE, reload +tr = Test.AddTestRun("Reload config path1 STLE") +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule + ' STALE') +tr.Disk.File(regex_revalidate_conf_path + "_tr5", typename="ats:config").AddLine(path1_rule + ' STALE') +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +#ps.Command = 'traffic_ctl config reload' +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 6 Test - Cache stale +tr = Test.AddTestRun("Cache stale path1") +ps = tr.Processes.Default +tr.DelayStart = 5 +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-stale", "expected cache hit stale response") +tr.StillRunningAfter = ts + +# 7 Stage - Switch back to MISS +tr = Test.AddTestRun("Reload config path1 MISS") +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule + ' MISS') +tr.Disk.File(regex_revalidate_conf_path + "_tr7", typename="ats:config").AddLine(path1_rule + ' MISS') +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +#ps.Command = 'traffic_ctl config reload' +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 8 Test - Cache stale +tr = Test.AddTestRun("Cache stale path1") +ps = tr.Processes.Default +tr.DelayStart = 5 +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: miss", "expected cache miss response") +tr.StillRunningAfter = ts + +# 9 Stage - Write out same contents, ensure rule not reset +tr = Test.AddTestRun("Reload config path1 MISS again") +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule + ' MISS') +tr.Disk.File(regex_revalidate_conf_path + "_tr9", typename="ats:config").AddLine(path1_rule + ' MISSSTALE') +tr.StillRunningAfter = ts +tr.StillRunningAfter = server +#ps.Command = 'traffic_ctl config reload' +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 8 Test - Cache stale +tr = Test.AddTestRun("Cache stale path1") +ps = tr.Processes.Default +tr.DelayStart = 5 +ps.Command = curl_and_args + ' http://ats/path1' +ps.ReturnCode = 0 +ps.Streams.stdout.Content = Testers.ContainsExpression("X-Cache: hit-fresh", "expected cache hit response") +tr.StillRunningAfter = ts diff --git a/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_reload.test.py b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_reload.test.py new file mode 100644 index 00000000000..4e7e843dc88 --- /dev/null +++ b/tests/gold_tests/pluginTest/regex_revalidate/regex_revalidate_reload.test.py @@ -0,0 +1,179 @@ +''' +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import time +Test.Summary = ''' +regex_revalidate plugin config file load test +''' + +# Test description: +# Load up cache, ensure fresh +# Create regex reval rule, config reload: +# ensure item is staled only once. +# Add a new rule, config reload: +# ensure item isn't restaled again, but rule still in effect. +# +# If the rule disappears from regex_revalidate.conf its still loaded!! +# A rule's expiry can't be changed after the fact! + +Test.SkipUnless( + Condition.PluginExists('regex_revalidate.so'), +) +Test.ContinueOnFail = False + +# Define ATS and configure +ts = Test.MakeATSProcess("ts", command="traffic_manager") + +ts.Disk.plugin_config.AddLine( + 'regex_revalidate.so -d -c regex_revalidate.conf' +) + +# **testname is required** +#testName = "regex_reval" + +ts.Disk.remap_config.AddLine( + 'map http://ats/ http://localhost/' +) + +# minimal configuration +ts.Disk.records_config.update({ + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'regex_revalidate', +}) + +# create regex revalidate rule to load +regex_revalidate_conf_path = os.path.join(ts.Variables.CONFIGDIR, 'regex_revalidate.conf') + +path1_expiry = int(time.time()) + 600 +path1_rule = 'path1 {}\n'.format(path1_expiry) + +# Define first revision for when trafficserver starts +ts.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ + path1_rule, +]) + +# 0 - unnecessary reload +tr = Test.AddTestRun("Reload unchanged file") +ps = tr.Processes.Default +ps.StartBefore(Test.Processes.ts) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 0 - load duplicate +tr = Test.AddTestRun("Reload duplicate config, new mtime") +tr.DelayStart = 2 +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 1 - change expiry and reload +path1_expiry_new = path1_expiry + 50 +path1_rule_new = 'path1 {}\n'.format(path1_expiry_new) + +tr = Test.AddTestRun("Reload config, new expiry") +tr.DelayStart = 1 +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path1_rule_new) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 2 - old rule first, new rule last +tr = Test.AddTestRun("Reload config, no change") +tr.DelayStart = 1 +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ + path1_rule, + path1_rule_new, +]) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 3 - new rule first, old rule last +tr = Test.AddTestRun("Reload config, change to original") +tr.DelayStart = 1 +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLines([ + path1_rule_new, + path1_rule, +]) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +path2_expiry = int(time.time()) + 200 +path2_rule = 'path2 {}\n'.format(path2_expiry) + +# 4 - different rule, old rule should still be in effect +tr = Test.AddTestRun("Reload config, just different rule") +tr.DelayStart = 1 +ps = tr.Processes.Default +tr.Disk.File(regex_revalidate_conf_path, typename="ats:config").AddLine(path2_rule) +tr.StillRunningAfter = ts +ps.Command = 'traffic_ctl plugin msg regex_revalidate config_reload' +ps.Env = ts.Env +ps.ReturnCode = 0 +ps.TimeOut = 5 +tr.TimeOut = 5 + +# 5 - stall so that logs can flush +tr = test.AddTestRun("Flush ats logs") +ps = tr.Processes.Default +ps.Command = "sleep 3" + +expected_answers = ["false", "true", "false", "true", "true"] + + +def check_ats_logs(event, tester): + with open(tester.GetContent(event)) as file: + lines = file.readlines() + newlines = [] + for line in lines: + if "Rules have been changed:" in line: + newlines.append(line.strip()) + if len(newlines) != len(expected_answers): + res = 'expected {} lines, got {}'.format(len(expected), len(newlines)) + return (False, "config_reload lines", res) + + for ans, line in zip(expected_answers, newlines): + if ans not in line: + return (False, "config_reload lines", "wrong answer line {}".format(ind)) + return (True, "config_reload lines", "saw expected answers") + + +ts.Streams.stderr = Testers.Lambda(check_ats_logs)