* Adds a new peering ring mode to next hop selection strategies. (apa…

…che#7897)

* Adds a new configuration parameter to the next hop strategies.yaml.
  The new config, cache_peer_result is enabled by default.  When the
  setting is set to false, responses from peer hosts will not be cached
  locally.  This config is used only for next hop strategies using a
  consistent_hash policy and a peering ring mode.

configs/strategies.yaml.default

@@ -95,15 +95,24 @@
 #     hash_key: hostname # optional key to use for Hashing. Enum of 'url' or 'uri' or 'hostname' or 'path' or 'path+query' or 'cache_key' or 'path+fragment'
 #     go_direct: true # transactions may routed directly to the origin true/false default is true.
 #     parent_is_proxy: false  # next hop hosts  are origin servers when set to 'false', defaults to true and indicates next hop hosts are ats cache's.
+#     cache_peer_result: true # only used when the 'ring_mode' is set to 'peering_ring' and the policy is 'consistent_hash'.  The default value is
+#                        true which means to always allow caching responses.  When set to 'false' reponses received from peer hosts will not be
+#                        cached, only responses received from upstream parents or origins will be cached.
 #     groups: # groups of hosts, these groups are used as rings in consistent hash and arrays of host groups for round_robin.
 #       - *g1
 #       - *g2
 #     scheme: http
 #     failover:
 #       max_simple_retries: 2 # default is 1, indicates the maximum number of simple retries for the listed response codes.
-#       ring_mode: exhaust_ring # enumerated as exhaust_ring or alternate_ring
+#       ring_mode: exhaust_ring # enumerated as exhaust_ring, alternate_ring, or peering_ring
 #           #1) in 'exhaust_ring' mode all the servers in a ring are exhausted before failing over to secondary ring
 #           #2) in 'alternate_ring' mode causes the failover to another server in secondary ring.
+#           #3) 'peering_ring' is implemented for only a policy of 'consistent_hash' and requires that the strategy
+#               has two host groups defined.  The first group is the 'peer' group of caches that also includes this host
+#               itself.  The second group is the 'upstream' group of caches.  All parent host lookups are looked up from
+#               the peer group using consistent hashing.  If the resolved parent is "this" host, a new parent from the
+#               upstream list using consistent hashing will be chosen instead.  If any of the peer caches are
+#               unreachable or timeout, a new parent is chosen from the upstream list for retries.
 #       response_codes: # defines the responses codes for failover in exhaust_ring mode
 #         - 404
 #         - 502

doc/admin-guide/files/strategies.yaml.en.rst

@@ -180,15 +180,17 @@ Each **strategy** in the list may using the following parameters::
 
 - **go_direct** - A boolean value indicating whether a transaction may bypass proxies and go direct to the origin. Defaults to **true**
 - **parent_is_proxy**: A boolean value which indicates if the groups of hosts are proxy caches or origins.  **true** (default) means all the hosts used in the remap are |TS| caches.  **false** means the hosts are origins that the next hop strategies may use for load balancing and/or failover.
+- **cache_peer_result** - A boolean value that is only used when the **policy** is 'consistent_hash' and a **peering_ring** mode is used for the strategy. When set to true, the default, all responses from upstream and peer endpoints are allowed to be cached.  Setting this to false will disable caching responses received from a peer host. Only responses from upstream origins or parents will be cached for this strategy.
 - **scheme** Indicates which scheme the strategy supports, *http* or *https*
   - **failover**: A map of **failover** information.
   - **max_simple_retries**: Part of the **failover** map and is an integer value of the maximum number of retries for a **simple retry** on the list of indicated response codes.  **simple retry** is used to retry an upstream request using another upstream server if the response received on from the original upstream request matches any of the response codes configured for this strategy in the **failover** map.  If no failover response codes are configured, no **simple retry** is attempted.
   - **max_unavailable_retries Part of the **failover** map and is an integer value of the maximum number of retries for a **unavailable retry** on the list of indicated markdown response codes.  **unavailable retry** is used to retry an upstream request using another upstream server if the response received on from the original upstream request matches any of the markdown response codes configured for this strategy in the **failover** map.  If no failover markdown response codes are configured, no **unavailable retry** is attempted.  **unavailable retry** differs from **simple retry** in that if a failover for retry is done, the previously retried server is marked down for rety.
 
-  - **ring_mode**: Part of the **failover** map. The host ring selection mode.  Use either **exhaust_ring** or **alternate_ring**
+  - **ring_mode**: Part of the **failover** map. The host ring selection mode.  Use either **exhaust_ring**,  **alternate_ring** or **peering_ring**
 
    #. **exhaust_ring**: when a host normally selected by the policy fails, another host is selected from the same group.  A new group is not selected until all hosts on the previous group have been exhausted
    #. **alternate_ring**: retry hosts are selected from groups in an alternating group fashion.
+   #. **peering_ring** This mode is only implemented for a policy of **consistent_hash** and requires that two host groups are defined. The first host group is a list of peer caches and "this" host itself, the second group is a list of upstream caches.  Parents are always selected from the peer list however, if the selected parent is "this" host itself a new parent from the upstream list is chosen.  In addition, if any peer host is unreachable or times out, a host from the upstream list is chosen for retries.
 
   - **response_codes**: Part of the **failover** map.  This is a list of **http** response codes that may be used for **simple retry**.
   - **markdown_codes**: Part of the **failover** map.  This is a list of **http** response codes that may be used for **unavailable retry** which will cause a parent markdown.

proxy/ParentSelection.h

@@ -189,15 +189,17 @@ struct ParentResult {
   bool retry;
   bool chash_init[MAX_GROUP_RINGS] = {false};
   TSHostStatus first_choice_status = TSHostStatus::TS_HOST_STATUS_INIT;
+  bool do_not_cache_response       = false;
 
   void
   reset()
   {
     ink_zero(*this);
-    line_number   = -1;
-    result        = PARENT_UNDEFINED;
-    mapWrapped[0] = false;
-    mapWrapped[1] = false;
+    line_number           = -1;
+    result                = PARENT_UNDEFINED;
+    mapWrapped[0]         = false;
+    mapWrapped[1]         = false;
+    do_not_cache_response = false;
   }
 
   bool

proxy/http/HttpTransact.cc

@@ -3637,6 +3637,12 @@ HttpTransact::handle_response_from_parent(State *s)
     if (s->parent_result.retry) {
       markParentUp(s);
     }
+    // the next hop strategy is configured not
+    // to cache a response from a next hop peer.
+    if (s->parent_result.do_not_cache_response) {
+      TxnDebug("http_trans", "response is from a next hop peer, do not cache.");
+      s->cache_info.action = CACHE_DO_NO_ACTION;
+    }
     handle_forward_server_connection_open(s);
     break;
   case PARENT_RETRY:

proxy/http/remap/NextHopConsistentHash.cc

@@ -25,6 +25,7 @@
 
 #include "tscore/HashSip.h"
 #include "HttpSM.h"
+#include "I_Machine.h"
 #include "NextHopConsistentHash.h"
 
 // hash_key strings.
@@ -112,7 +113,7 @@ NextHopConsistentHash::Init(const YAML::Node &n)
       p->group_index = host_groups[i][j]->group_index;
       p->host_index  = host_groups[i][j]->host_index;
       hash_ring->insert(p, p->weight, &hash);
-      NH_Debug(NH_DEBUG_TAG, "Loading hash rings - ring: %d, host record: %d, name: %s, hostname: %s, stategy: %s", i, j, p->name,
+      NH_Debug(NH_DEBUG_TAG, "Loading hash rings - ring: %d, host record: %d, name: %s, hostname: %s, strategy: %s", i, j, p->name,
                p->hostname.c_str(), strategy_name.c_str());
     }
     hash.clear();
@@ -229,6 +230,8 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
   HostStatus &pStatus              = HostStatus::instance();
   TSHostStatus host_stat           = TSHostStatus::TS_HOST_STATUS_INIT;
   HostStatRec *hst                 = nullptr;
+  Machine *machine                 = Machine::instance();
+  ;
 
   if (result->line_number == -1 && result->result == PARENT_UNDEFINED) {
     firstcall = true;
@@ -254,6 +257,12 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
         cur_ring = result->last_group;
       }
       break;
+    case NH_PEERING_RING:
+      ink_assert(groups == 2);
+      // look for the next parent on the
+      // upstream ring.
+      result->last_group = cur_ring = 1;
+      break;
     case NH_EXHAUST_RING:
     default:
       if (!wrapped) {
@@ -280,6 +289,13 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
       if (firstcall) {
         hst                         = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
         result->first_choice_status = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
+        // if peering and the selected host is myself, change rings and search for an upstream
+        // parent.
+        if (ring_mode == NH_PEERING_RING && machine->is_self(pRec->hostname.c_str())) {
+          // switch to the upstream ring.
+          cur_ring = 1;
+          continue;
+        }
         break;
       }
     } else {
@@ -399,6 +415,13 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
       break;
     }
     result->retry = nextHopRetry;
+    // if using a peering ring mode and the parent selected came from the 'peering' group,
+    // cur_ring == 0, then if the config allows it, set the flag to not cache the result.
+    if (ring_mode == NH_PEERING_RING && !cache_peer_result && cur_ring == 0) {
+      result->do_not_cache_response = true;
+      NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] setting do not cache response from a peer per config: %s", sm_id,
+               (result->do_not_cache_response) ? "true" : "false");
+    }
     ink_assert(result->hostname != nullptr);
     ink_assert(result->port != 0);
     NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] result->result: %s Chosen parent: %s.%d", sm_id, ParentResultStr[result->result],

proxy/http/remap/NextHopSelectionStrategy.cc

@@ -29,6 +29,7 @@
 // ring mode strings
 constexpr std::string_view alternate_rings = "alternate_ring";
 constexpr std::string_view exhaust_rings   = "exhaust_ring";
+constexpr std::string_view peering_rings   = "peering_ring";
 
 // health check strings
 constexpr std::string_view active_health_check  = "active";
@@ -89,6 +90,10 @@ NextHopSelectionStrategy::Init(const YAML::Node &n)
       ignore_self_detect = n["ignore_self_detect"].as<bool>();
     }
 
+    if (n["cache_peer_result"]) {
+      cache_peer_result = n["cache_peer_result"].as<bool>();
+    }
+
     // failover node.
     YAML::Node failover_node;
     if (n["failover"]) {
@@ -99,6 +104,8 @@ NextHopSelectionStrategy::Init(const YAML::Node &n)
           ring_mode = NH_ALTERNATE_RING;
         } else if (ring_mode_val == exhaust_rings) {
           ring_mode = NH_EXHAUST_RING;
+        } else if (ring_mode_val == peering_rings) {
+          ring_mode = NH_PEERING_RING;
         } else {
           ring_mode = NH_ALTERNATE_RING;
           NH_Note("Invalid 'ring_mode' value, '%s', for the strategy named '%s', using default '%s'.", ring_mode_val.c_str(),
@@ -221,6 +228,17 @@ NextHopSelectionStrategy::Init(const YAML::Node &n)
     return false;
   }
 
+  if (ring_mode == NH_PEERING_RING) {
+    if (groups != 2) {
+      NH_Error("ring mode is '%s', requires two host groups (peering group and an upstream group).", peering_rings.data());
+      return false;
+    }
+    if (policy_type != NH_CONSISTENT_HASH) {
+      NH_Error("ring mode '%s', is only implemented for a 'consistent_hash' policy.", peering_rings.data());
+      return false;
+    }
+  }
+
   return true;
 }
 

proxy/http/remap/NextHopSelectionStrategy.h

@@ -64,7 +64,7 @@ enum NHPolicyType {
 
 enum NHSchemeType { NH_SCHEME_NONE = 0, NH_SCHEME_HTTP, NH_SCHEME_HTTPS };
 
-enum NHRingMode { NH_ALTERNATE_RING = 0, NH_EXHAUST_RING };
+enum NHRingMode { NH_ALTERNATE_RING = 0, NH_EXHAUST_RING, NH_PEERING_RING };
 
 enum NH_HHealthCheck { NH_ACTIVE, NH_PASSIVE };
 
@@ -247,6 +247,7 @@ class NextHopSelectionStrategy
   bool go_direct           = true;
   bool parent_is_proxy     = true;
   bool ignore_self_detect  = false;
+  bool cache_peer_result   = true;
   NHPolicyType policy_type = NH_UNDEFINED;
   NHSchemeType scheme      = NH_SCHEME_NONE;
   NHRingMode ring_mode     = NH_ALTERNATE_RING;

proxy/http/remap/unit-tests/peering.yaml

@@ -0,0 +1,128 @@
+# @file
+#
+#  Unit test data strategy.yaml file for testing the NextHopStrategyFactory
+#
+#  @section license License
+#
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+#  @section details Details
+#
+#
+# unit test 'peering' with combined hosts and strategies, peering.yaml example
+#
+hosts:
+  - &p1 # shorthand name of host object, with an "anchor name"
+    host: p1.bar.com # name or IP of host
+    hash_string: slsklslsk # optional hash string that replaces the hostname in consistent hashing.
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.1.1:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.1.1:443
+  - &p2
+    host: p2.bar.com
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.1.2:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.1.2:443
+  - &p3
+    host: p3.bar.com
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.1.3:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.1.3:443
+  - &p4
+    host: p4.bar.com
+    protocol:
+      - scheme: http
+        port: 8080
+        health_check_url: http://192.168.1.4:8080
+      - scheme: https
+        port: 8443
+        health_check_url: https://192.168.1.5:8443
+  - &m1
+    host: m1.bar.com
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.2.1:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.2.1:443
+  - &m2
+    host: m2.bar.com
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.2.2:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.2.2:443
+  - &m3
+    host: m3.bar.com
+    protocol:
+      - scheme: http
+        port: 80
+        health_check_url: http://192.168.2.3:80
+      - scheme: https
+        port: 443
+        health_check_url: https://192.168.2.3:443
+groups:
+  - &g1
+    - <<: *p1
+      weight: 0.25
+    - <<: *p2
+      weight: 0.25
+    - <<: *p2
+      weight: 0.25
+    - <<: *p3
+      weight: 0.25
+  - &g2
+    - <<: *m1
+      weight: 0.33
+    - <<: *m2
+      weight: 0.33
+    - <<: *m3
+      weight: 0.33
+strategies:
+  - strategy: "peering-group-1"
+    policy: consistent_hash
+    go_direct: false
+    parent_is_proxy: true
+    cache_peer_result: false
+    groups:
+      - *g1
+      - *g2
+    scheme: http
+    failover:
+      max_simple_retries: 2
+      ring_mode:
+        peering_ring
+      response_codes: # defines the responses codes for failover in exhaust_ring mode
+        - 404
+      health_check: # specifies the list of healthchecks that should be considered for failover. A list of enums: 'passive' or 'active'
+        - passive
+        - active

proxy/http/remap/unit-tests/test_NextHopConsistentHash.cc

@@ -620,3 +620,33 @@ SCENARIO("Testing NextHopConsistentHash class (alternating rings), using policy
     }
   }
 }
+
+// jjr
+//
+SCENARIO("Testing NextHopConsistentHash using a peering ring_mode.")
+{
+  // We need this to build a HdrHeap object in build_request();
+  // No thread setup, forbid use of thread local allocators.
+  cmd_disable_pfreelist = true;
+  // Get all of the HTTP WKS items populated.
+  http_init();
+
+  GIVEN("Loading the consistent-hash-tests.yaml config for 'consistent_hash' tests.")
+  {
+    std::shared_ptr<NextHopSelectionStrategy> strategy;
+    NextHopStrategyFactory nhf(TS_SRC_DIR "unit-tests/peering.yaml");
+    strategy = nhf.strategyInstance("peering-group-1");
+
+    WHEN("the config is loaded.")
+    {
+      THEN("then testing consistent hash.")
+      {
+        REQUIRE(nhf.strategies_loaded == true);
+        REQUIRE(strategy != nullptr);
+        REQUIRE(strategy->groups == 2);
+        REQUIRE(strategy->ring_mode == NH_PEERING_RING);
+        REQUIRE(strategy->policy_type == NH_CONSISTENT_HASH);
+      }
+    }
+  }
+}