diff --git a/doc/admin-guide/plugins/rate_limit.en.rst b/doc/admin-guide/plugins/rate_limit.en.rst index ef3f3246f7a..5d51ac895c5 100644 --- a/doc/admin-guide/plugins/rate_limit.en.rst +++ b/doc/admin-guide/plugins/rate_limit.en.rst @@ -30,6 +30,17 @@ The limit counters and queues are per remap rule only, i.e. there is (currently) no way to group transaction limits from different remap rules into a single rate limiter. +.. Note:: + This is still work in progress, in particularly the configuration and + the IP reputation system needs some work. In particular: + + * We need a proper YAML configuration overall, allowing us to configure + better per service controls as well as sharing resources between remap + rules or SNI. + * We need reloadable configurations. + * The IP reputation currently only works with the global plugin settings. + * There is no support for adding allow listed IPs to the IP reputation. + Remap Plugin ------------ @@ -96,7 +107,10 @@ Global Plugin ------------- As a global plugin, the rate limiting currently applies only for TLS enabled -connections, based on the SNI from the TLS handshake. The basic use is as:: +connections, based on the SNI from the TLS handshake. As a global plugin we +also have the support of an IP reputation system, see below for configurations. + +The basic use is as:: rate_limit.so SNI=www1.example.com,www2.example.com --limit=2 --queue=2 --maxage=10000 @@ -144,6 +158,37 @@ The following options are available: the plugin will use the FQDN of the SNI associated with each rate limiter instance created during plugin initialization. +.. option:: --iprep_buckets + The number of LRU buckets to use for the IP reputation. A good number here + is 10, but can be configured. The reason for the different buckets is to + account for a pseudo-sorted list of IPs on the frequency seen. Too few buckets + will not be enough to keep such a sorting, rendering the algorithm useless. To + function in our setup, the number of buckets must be less than ``100``. + +.. option:: --iprep_bucketsize + This is the size of the largest LRU bucket (the `entry bucket`), `15` is a good + value. This is a power of 2, so `15` means the largest LRU can hold `32768` entries. + Note that this option must be bigger then the `--iprep_buckets` setting, for the + bucket halfing to function. + +.. option:: --iprep_maxage + This is used for aging out entries out of the LRU, the default is `0` which means + no aging happens. Even with no aging, entries will eventually fall out of buckets + because of the LRU mechanism that kicks in. The aging is here to make sure a spike + in traffic from an IP doesn't keep the entry for too long in the LRUs. + +.. option:: --iprep_permablock_limit + The minimum number of hits an IP must reach to get moved to the permanent bucket. + In this bucket, entries will stay for 2x + +.. option:: --iprep_permablock_pressure + This option specifies from which bucket an IP is allowed to move from into the + perma block bucket. A good value here is likely `0` or `1`, which is very conservative. + +.. option:: --iprep_permablock_maxage + Similar to `--iprep_maxage` above, but only applies to the long term (`perma-block`) + bucket. Default is `0`, which means no aging to this bucket is applied. + Metrics ------- Metric names are generated either using defaults or user-supplied values. In either @@ -189,6 +234,21 @@ A user can specify their own prefixes and tags, but not types or metrics. ``resumed`` Queued connection is resumed. ============== =================================================================== +IP Reputation +------------- + +The goal of the IP reputation system is to simply try to identify IPs which are more +likely to be abusive than others. It's not a perfect system, and it relies heavily on +the notion of pressure. The Sieve LRUs are always filled, so you have to make sure that +you only start using them when the system thinks it's under pressure. + +The Sieve LRU is a chained set of (configurable) LRUs, each with smaller and smaller +capacity. This essentially adds a notion of partially sorted elements; All IPs in +LRU generally are more active than the IPs in LRU . LRU is specially marked +for longer term blocking, only the most abusive elements would end up here. + +.. figure:: /static/images/sdk/SieveLRU.png + Examples -------- diff --git a/doc/static/images/sdk/SieveLRU.png b/doc/static/images/sdk/SieveLRU.png new file mode 100644 index 00000000000..3e138e46d21 Binary files /dev/null and b/doc/static/images/sdk/SieveLRU.png differ diff --git a/plugins/experimental/rate_limit/Makefile.inc b/plugins/experimental/rate_limit/Makefile.inc index 72469de5c68..95ab01f43b8 100644 --- a/plugins/experimental/rate_limit/Makefile.inc +++ b/plugins/experimental/rate_limit/Makefile.inc @@ -21,4 +21,5 @@ experimental_rate_limit_rate_limit_la_SOURCES = \ experimental/rate_limit/txn_limiter.cc \ experimental/rate_limit/sni_limiter.cc \ experimental/rate_limit/sni_selector.cc \ + experimental/rate_limit/ip_reputation.cc \ experimental/rate_limit/utilities.cc diff --git a/plugins/experimental/rate_limit/ip_reputation.cc b/plugins/experimental/rate_limit/ip_reputation.cc new file mode 100644 index 00000000000..129cd9584a1 --- /dev/null +++ b/plugins/experimental/rate_limit/ip_reputation.cc @@ -0,0 +1,323 @@ +/** @file + + Implementation details for the IP reputation classes. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include +#include + +#include "ip_reputation.h" + +namespace IpReputation +{ +// These static class members are here to calculate a uint64_t hash of an IP +uint64_t +SieveLru::hasher(const sockaddr *sock) +{ + switch (sock->sa_family) { + case AF_INET: { + const sockaddr_in *sa = reinterpret_cast(sock); + + return (0xffffffff00000000 | sa->sin_addr.s_addr); + } break; + case AF_INET6: { + const sockaddr_in6 *sa6 = reinterpret_cast(sock); + + return (*reinterpret_cast(sa6->sin6_addr.s6_addr) ^ + *reinterpret_cast(sa6->sin6_addr.s6_addr + sizeof(uint64_t))); + } break; + default: + // Clearly shouldn't happen ... + return 0; + break; + } +} + +uint64_t +SieveLru::hasher(const std::string &ip, u_short family) // Mostly a convenience function for testing +{ + switch (family) { + case AF_INET: { + sockaddr_in sa4; + + inet_pton(AF_INET, ip.c_str(), &(sa4.sin_addr)); + sa4.sin_family = AF_INET; + return hasher(reinterpret_cast(&sa4)); + } break; + case AF_INET6: { + sockaddr_in6 sa6; + + inet_pton(AF_INET6, ip.c_str(), &(sa6.sin6_addr)); + sa6.sin6_family = AF_INET6; + return hasher(reinterpret_cast(&sa6)); + } break; + default: + // Really shouldn't happen ... + return 0; + } +} +// Constructor, setting up the pre-sized LRU buckets etc. +SieveLru::SieveLru(uint32_t num_buckets, uint32_t size) : _lock(TSMutexCreate()) +{ + initialize(num_buckets, size); +} + +// Initialize the Sieve LRU object +void +SieveLru::initialize(uint32_t num_buckets, uint32_t size) +{ + TSMutexLock(_lock); + TSAssert(!_initialized); // Don't allow it to be initialized more than once! + TSReleaseAssert(size > num_buckets); // Otherwise we can't half the bucket sizes + + _initialized = true; + _num_buckets = num_buckets; + _size = size; + + uint32_t cur_size = pow(2, 1 + _size - num_buckets); + + _map.reserve(pow(2, size + 2)); // Allow for all the sieve LRUs, and extra room for the allow list + _buckets.reserve(_num_buckets + 2); // Two extra buckets, for the block list and allow list + + // Create the other buckets, in smaller and smaller sizes (power of 2) + for (uint32_t i = lastBucket(); i <= entryBucket(); ++i) { + _buckets[i] = new SieveBucket(cur_size); + cur_size *= 2; + } + + _buckets[blockBucket()] = new SieveBucket(cur_size / 2); // Block LRU, same size as entry bucket + _buckets[allowBucket()] = new SieveBucket(0); // Allow LRU, this is unlimited + TSMutexUnlock(_lock); +} + +// Increment the count for an element (will be created / added if new). +std::tuple +SieveLru::increment(KeyClass key) +{ + TSMutexLock(_lock); + TSAssert(_initialized); + + auto map_it = _map.find(key); + + if (_map.end() == map_it) { + // This is a new entry, this can only be added to the last LRU bucket + SieveBucket *lru = _buckets[entryBucket()]; + + if (lru->full()) { // The LRU is full, replace the last item with a new one + auto last = std::prev(lru->end()); + auto &[l_key, l_count, l_bucket, l_added] = *last; + + lru->moveTop(lru, last); + _map.erase(l_key); + *last = {key, 1, entryBucket(), SystemClock::now()}; + } else { + // Create a new entry, the date is not used now (unless perma blocked), but could be useful for aging out stale elements. + lru->push_front({key, 1, entryBucket(), SystemClock::now()}); + } + _map[key] = lru->begin(); + TSMutexUnlock(_lock); + + return {entryBucket(), 1}; + } else { + auto &[map_key, map_item] = *map_it; + auto &[list_key, count, bucket, added] = *map_item; + auto lru = _buckets[bucket]; + auto max_age = (bucket == blockBucket() ? _perma_max_age : _max_age); + + // Check if the entry is older than max_age (if set), if so just move it to the entry bucket and restart + // Yes, this will move likely abusive IPs but they will earn back a bad reputation; The goal here is to + // not let "spiked" entries sit in small buckets indefinitely. It also cleans up the code. We only check + // the actual system time every 10 request for an IP, if traffic is less frequent than that, the LRU will + // age it out properly. + if ((_max_age > std::chrono::seconds::zero()) && ((count % 10) == 0) && + (std::chrono::duration_cast(SystemClock::now() - added) > max_age)) { + auto last_lru = _buckets[entryBucket()]; + + count >>= 3; // Age the count by a factor of 1/8th + bucket = entryBucket(); + last_lru->moveTop(lru, map_item); + } else { + ++count; + + if (bucket > lastBucket()) { // Not in the smallest bucket, so we may promote + auto p_lru = _buckets[bucket - 1]; // Move to previous bucket + + if (!p_lru->full()) { + p_lru->moveTop(lru, map_item); + --bucket; + } else { + auto p_item = std::prev(p_lru->end()); + auto &[p_key, p_count, p_bucket, p_added] = *p_item; + + if (p_count <= count) { + // Swap places on the two elements, moving both to the top of their respective LRU buckets + p_lru->moveTop(lru, map_item); + lru->moveTop(p_lru, p_item); + --bucket; + ++p_bucket; + } + } + } else { + // Just move it to the top of the current LRU + lru->moveTop(lru, map_item); + } + } + TSMutexUnlock(_lock); + + return {bucket, count}; + } +} + +// Lookup the status of the IP in the current tables, without modifying anything +std::tuple +SieveLru::lookup(KeyClass key) const +{ + TSMutexLock(_lock); + TSAssert(_initialized); + + auto map_it = _map.find(key); + + if (_map.end() == map_it) { + TSMutexUnlock(_lock); + + return {0, entryBucket()}; // Nothing found, return 0 hits and the entry bucket # + } else { + auto &[map_key, map_item] = *map_it; + auto &[list_key, count, bucket, added] = *map_item; + + TSMutexUnlock(_lock); + + return {bucket, count}; + } +} + +// A little helper function, to properly move an IP to one of the two special buckets, +// allow-bucket and block-bucket. +int32_t +SieveLru::move_bucket(KeyClass key, uint32_t to_bucket) +{ + TSMutexLock(_lock); + TSAssert(_initialized); + + auto map_it = _map.find(key); + + if (_map.end() == map_it) { + // This is a new entry, add it directly to the special bucket + SieveBucket *lru = _buckets[to_bucket]; + + if (lru->full()) { // The LRU is full, replace the last item with a new one + auto last = std::prev(lru->end()); + auto &[l_key, l_count, l_bucket, l_added] = *last; + + lru->moveTop(lru, last); + _map.erase(l_key); + *last = {key, 1, to_bucket, SystemClock::now()}; + } else { + // Create a new entry + lru->push_front({key, 1, to_bucket, SystemClock::now()}); + } + _map[key] = lru->begin(); + } else { + auto &[map_key, map_item] = *map_it; + auto &[list_key, count, bucket, added] = *map_item; + auto lru = _buckets[bucket]; + + if (bucket != to_bucket) { // Make sure it's not already blocked + auto move_lru = _buckets[to_bucket]; + + // Free a space for a new entry, if needed + if (move_lru->size() >= move_lru->max_size()) { + auto d_entry = std::prev(move_lru->end()); + auto &[d_key, d_count, d_bucket, d_added] = *d_entry; + + move_lru->erase(d_entry); + _map.erase(d_key); + } + move_lru->moveTop(lru, map_item); // Move the LRU item to the perma-blocks + bucket = to_bucket; + added = SystemClock::now(); + } + } + TSMutexUnlock(_lock); + + return to_bucket; // Just as a convenience, return the destination bucket for this entry +} + +void +SieveLru::dump() +{ + TSMutexLock(_lock); + TSAssert(_initialized); + + for (uint32_t i = 0; i < _num_buckets + 1; ++i) { + long long cnt = 0, sum = 0; + auto lru = _buckets[i]; + + std::cout << std::endl + << "Dumping bucket " << i << " (size=" << lru->size() << ", max_size=" << lru->max_size() << ")" << std::endl; + for (auto &it : *lru) { + auto &[key, count, bucket, added] = it; + + ++cnt; + sum += count; +#if 0 + if (0 == i) { // Also dump the content of the top bucket + std::cout << "\t" << key << "; Count=" << count << ", Bucket=" << bucket << std::endl; + } +#endif + } + + std::cout << "\tAverage count=" << (cnt > 0 ? sum / cnt : 0) << std::endl; + } + TSMutexUnlock(_lock); +} + +// Debugging tools, these memory sizes are best guesses to how much memory the containers will actually use +size_t +SieveBucket::memorySize() const +{ + size_t total = sizeof(SieveBucket); + + total += size() * (2 * sizeof(void *) + sizeof(LruEntry)); // Double linked list + object + + return total; +} + +size_t +SieveLru::memoryUsed() const +{ + TSMutexLock(_lock); + TSAssert(_initialized); + + size_t total = sizeof(SieveLru); + + for (uint32_t i = 0; i <= _num_buckets + 1; ++i) { + total += _buckets[i]->memorySize(); + } + + total += _map.size() * (sizeof(void *) + sizeof(SieveBucket::iterator)); + total += _map.bucket_count() * (sizeof(size_t) + sizeof(void *)); + TSMutexUnlock(_lock); + + return total; +} + +} // namespace IpReputation diff --git a/plugins/experimental/rate_limit/ip_reputation.h b/plugins/experimental/rate_limit/ip_reputation.h new file mode 100644 index 00000000000..4abbbcaa088 --- /dev/null +++ b/plugins/experimental/rate_limit/ip_reputation.h @@ -0,0 +1,236 @@ +/** @file + + Include file for all the IP reputation classes. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ +#pragma once + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ts/ts.h" + +namespace IpReputation +{ +using KeyClass = uint64_t; +using SystemClock = std::chrono::system_clock; + +// Key / Count / bucket # (rank, 0-) / time added +using LruEntry = std::tuple>; + +// This is a wrapper around a std::list which lets us size limit the list to a +// certain size. +class SieveBucket : public std::list +{ +public: + SieveBucket(uint32_t max_size) : _max_size(max_size) {} + + bool + full() const + { + return (_max_size > 0 ? (size() >= _max_size) : false); + } + + size_t + max_size() const + { + return _max_size; + } + + // Move an element to the top of an LRU. This *can* move it from the current LRU (bucket) + // to another, when promoted to a higher rank. + void + moveTop(SieveBucket *source_lru, SieveBucket::iterator &item) + { + splice(begin(), *source_lru, item); + } + + // Debugging tools + size_t memorySize() const; + +private: + uint32_t _max_size; +}; + +using HashMap = std::unordered_map; // The hash map for finding the entry + +// This is a concept / POC: Ranked LRU buckets +// +// Also, obviously the std::string here is not awesome, rather, we ought to save the +// hashed value from the IP as the key (just like the hashed in cache_promote). +class SieveLru +{ +public: + SieveLru() : _lock(TSMutexCreate()){}; // The unitialized version + SieveLru(uint32_t num_buckets, uint32_t size); + ~SieveLru() + { + for (uint32_t i = 0; i <= _num_buckets + 1; ++i) { // Remember to delete the two special allow/block buckets too + delete _buckets[i]; + } + } + + void initialize(uint32_t num_buckets = 10, uint32_t size = 15); + + // Return value is the bucket (0 .. num_buckets) that the IP is in, and the + // current count of "hits". The lookup version is similar, except it doesn't + // modify the status of the IP (read-only). + std::tuple increment(KeyClass key); + + std::tuple + increment(const sockaddr *sock) + { + return increment(hasher(sock)); + } + + // Move an IP to the perm-block or perma-allow LRUs. A zero (default) maxage is indefinite (no timeout). + uint32_t + block(KeyClass key) + { + return move_bucket(key, blockBucket()); + } + + uint32_t + allow(KeyClass key) + { + return move_bucket(key, allowBucket()); + } + + uint32_t + block(const sockaddr *sock) + { + return move_bucket(hasher(sock), blockBucket()); + } + + uint32_t + allow(const sockaddr *sock) + { + return move_bucket(hasher(sock), allowBucket()); + } + + // Lookup the current state of an IP + std::tuple lookup(KeyClass key) const; + + std::tuple + lookup(const sockaddr *sock) const + { + return lookup(hasher(sock)); + } + + // A helper function to hash an INET or INET6 sockaddr to a 64-bit hash. + static uint64_t hasher(const sockaddr *sock); + static uint64_t hasher(const std::string &ip, u_short family = AF_INET); + + // Identifying some of the special buckets: + // + // entryBucket == the highest bucket, where new IPs enter (also the biggest bucket) + // lastBucket == the last bucket, which is most likely to be abusive + // blockBucket == the bucket where we "permanently" block bad IPs + // allowBucket == the bucket where we "permanently" allow good IPs (can not be blocked) + uint32_t + entryBucket() const + { + return _num_buckets; + } + + constexpr uint32_t + lastBucket() const + { + return 1; + } + + constexpr uint32_t + blockBucket() const + { + return 0; + } + + uint32_t + allowBucket() const + { + return _num_buckets + 1; + } + + size_t + bucketSize(uint32_t bucket) const + { + if (bucket <= (_num_buckets + 1)) { + return _buckets[bucket]->size(); + } else { + return 0; + } + } + + bool + initialized() const + { + return _initialized; + } + + // Aging getters and setters + std::chrono::seconds + maxAge() const + { + return _max_age; + } + + std::chrono::seconds + permaMaxAge() const + { + return _perma_max_age; + } + + void + maxAge(std::chrono::seconds maxage) + { + _max_age = maxage; + } + + void + permaMaxAge(std::chrono::seconds maxage) + { + _perma_max_age = maxage; + } + + // Debugging tool, dumps some info around the buckets + void dump(); + size_t memoryUsed() const; + +protected: + int32_t move_bucket(KeyClass key, uint32_t to_bucket); + +private: + HashMap _map; + std::vector _buckets; + uint32_t _num_buckets = 10; // Leave this at 10 ... + uint32_t _size = 0; // Set this up to initialize + std::chrono::seconds _max_age = std::chrono::seconds::zero(); // Aging time in the SieveLru (default off) + std::chrono::seconds _perma_max_age = std::chrono::seconds::zero(); // Aging time in the SieveLru for perma-blocks + bool _initialized = false; // If this has been properly initialized yet + TSMutex _lock; // The lock around all data access +}; + +} // namespace IpReputation diff --git a/plugins/experimental/rate_limit/iprep_simu.cc b/plugins/experimental/rate_limit/iprep_simu.cc new file mode 100644 index 00000000000..887df68b861 --- /dev/null +++ b/plugins/experimental/rate_limit/iprep_simu.cc @@ -0,0 +1,299 @@ + +/** @file + + Simulator application, for testing the behavior of the SieveLRU. This does + not build as part of the system, but put here for future testing etc. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ +#include + +#include +#include +#include +#include +#include +#include + +// Yeh well, sue me, boost is useful here, and this is not part of the actual core code +#include + +#include "ip_reputation.h" + +// Convenience class declarations +using IpMap = std::unordered_map>; // count / false = good, true = bad +using IpList = std::vector; + +// Holds all command line options +struct CmdConfigs { + uint32_t start_buckets, end_buckets, incr_buckets; + uint32_t start_size, end_size, incr_size; + uint32_t start_threshold, end_threshold, incr_threshold; + uint32_t start_permablock, end_permablock, incr_permablock; +}; + +/////////////////////////////////////////////////////////////////////////////// +// Command line options / parsing, returns the parsed and populate CmdConfig +// structure (from above). +// +std::tuple +splitArg(std::string str) +{ + int32_t start = 0, end = 0, incr = 1; + std::vector results; + + boost::split(results, str, [](char c) { return c == '-' || c == '/'; }); + + if (results.size() > 0) { + start = std::stoi(results[0]); + if (results.size() > 1) { + end = std::stoi(results[1]); + if (results.size() > 2) { + incr = std::stoi(results[2]); + } + } else { + end = start; + } + } else { + std::cerr << "Malformed argument: " << str << std::endl; + } + + return {start, end, incr}; +} + +CmdConfigs +parseArgs(int argc, char **argv) +{ + CmdConfigs options; + int c; + constexpr struct option long_options[] = { + {"help", no_argument, NULL, 'h'}, {"buckets", required_argument, NULL, 'b'}, {"perma", required_argument, NULL, 'p'}, + {"size", required_argument, NULL, 's'}, {"threshold", required_argument, NULL, 't'}, {NULL, 0, NULL, 0}}; + + // Make sure the optional values have been set + + options.start_permablock = 0; + options.end_permablock = 0; + options.incr_permablock = 1; + + while (1) { + int ix = 0; + + c = getopt_long(argc, argv, "b:f:p:s:t:h?", long_options, &ix); + if (c == -1) + break; + + switch (c) { + case 'h': + case '?': + std::cerr << "usage: iprep_simu -b|--buckets [-[/]]" << std::endl; + std::cerr << " -s|--size [-[/]]" << std::endl; + std::cerr << " -t|--threshold [-[/]]" << std::endl; + std::cerr << " [-p|--perma [-[/]]]" << std::endl; + std::cerr << " [-h|--help" << std::endl; + exit(0); + break; + case 'b': + std::tie(options.start_buckets, options.end_buckets, options.incr_buckets) = splitArg(optarg); + break; + case 's': + std::tie(options.start_size, options.end_size, options.incr_size) = splitArg(optarg); + break; + case 'p': + std::tie(options.start_permablock, options.end_permablock, options.incr_permablock) = splitArg(optarg); + break; + case 't': + std::tie(options.start_threshold, options.end_threshold, options.incr_threshold) = splitArg(optarg); + break; + default: + fprintf(stderr, "getopt returned weird stuff: 0%o\n", c); + exit(-1); + break; + } + } + + return options; // RVO +} + +/////////////////////////////////////////////////////////////////////////////// +// Load a configuration file, and populate the two structures with the +// list of IPs (and their status) as well as the full sequence of requests. +// +// Returns a tuple with the number of good requests and bad requests, respectively. +// +std::tuple +loadFile(std::string fname, IpMap &all_ips, IpList &ips) +{ + std::ifstream infile(fname); + + float timestamp; // The timestamp from the request(relative) + std::string ip; // The IP + bool status; // Bad (false) or Good (true) request? + + uint32_t good_ips = 0; + uint32_t bad_ips = 0; + uint32_t good_requests = 0; + uint32_t bad_requests = 0; + + // Load in the entire file, and fill the request vector as well as the IP lookup table (state) + while (infile >> timestamp >> ip >> status) { + auto ip_hash = IpReputation::SieveLru::hasher(ip, ip.find(':') != std::string::npos ? AF_INET6 : AF_INET); + auto it = all_ips.find(ip_hash); + + if (!status) { + ++good_requests; + } else { + ++bad_requests; + } + + if (all_ips.end() != it) { + auto &[key, data] = *it; + auto &[count, d_status] = data; + + ++count; + ips.push_back(it); + } else { + all_ips[ip_hash] = {0, status}; + ips.push_back(all_ips.find(ip_hash)); + if (!status) { + ++good_ips; + } else { + ++bad_ips; + } + } + } + + std::cout << std::setprecision(3); + std::cout << "Total number of requests: " << ips.size() << std::endl; + std::cout << "\tGood requests: " << good_requests << " (" << 100.0 * good_requests / ips.size() << "%)" << std::endl; + std::cout << "\tBad requests: " << bad_requests << " (" << 100.0 * bad_requests / ips.size() << "%)" << std::endl; + std::cout << "Unique IPs in set: " << all_ips.size() << std::endl; + std::cout << "\tGood IPs: " << good_ips << " (" << 100.0 * good_ips / all_ips.size() << "%)" << std::endl; + std::cout << "\tBad IPs: " << bad_ips << " (" << 100.0 * bad_ips / all_ips.size() << "%)" << std::endl; + std::cout << std::endl; + + return {good_requests, bad_requests}; +} + +int +main(int argc, char *argv[]) +{ + auto options = parseArgs(argc, argv); + + // All remaining arguments should be files, so lets process them one by one + for (int file_num = optind; file_num < argc; ++file_num) { + IpMap all_ips; + IpList ips; + + // Load the data from file + auto [good_requests, bad_requests] = loadFile(argv[file_num], all_ips, ips); + + // Here starts the actual simulation, loop through variations + for (uint32_t size = options.start_size; size <= options.end_size; size += options.incr_size) { + for (uint32_t buckets = options.start_buckets; buckets <= options.end_buckets; buckets += options.incr_buckets) { + for (uint32_t threshold = options.start_threshold; threshold <= options.end_threshold; + threshold += options.incr_threshold) { + for (uint32_t permablock = options.start_permablock; permablock <= options.end_permablock; + permablock += options.incr_permablock) { + // Setup the buckets and metrics for this loop + IpReputation::SieveLru ipt(buckets, size); + + auto start = std::chrono::system_clock::now(); + + // Some metrics + uint32_t good_blocked = 0; + uint32_t good_allowed = 0; + uint32_t bad_blocked = 0; + uint32_t bad_allowed = 0; + uint32_t good_perm_blocked = 0; + uint32_t bad_perm_blocked = 0; + + for (auto iter : ips) { + auto &[ip, data] = *iter; + auto &[count, status] = data; + auto [bucket, cur_cnt] = ipt.increment(ip); + + // Currently we only allow perma-blocking on items in bucket 1, so check for that first. + if (cur_cnt > permablock && bucket == ipt.lastBucket()) { + bucket = ipt.block(ip); + } + + if (bucket == ipt.blockBucket()) { + if (!status) { + ++good_perm_blocked; + } else { + ++bad_perm_blocked; + } + } else if (bucket <= threshold) { + if (!status) { + ++good_blocked; + } else { + ++bad_blocked; + } + } else { + if (!status) { + ++good_allowed; + } else { + ++bad_allowed; + } + } + } + + auto end = std::chrono::system_clock::now(); + + uint32_t total_blocked = bad_blocked + good_blocked; + uint32_t total_perm_blocked = bad_perm_blocked + good_perm_blocked; + uint32_t total_allowed = bad_allowed + good_allowed; + + // ipt.dump(); + + std::chrono::duration elapsed_seconds = end - start; + + std::cout << "Running with size=" << size << ", buckets=" << buckets << ", threshold=" << threshold + << ", permablock=" << permablock << std::endl; + std::cout << "Processing time: " << elapsed_seconds.count() << std::endl; + std::cout << "Denied requests: " << total_blocked + total_perm_blocked << std::endl; + std::cout << "\tGood requests denied: " << good_blocked + good_perm_blocked << " (" + << 100.0 * (good_blocked + good_perm_blocked) / good_requests << "%)" << std::endl; + std::cout << "\tBad requests denied: " << bad_blocked + bad_perm_blocked << " (" + << 100.0 * (bad_blocked + bad_perm_blocked) / bad_requests << "%)" << std::endl; + std::cout << "Allowed requests: " << total_allowed << std::endl; + std::cout << "\tGood requests allowed: " << good_allowed << " (" << 100.0 * good_allowed / good_requests << "%)" + << std::endl; + std::cout << "\tBad requests allowed: " << bad_allowed << " (" << 100.0 * bad_allowed / bad_requests << "%)" + << std::endl; + if (permablock) { + std::cout << "Permanently blocked IPs: " << ipt.bucketSize(ipt.blockBucket()) << std::endl; + std::cout << "\tGood requests permanently denied: " << good_perm_blocked << " (" + << 100.0 * good_perm_blocked / good_requests << "%)" << std::endl; + std::cout << "\tBad requests permanently denied: " << bad_perm_blocked << " (" + << 100.0 * bad_perm_blocked / bad_requests << "%)" << std::endl; + } + std::cout << "Estimated score (lower is better): " + << 100.0 * ((100.0 * good_blocked / good_requests + 100.0 * bad_allowed / bad_requests) / + (100.0 * good_allowed / good_requests + 100.0 * bad_blocked / bad_requests)) + << std::endl; + std::cout << "Memory used for IP Reputation data: " << ipt.memoryUsed() / (1024.0 * 1024.0) << "MB" << std::endl + << std::endl; + } + } + } + } + } +} diff --git a/plugins/experimental/rate_limit/sni_limiter.cc b/plugins/experimental/rate_limit/sni_limiter.cc index b63c50b1df9..fcc8fb838f4 100644 --- a/plugins/experimental/rate_limit/sni_limiter.cc +++ b/plugins/experimental/rate_limit/sni_limiter.cc @@ -43,18 +43,58 @@ sni_limit_cont(TSCont contp, TSEvent event, void *edata) int len; const char *server_name = TSVConnSslSniGet(vc, &len); std::string_view sni_name(server_name, len); + SniRateLimiter *limiter = selector->find(sni_name); - if (!sni_name.empty()) { // This should likely always succeed, but without it we can't do anything - SniRateLimiter *limiter = selector->find(sni_name); + if (limiter) { + // Check if we have an IP reputation for this SNI, and if we should block + if (limiter->iprep.initialized()) { + const sockaddr *sock = TSNetVConnRemoteAddrGet(vc); + int pressure = limiter->pressure(); + + TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s, pressure=%d", static_cast(sni_name.length()), sni_name.data(), pressure); + + // TSDebug(PLUGIN_NAME, "IP Reputation: pressure is currently %d", pressure); + + if (pressure >= 0) { // When pressure is < 0, we're not yet at a level of pressure to be concerned about + char client_ip[INET6_ADDRSTRLEN] = "[unknown]"; + auto [bucket, cur_cnt] = limiter->iprep.increment(sock); + + // Get the client IP string if debug is enabled + if (TSIsDebugTagSet(PLUGIN_NAME)) { + if (sock->sa_family == AF_INET) { + inet_ntop(AF_INET, &(((struct sockaddr_in *)sock)->sin_addr), client_ip, INET_ADDRSTRLEN); + } else if (sock->sa_family == AF_INET6) { + inet_ntop(AF_INET6, &(((struct sockaddr_in6 *)sock)->sin6_addr), client_ip, INET6_ADDRSTRLEN); + } + } + + if (cur_cnt > limiter->iprep_permablock_count && + bucket <= limiter->iprep_permablock_threshold) { // Mark for long-term blocking + TSDebug(PLUGIN_NAME, "Marking IP=%s for perma-blocking", client_ip); + bucket = limiter->iprep.block(sock); + } + + if (static_cast(pressure) > bucket) { // Remember the perma-block bucket is always 0, and we are >=0 already + // Block this IP from finishing the handshake + TSDebug(PLUGIN_NAME, "Rejecting connection from IP=%s, we're at pressure and IP was chosen to be blocked", client_ip); + TSUserArgSet(vc, gVCIdx, nullptr); + TSVConnReenableEx(vc, TS_EVENT_ERROR); + + return TS_ERROR; + } + } + } else { + TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s, no IP reputation", static_cast(sni_name.length()), sni_name.data()); + } - TSDebug(PLUGIN_NAME, "CLIENT_HELLO on %.*s", static_cast(sni_name.length()), sni_name.data()); - if (limiter && !limiter->reserve()) { + // If we passed the IP reputation filter, continue rate limiting these connections + if (!limiter->reserve()) { if (!limiter->max_queue || limiter->full()) { // We are running at limit, and the queue has reached max capacity, give back an error and be done. - TSVConnReenableEx(vc, TS_EVENT_ERROR); TSDebug(PLUGIN_NAME, "Rejecting connection, we're at capacity and queue is full"); TSUserArgSet(vc, gVCIdx, nullptr); limiter->incrementMetric(RATE_LIMITER_METRIC_REJECTED); + TSVConnReenableEx(vc, TS_EVENT_ERROR); return TS_ERROR; } else { @@ -69,11 +109,11 @@ sni_limit_cont(TSCont contp, TSEvent event, void *edata) TSVConnReenable(vc); } } else { + // No limiter for this SNI at all, clear the args etc. just in case + TSUserArgSet(vc, gVCIdx, nullptr); TSVConnReenable(vc); } - - break; - } + } break; case TS_EVENT_VCONN_CLOSE: { SniRateLimiter *limiter = static_cast(TSUserArgGet(vc, gVCIdx)); @@ -107,10 +147,19 @@ SniRateLimiter::initialize(int argc, const char *argv[]) {const_cast("maxage"), required_argument, nullptr, 'm'}, {const_cast("prefix"), required_argument, nullptr, 'p'}, {const_cast("tag"), required_argument, nullptr, 't'}, + // These are all for the IP reputation system. ToDo: These should be global rather than per SNI ? + {const_cast("iprep_maxage"), required_argument, nullptr, 'a'}, + {const_cast("iprep_buckets"), required_argument, nullptr, 'B'}, + {const_cast("iprep_bucketsize"), required_argument, nullptr, 'S'}, + {const_cast("iprep_permablock_limit"), required_argument, nullptr, 'L'}, + {const_cast("iprep_permablock_pressure"), required_argument, nullptr, 'P'}, + {const_cast("iprep_permablock_maxage"), required_argument, nullptr, 'A'}, // EOF {nullptr, no_argument, nullptr, '\0'}, }; + TSDebug(PLUGIN_NAME, "Initializing an SNI Rate Limiter"); + while (true) { int opt = getopt_long(argc, const_cast(argv), "", longopt, nullptr); @@ -130,11 +179,50 @@ SniRateLimiter::initialize(int argc, const char *argv[]) case 't': this->tag = std::string(optarg); break; + case 'a': + this->_iprep_max_age = std::chrono::seconds(strtol(optarg, nullptr, 10)); + break; + case 'B': + this->_iprep_num_buckets = strtol(optarg, nullptr, 10); + if (this->_iprep_num_buckets >= 100) { + TSError("sni_limiter: iprep_num_buckets must be in the range 1 .. 99, IP reputation disabled"); + this->_iprep_num_buckets = 0; + } + break; + case 'S': + this->_iprep_size = strtol(optarg, nullptr, 10); + break; + case 'L': + this->iprep_permablock_count = strtol(optarg, nullptr, 10); + break; + case 'P': + this->iprep_permablock_threshold = strtol(optarg, nullptr, 10); + break; + case 'A': + this->_iprep_perma_max_age = std::chrono::seconds(strtol(optarg, nullptr, 10)); + break; } if (opt == -1) { break; } } + // Enable and initialize the IP reputation if asked for + if (this->_iprep_num_buckets > 0 && this->_iprep_size > 0) { + TSDebug(PLUGIN_NAME, "Calling and _initialized is %d\n", this->iprep.initialized()); + this->iprep.initialize(this->_iprep_num_buckets, this->_iprep_size); + TSDebug(PLUGIN_NAME, "IP-reputation enabled with %u buckets, max size is 2^%u", this->_iprep_num_buckets, this->_iprep_size); + + TSDebug(PLUGIN_NAME, "Called and _initialized is %d\n", this->iprep.initialized()); + + // These settings are optional + if (this->_iprep_max_age != std::chrono::seconds::zero()) { + this->iprep.maxAge(this->_iprep_max_age); + } + if (this->_iprep_perma_max_age != std::chrono::seconds::zero()) { + this->iprep.permaMaxAge(this->_iprep_perma_max_age); + } + } + return true; } diff --git a/plugins/experimental/rate_limit/sni_limiter.h b/plugins/experimental/rate_limit/sni_limiter.h index 3889a0819f4..93b1b5558d0 100644 --- a/plugins/experimental/rate_limit/sni_limiter.h +++ b/plugins/experimental/rate_limit/sni_limiter.h @@ -18,6 +18,7 @@ #pragma once #include "limiter.h" +#include "ip_reputation.h" #include "ts/ts.h" int sni_limit_cont(TSCont contp, TSEvent event, void *edata); @@ -40,4 +41,25 @@ class SniRateLimiter : public RateLimiter } bool initialize(int argc, const char *argv[]); + + // ToDo: this ought to go into some better global IP reputation pool / settings. Waiting for YAML... + IpReputation::SieveLru iprep; + uint32_t iprep_permablock_count = 0; // "Hits" limit for blocking permanently + uint32_t iprep_permablock_threshold = 0; // Pressure threshold for permanent block + + // Calculate the pressure, which is either a negative number (ignore), or a number 0-. + // 0 == block only perma-blocks. + int32_t + pressure() const + { + return ((active() - 1) / static_cast(limit) * 100) - (99 - _iprep_num_buckets); + } + +private: + // ToDo: These should be moved to global configurations to have one shared IP Reputation. + // today the configuration of this is so klunky, that there is no easy way to make it "global". + std::chrono::seconds _iprep_max_age = std::chrono::seconds::zero(); // Max age in the SieveLRUs for regular buckets + std::chrono::seconds _iprep_perma_max_age = std::chrono::seconds::zero(); // Max age in the SieveLRUs for perma-block buckets + uint32_t _iprep_num_buckets = 10; // Number of buckets. ToDo: leave this at 10 always + uint32_t _iprep_size = 15; // Size of the biggest bucket; 15 == 2^15 == 32768 }; diff --git a/plugins/experimental/rate_limit/sni_selector.cc b/plugins/experimental/rate_limit/sni_selector.cc index d41b4df0690..e4c22d02d9d 100644 --- a/plugins/experimental/rate_limit/sni_selector.cc +++ b/plugins/experimental/rate_limit/sni_selector.cc @@ -86,6 +86,10 @@ SniSelector::insert(std::string_view sni, SniRateLimiter *limiter) SniRateLimiter * SniSelector::find(std::string_view sni) { + if (sni.empty()) { // Likely shouldn't happen, but we can shortcircuit + return nullptr; + } + auto limiter = _limiters.find(sni); if (limiter != _limiters.end()) { @@ -105,17 +109,18 @@ SniSelector::factory(const char *sni_list, int argc, const char *argv[]) char *saveptr; char *sni = strdup(sni_list); // We make a copy of the sni list, to not touch the original string char *token = strtok_r(sni, ",", &saveptr); - SniRateLimiter def_limiter; - - def_limiter.initialize(argc, argv); // Creates the template limiter - _needs_queue_cont = (def_limiter.max_queue > 0); + // Todo: We are repeating initializing here with the same configurations, but once we move this to + // YAML, and refactor this, it'll be better. And this is not particularly expensive. while (nullptr != token) { - SniRateLimiter *limiter = new SniRateLimiter(def_limiter); // Make a shallow copy + SniRateLimiter *limiter = new SniRateLimiter(); TSReleaseAssert(limiter); + limiter->initialize(argc, argv); limiter->description = token; + _needs_queue_cont = (limiter->max_queue > 0); + insert(std::string_view(limiter->description), limiter); token = strtok_r(nullptr, ",", &saveptr); }