Node/CCQ: Add rate limiting to proxy (#4080)

* Node/CCQ: Add rate limiting

* Code review rework

* Node/CCQ: Make burst size default to one not zero

* Tweak description of burst size in doc

devnet/query-server.yaml

@@ -60,7 +60,6 @@ spec:
             - --logLevel=warn
             - --shutdownDelay1
             - "0"
-            - --allowAnything
           ports:
             - containerPort: 6069
               name: rest

docs/query_proxy.md

@@ -62,7 +62,6 @@ Optional Parameters
 
 - The `gossipAdvertiseAddress` argument allows you to specify an external IP to advertize on P2P (use if behind a NAT or running in k8s).
 - The `monitorPeers` flag will cause the proxy server to periodically check its connectivity to the P2P bootstrap peers, and attempt to reconnect if necessary.
-- The `allowAnything` flag enables defining users with the `allowAnything` flag set to true. This is only allowed in testnet and devnet.
 
 #### Creating the Signing Key File
 
@@ -96,6 +95,9 @@ The simplest file would look something like this
 
 ```json
 {
+  "allowAnythingSupported": false,
+  "defaultRateLimit": 0.5,
+  "defaultBurstSize": 1,
   "permissions": [
     {
       "userName": "Monitor",
@@ -162,8 +164,10 @@ as soon as you save the file, the changes will be picked up (whether they are lo
 
 #### The `allowAnything` flag
 
+The `allowAnything` flag may only be specified for a user if you are running in testnet and the `allowAnythingSupported` flag in the
+permissions file is set to true.
+
 If this flag is specified for a user, then that user may make any call on any supported chain, without restriction.
-This flag is only allowed if the `allowAnything` command line argument is specified.
 If this flag is specified, then `allowedCalls` must not be specified.
 
 ```json
@@ -179,6 +183,22 @@ If this flag is specified, then `allowedCalls` must not be specified.
 }
 ```
 
+### Rate Limiting
+
+The query proxy server supports rate limiting by specifying two parameters. The rate limit, which is a floating point value, and the burst size,
+which is an int. See [here](https://pkg.go.dev/golang.org/x/time/rate#Limiter) for a description of how the rate limiter works.
+
+Note that if the rate limits are not specified, or the rate is set to zero, rate limiting will be disabled, allowing unlimited queries per second. The burst size only has meaning if the rate limit is specified. It defaults to one, and zero is not a valid value.
+
+The rate limits may be specified at either of two levels.
+
+First, you may specify global defaults for rate limiting by specifying the `defaultRateLimit` and `defaultBurstSize` parameters
+in the permissions file. If these parameters are specified, they apply to all users for which per-user parameters are not specified.
+This means that each of these users will be allowed that many queries per second.
+
+Second, you may override the global defaults for a given user by specifying `rateLimit` and `burstSize` for that user. Also note that
+you can disable rate limits for a given user (overriding the default) by setting their `rateLimit` to zero.
+
 ### Validating Permissions File Changes
 
 The query server automatically detects changes to the permissions file and attempts to reload them. If there are errors in the updated
@@ -188,12 +208,10 @@ the server from coming up on the next restart. You can avoid this problem by ver
 To do this, you can copy the permissions file to some other file, make your changes to the copy, and then do the following:
 
 ```sh
-$ guardiand query-server --verifyPermissions --permFile new.permissions.file.json --allowAnything
+$ guardiand query-server --env mainnet --verifyPermissions --permFile new.permissions.file.json
 ```
 
-where `new.permissions.file.json` is the path to the updated file. Additionally, if your permission file includes the `allowAnything`
-flag for any of the users, you must specify that flag on the command line when doing the verify.
-
+where the `--env` flag should be either `mainnet` or `testnet` and `new.permissions.file.json` is the path to the updated file.
 If the updated file is good, the program will exit immediately with no output and an exit code of zero. If the file contains
 errors, the first error will be printed, and the exit code will be one.
 

node/cmd/ccq/devnet.permissions.json

@@ -1,4 +1,5 @@
 {
+  "allowAnythingSupported": true,
   "permissions": [
     {
       "userName": "Test User",
@@ -184,6 +185,14 @@
       "apiKey": "my_secret_key_3",
       "allowUnsigned": true,
       "allowAnything": true
+    },
+    {
+      "userName": "Rate Limited User",
+      "apiKey": "rate_limited_key",
+      "rateLimit": 1.0,
+      "burstSize": 2,
+      "allowUnsigned": true,
+      "allowAnything": true
     }
   ]
 }

node/cmd/ccq/http.go

@@ -87,6 +87,14 @@ func (s *httpServer) handleQuery(w http.ResponseWriter, r *http.Request) {
 		invalidQueryRequestReceived.WithLabelValues("invalid_api_key").Inc()
 		return
 	}
+
+	if permEntry.rateLimiter != nil && !permEntry.rateLimiter.Allow() {
+		s.logger.Debug("denying request due to rate limit", zap.String("userId", permEntry.userName))
+		http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
+		rateLimitExceededByUser.WithLabelValues(permEntry.userName).Inc()
+		return
+	}
+
 	totalRequestsByUser.WithLabelValues(permEntry.userName).Inc()
 
 	queryRequestBytes, err := hex.DecodeString(q.Bytes)

node/cmd/ccq/metrics.go

@@ -45,6 +45,12 @@ var (
 			Help: "Total number of successful queries by user name",
 		}, []string{"user_name"})
 
+	rateLimitExceededByUser = promauto.NewCounterVec(
+		prometheus.CounterOpts{
+			Name: "ccq_server_rate_limit_exceeded_by_user",
+			Help: "Total number of queries rejected due to rate limiting per user name",
+		}, []string{"user_name"})
+
 	failedQueriesByUser = promauto.NewCounterVec(
 		prometheus.CounterOpts{
 			Name: "ccq_server_failed_queries_by_user",