NFR Test Results for NGF version 1.3.0

tests/results/dp-perf/1.3.0/1.3.0.md

@@ -0,0 +1,84 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+- Performance seems to have improved.
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 934.498µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  594.289µs, 940.484µs, 894.291µs, 1.076ms, 1.155ms, 1.475ms, 26.622ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.00, 999.97
+Duration      [total, attack, wait]             30.001s, 30s, 885.05µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  630.729µs, 925.423µs, 909.768µs, 1.061ms, 1.117ms, 1.264ms, 10.285ms
+Bytes In      [total, mean]                     4770000, 159.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 867.406µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  649.447µs, 939.602µs, 913.399µs, 1.068ms, 1.137ms, 1.32ms, 36.366ms
+Bytes In      [total, mean]                     5010000, 167.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.02, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 798.43µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  639.884µs, 941.213µs, 920.839µs, 1.067ms, 1.117ms, 1.266ms, 16.095ms
+Bytes In      [total, mean]                     4680000, 156.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.00, 999.97
+Duration      [total, attack, wait]             30.001s, 30s, 1.006ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  638.741µs, 951.815µs, 936.228µs, 1.085ms, 1.138ms, 1.261ms, 9.666ms
+Bytes In      [total, mean]                     4680000, 156.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```

tests/results/ngf-upgrade/1.3.0/1.3.0.md

@@ -0,0 +1,49 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+- Similar results as last time.
+
+## Test: Send http /coffee traffic
+
+```text
+Requests      [total, rate, throughput]         6000, 100.02, 100.01
+Duration      [total, attack, wait]             59.992s, 59.99s, 1.099ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  795.938µs, 1.153ms, 1.119ms, 1.308ms, 1.386ms, 1.658ms, 13.45ms
+Bytes In      [total, mean]                     954000, 159.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:6000  
+Error Set:
+```
+
+![http.png](http.png)
+
+## Test: Send https /tea traffic
+
+```text
+Requests      [total, rate, throughput]         6000, 100.02, 100.00
+Duration      [total, attack, wait]             59.999s, 59.991s, 7.907ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  836.052µs, 1.21ms, 1.16ms, 1.38ms, 1.481ms, 1.842ms, 13.741ms
+Bytes In      [total, mean]                     912000, 152.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:6000  
+Error Set:
+```
+
+![https.png](https.png)

tests/results/scale/1.3.0/1.3.0.md

@@ -0,0 +1,202 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.28.9-gke.1000000
+- vCPUs per node: 16
+- RAM per node: 65855088Ki
+- Max pods per node: 110
+- Zone: us-east1-b
+- Instance Type: n2d-standard-16
+
+## Summary
+
+- New look since we are now auto-generating these results
+- Batch processing time has reduced
+- Reload time is similar (slightly better in some cases)
+- Metrics (CPU, memory, TTR) all seem to be better, though this could be because of the automation instead of being a manual process.
+
+## Test TestScale_Listeners
+
+### Reloads
+
+- Total: 126
+- Total Errors: 0
+- Average Time: 140ms
+- Reload distribution:
+	- 500ms: 126
+	- 1000ms: 126
+	- 5000ms: 126
+	- 10000ms: 126
+	- 30000ms: 126
+	- +Infms: 126
+
+### Event Batch Processing
+
+- Total: 385
+- Average Time: 124ms
+- Event Batch Processing distribution:
+	- 500ms: 350
+	- 1000ms: 383
+	- 5000ms: 385
+	- 10000ms: 385
+	- 30000ms: 385
+	- +Infms: 385
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_Listeners) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPSListeners
+
+### Reloads
+
+- Total: 128
+- Total Errors: 0
+- Average Time: 165ms
+- Reload distribution:
+	- 500ms: 128
+	- 1000ms: 128
+	- 5000ms: 128
+	- 10000ms: 128
+	- 30000ms: 128
+	- +Infms: 128
+
+### Event Batch Processing
+
+- Total: 451
+- Average Time: 115ms
+- Event Batch Processing distribution:
+	- 500ms: 411
+	- 1000ms: 450
+	- 5000ms: 451
+	- 10000ms: 451
+	- 30000ms: 451
+	- +Infms: 451
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_HTTPSListeners) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPRoutes
+
+### Reloads
+
+- Total: 1001
+- Total Errors: 0
+- Average Time: 363ms
+- Reload distribution:
+	- 500ms: 756
+	- 1000ms: 1001
+	- 5000ms: 1001
+	- 10000ms: 1001
+	- 30000ms: 1001
+	- +Infms: 1001
+
+### Event Batch Processing
+
+- Total: 1008
+- Average Time: 408ms
+- Event Batch Processing distribution:
+	- 500ms: 665
+	- 1000ms: 1008
+	- 5000ms: 1008
+	- 10000ms: 1008
+	- 30000ms: 1008
+	- +Infms: 1008
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_HTTPRoutes) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_UpstreamServers
+
+### Reloads
+
+- Total: 117
+- Total Errors: 0
+- Average Time: 125ms
+- Reload distribution:
+	- 500ms: 117
+	- 1000ms: 117
+	- 5000ms: 117
+	- 10000ms: 117
+	- 30000ms: 117
+	- +Infms: 117
+
+### Event Batch Processing
+
+- Total: 119
+- Average Time: 194ms
+- Event Batch Processing distribution:
+	- 500ms: 119
+	- 1000ms: 119
+	- 5000ms: 119
+	- 10000ms: 119
+	- 30000ms: 119
+	- +Infms: 119
+
+### Errors
+
+- NGF errors: 0
+- NGF container restarts: 0
+- NGINX errors: 0
+- NGINX container restarts: 0
+
+### Graphs and Logs
+
+See [output directory](./TestScale_UpstreamServers) for more details.
+The logs are attached only if there are errors.
+
+## Test TestScale_HTTPMatches
+
+```text
+Requests      [total, rate, throughput]         29999, 1000.01, 999.04
+Duration      [total, attack, wait]             30s, 29.999s, 933.413µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  543.448µs, 920.709µs, 856.122µs, 988.132µs, 1.045ms, 1.255ms, 35.219ms
+Bytes In      [total, mean]                     4799756, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           99.91%
+Status Codes  [code:count]                      200:29971  502:28  
+Error Set:
+502 Bad Gateway
+```
+```text
+Requests      [total, rate, throughput]         29999, 1000.03, 1000.00
+Duration      [total, attack, wait]             29.999s, 29.998s, 932.511µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  693.08µs, 957.533µs, 941.256µs, 1.085ms, 1.133ms, 1.262ms, 3.726ms
+Bytes In      [total, mean]                     4799840, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:29999  
+Error Set:
+```