During each release, we execute various automated performance test scenarios and publish the results.
| Test Scenarios | Description |
|---|---|
| Passthrough HTTP service (h1c -> h1c) | An HTTP Service, which forwards all requests to an HTTP back-end service. |
| Passthrough HTTPS service (h1 -> h1) | An HTTPS Service, which forwards all requests to an HTTPS back-end service. |
| JSON to XML transformation HTTP service | An HTTP Service, which transforms JSON requests to XML and then forwards all requests to an HTTP back-end service. |
| JSON to XML transformation HTTPS service | An HTTPS Service, which transforms JSON requests to XML and then forwards all requests to an HTTPS back-end service. |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | An HTTPS Service exposed over HTTP/2 protocol, which forwards all requests to an HTTP/2(over TLS) back-end service. |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | An HTTPS Service exposed over HTTP/2 protocol, which forwards all requests to an HTTPS back-end service. |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | An HTTPS Service exposed over HTTP/2 protocol, which forwards all requests to an HTTP back-end service. |
| HTTP/2 client and server downgrade service (h2 -> h2) | An HTTP/2(with TLS) server accepts requests from an HTTP/1.1(with TLS) client and the HTTP/2(with TLS) client sends requests to an HTTP/1.1(with TLS) back-end service. Both the upstream and the downgrade connection is downgraded to HTTP/1.1(with TLS). |
Our test client is Apache JMeter. We test each scenario for a fixed duration of time. We split the test results into warmup and measurement parts and use the measurement part to compute the performance metrics.
A majority of test scenarios use a Netty based back-end service which echoes back any request posted to it after a specified period of time.
We run the performance tests under different numbers of concurrent users, message sizes (payloads) and back-end service delays.
The main performance metrics:
- Throughput: The number of requests that the Ballerina service processes during a specific time interval (e.g. per second).
- Response Time: The end-to-end latency for an operation of invoking a Ballerina service. The complete distribution of response times was recorded.
In addition to the above metrics, we measure the load average and several memory-related metrics.
The following are the test parameters.
| Test Parameter | Description | Values |
|---|---|---|
| Scenario Name | The name of the test scenario. | Refer to the above table. |
| Heap Size | The amount of memory allocated to the application | 2G |
| Concurrent Users | The number of users accessing the application at the same time. | 100, 300, 1000 |
| Message Size (Bytes) | The request payload size in Bytes. | 50, 1024 |
| Back-end Delay (ms) | The delay added by the back-end service. | 0 |
The duration of each test is 900 seconds. The warm-up period is 300 seconds. The measurement results are collected after the warm-up period.
A c5.xlarge Amazon EC2 instance was used to install Ballerina.
The following are the measurements collected from each performance test conducted for a given combination of test parameters.
| Measurement | Description |
|---|---|
| Error % | Percentage of requests with errors |
| Average Response Time (ms) | The average response time of a set of results |
| Standard Deviation of Response Time (ms) | The “Standard Deviation” of the response time. |
| 99th Percentile of Response Time (ms) | 99% of the requests took no more than this time. The remaining samples took at least as long as this |
| Throughput (Requests/sec) | The throughput measured in requests per second. |
| Average Memory Footprint After Full GC (M) | The average memory consumed by the application after a full garbage collection event. |
The following is the summary of performance test results collected for the measurement period.
| Scenario Name | Concurrent Users | Message Size (Bytes) | Back-end Service Delay (ms) | Error % | Throughput (Requests/sec) | Average Response Time (ms) | Standard Deviation of Response Time (ms) | 99th Percentile of Response Time (ms) | Ballerina GC Throughput (%) | Average Ballerina Memory Footprint After Full GC (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| Passthrough HTTP service (h1c -> h1c) | 100 | 50 | 0 | 0 | 14455.27 | 6.87 | 3.8 | 17 | N/A | N/A |
| Passthrough HTTP service (h1c -> h1c) | 100 | 1024 | 0 | 0 | 13936.99 | 7.13 | 3.83 | 17 | N/A | N/A |
| Passthrough HTTP service (h1c -> h1c) | 300 | 50 | 0 | 0 | 14986.25 | 19.96 | 7.64 | 40 | N/A | N/A |
| Passthrough HTTP service (h1c -> h1c) | 300 | 1024 | 0 | 0 | 14558.43 | 20.55 | 7.94 | 42 | N/A | N/A |
| Passthrough HTTP service (h1c -> h1c) | 1000 | 50 | 0 | 0 | 14071.93 | 70.98 | 19.17 | 125 | N/A | N/A |
| Passthrough HTTP service (h1c -> h1c) | 1000 | 1024 | 0 | 0 | 13984.43 | 71.42 | 18.86 | 125 | N/A | N/A |
| JSON to XML transformation HTTP service | 100 | 50 | 0 | 0 | 9446.15 | 10.54 | 5.61 | 27 | N/A | N/A |
| JSON to XML transformation HTTP service | 100 | 1024 | 0 | 0 | 6915.35 | 14.41 | 6.31 | 34 | N/A | N/A |
| JSON to XML transformation HTTP service | 300 | 50 | 0 | 0 | 9608.57 | 31.16 | 9.49 | 59 | N/A | N/A |
| JSON to XML transformation HTTP service | 300 | 1024 | 0 | 0 | 7085.63 | 42.28 | 9.72 | 73 | N/A | N/A |
| JSON to XML transformation HTTP service | 1000 | 50 | 0 | 0 | 9449.33 | 105.75 | 17.82 | 159 | N/A | N/A |
| JSON to XML transformation HTTP service | 1000 | 1024 | 0 | 0 | 6992.01 | 142.93 | 25.15 | 209 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 100 | 50 | 0 | 0 | 12567.74 | 7.91 | 3.98 | 19 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 100 | 1024 | 0 | 0 | 11160.17 | 8.91 | 4.16 | 20 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 300 | 50 | 0 | 0 | 12306.86 | 24.31 | 8.91 | 49 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 300 | 1024 | 0 | 0 | 11516.14 | 25.98 | 9.3 | 52 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 1000 | 50 | 0 | 0 | 11935.86 | 83.7 | 22.85 | 148 | N/A | N/A |
| Passthrough HTTPS service (h1 -> h1) | 1000 | 1024 | 0 | 0 | 11237.43 | 88.9 | 23.35 | 154 | N/A | N/A |
| JSON to XML transformation HTTPS service | 100 | 50 | 0 | 0 | 8338.35 | 11.94 | 6.06 | 30 | N/A | N/A |
| JSON to XML transformation HTTPS service | 100 | 1024 | 0 | 0 | 6295.11 | 15.83 | 7.72 | 39 | N/A | N/A |
| JSON to XML transformation HTTPS service | 300 | 50 | 0 | 0 | 8484.44 | 35.3 | 12.4 | 71 | N/A | N/A |
| JSON to XML transformation HTTPS service | 300 | 1024 | 0 | 0 | 6407.36 | 46.75 | 13.92 | 88 | N/A | N/A |
| JSON to XML transformation HTTPS service | 1000 | 50 | 0 | 0 | 8275.23 | 120.73 | 25.05 | 193 | N/A | N/A |
| JSON to XML transformation HTTPS service | 1000 | 1024 | 0 | 0 | 6027.15 | 165.81 | 30.8 | 248 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 100 | 50 | 0 | 0 | 11571.61 | 8.44 | 4.09 | 19 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 100 | 1024 | 0 | 0 | 11264.37 | 8.58 | 4.05 | 20 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 300 | 50 | 0 | 0 | 11420.76 | 25.82 | 9.25 | 51 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 300 | 1024 | 0 | 0 | 10887.93 | 26.89 | 9.52 | 54 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 1000 | 50 | 0 | 0 | 10965.21 | 90.32 | 25.36 | 163 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1c) | 1000 | 1024 | 0 | 0 | 10709.08 | 92.31 | 26.06 | 168 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 100 | 50 | 0 | 0 | 10664.01 | 9.18 | 4.25 | 21 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 100 | 1024 | 0 | 0 | 10135.61 | 9.59 | 4.25 | 22 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 300 | 50 | 0 | 0 | 10708.64 | 27.59 | 9.77 | 55 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 300 | 1024 | 0 | 0 | 10385.17 | 28.26 | 9.94 | 57 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 1000 | 50 | 0 | 0 | 10338.18 | 96.12 | 26.53 | 172 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h1) | 1000 | 1024 | 0 | 0 | 10198.86 | 96.78 | 27.28 | 175 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 100 | 50 | 0 | 0 | 12423.57 | 8 | 4.04 | 19 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 100 | 1024 | 0 | 0 | 11315.51 | 8.78 | 4.18 | 20 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 300 | 50 | 0 | 0 | 12433.65 | 24.06 | 8.84 | 48 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 300 | 1024 | 0 | 0 | 11628.96 | 25.73 | 9.28 | 51 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 1000 | 50 | 0 | 0 | 12408.6 | 80.49 | 21.75 | 142 | N/A | N/A |
| HTTP/2 client and server downgrade service (h2 -> h2) | 1000 | 1024 | 0 | 0 | 11448.62 | 87.25 | 23.56 | 152 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 100 | 50 | 0 | 0 | 11473.07 | 8.51 | 4.11 | 20 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 100 | 1024 | 0 | 0 | 11072.85 | 8.74 | 4.12 | 20 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 300 | 50 | 0 | 0 | 12007.72 | 24.57 | 8.91 | 49 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 300 | 1024 | 0 | 0 | 11778.81 | 24.79 | 8.9 | 50 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 1000 | 50 | 0 | 0 | 11874.22 | 83.79 | 22.22 | 146 | N/A | N/A |
| Passthrough HTTP/2(over TLS) service (h2 -> h2) | 1000 | 1024 | 0 | 0 | 11596.55 | 85.21 | 23.84 | 154 | N/A | N/A |