Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sa-east-1 (Sao Paulo) -- > elasticache eu-central-1 (Frankfurt) ~200ms 3 limits with low time periods #87

Closed
Tracked by #62
eguzki opened this issue Mar 22, 2024 · 0 comments
Closed
Tracked by #62

sa-east-1 (Sao Paulo) -- > elasticache eu-central-1 (Frankfurt) ~200ms 3 limits with low time periods #87

eguzki opened this issue Mar 22, 2024 · 0 comments
Assignees
@eguzki

Comments

@eguzki
Copy link
Contributor

eguzki commented Mar 22, 2024 •
edited
Loading

Test setup

Elasticache <- EC2 Redis Proxy (socat)  < ---------------- INET --------------- EC2 Limitador - Driver

Redis proxy

1 EC2 instance

amazon linux 2023 AMI

type: c7i.large 2vCPU 4Gib Memory

sudo yum update -y
sudo yum install -y socat
socat TCP-LISTEN:9876,fork,reuseaddr TCP:${ELASTICACHE_PRIMARY_ENDPOINT}

Elasticache

  • Deployment option -> Design your own cache
  • Creation Method -> Cluster cache
  • Cluster mode -> disabled
  • Multi-AZ -> disabled
  • Auto-failover -> enabled
  • Engine version -> 7.0.7
  • Node type -> cache.m5.2xlarge (x86_64, 8vCPU, 26GiB memory, Enhanced I/O, Enhanced I/O Multiplexing)
  • VPC -> same as the client EC2 instance
  • AZ -> us-east-1a
  • 2 node (1 replicas)
  • Backup -> disabled
  • Security groups -> Allow TCP traffic to port 6379 only from client node.
  • Logs -> disabled

Limtador and Driver

1 EC2 instance

amazon linux 2023 AMI

type: c7i xlarge 4vCPU 8Gib Memory

sudo yum update -y
sudo yum install -y docker
sudo service docker start
sudo usermod -a -G docker ec2-user
# (logout and login again)
docker ps

Create docker image with limitador and limitador-driver

cat <<'EOF' >Dockerfile
ARG LIMITADOR_IMAGE

FROM $LIMITADOR_IMAGE as limitador

# Stage 1: Build limitador-driver
FROM rust:1.76-bullseye as builder

ARG LIMITADOR_DRIVER_REPO=https://github.com/alexsnaps/limitador-driver.git
ARG LIMITADOR_DRIVER_BRANCH=main

RUN apt update && apt upgrade -y \
    && apt install -y protobuf-compiler git

RUN git clone --depth 1 $LIMITADOR_DRIVER_REPO -b $LIMITADOR_DRIVER_BRANCH /opt/limitador-driver

WORKDIR /opt/limitador-driver

RUN cargo build --release

# Stage 2: Setup the environment to run limitador-driver and limitador
FROM rust:1.76-bullseye

# Copy the Rust binary from the builder stage
COPY --from=builder /opt/limitador-driver/target/release/limitador-driver /usr/local/bin

# Copy the limitador service into the container
COPY --from=limitador /home/limitador/bin/limitador-server /usr/local/bin

# Run the script
CMD "/bin/bash"
EOF

Build image based on some limitador image, for example quay.io/kuadrant/limitador:write-behind-lock

docker build --build-arg="LIMITADOR_IMAGE=quay.io/kuadrant/limitador:write-behind-lock" -t write-behind-lock .

Limits file

cat <<EOF >limits.yaml
---
- namespace: test_namespace
  max_value: 100000
  seconds: 5
  conditions:
  variables:
- namespace: test_namespace
  max_value: 100000
  seconds: 4
  conditions:
  variables:
- namespace: test_namespace
  max_value: 100000
  seconds: 3
  conditions:
  variables:
EOF

Start container

docker run --name limitador-driver --rm -ti -v $PWD/limits.yaml:/tmp/limits.yaml write-behind-lock

Start limitador

/usr/local/bin/limitador-server -vv /tmp/limits.yaml redis_cached --ratio 1 redis://$REDIS_PROXY:9876
Limitador Server v1.4.0-dev (83fe4e51) [] release
2024-03-22T16:03:12.885274Z  INFO limitador_server: Version: v1.4.0-dev (83fe4e51) [] release    
2024-03-22T16:03:12.885291Z  INFO limitador_server: Using config: Configuration { limits_file: "/tmp/limits.yaml", storage: Redis(RedisStorageConfiguration { url: "redis://ec2-54-93-63-180.eu-central-1.compute.amazonaws.com:9876", cache: Some(RedisStorageCacheConfiguration { flushing_period: 1000, max_ttl: 5000, ttl_ratio: 1, max_counters: 10000 }) }), rls_host: "0.0.0.0", rls_port: 8081, http_host: "0.0.0.0", http_port: 8080, limit_name_in_labels: false, tracing_endpoint: "", log_level: Some(LevelFilter::INFO), rate_limit_headers: None, grpc_reflection_service: false }    
2024-03-22T16:03:13.087394Z  INFO limitador_server: limits file path: /tmp/limits.yaml    
2024-03-22T16:03:13.087781Z  INFO limitador_server: Envoy RLS server starting on 0.0.0.0:8081    
2024-03-22T16:03:13.087804Z  INFO limitador_server: HTTP server starting on 0.0.0.0:8080    
2024-03-22T16:03:13.087982Z  INFO actix_server::builder: starting 4 workers
2024-03-22T16:03:13.087995Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime

Open a new shell the ec2 instance

docker exec -ti limitador-driver /bin/bash

Run the driver to start the test

/usr/local/bin/limitador-driver rpc://127.0.0.1:8081 1m

Results

Hitting Limitador at rpc://127.0.0.1:8081 for a total duration of 1 minutes
Current: hits: 4850, mean: 0.205ms, p50: 0.077ms, p90: 0.090ms, p99: 0.117ms, p999: 0.423ms, max: 606.207ms
Current: hits: 12238, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.120ms, p999: 0.313ms, max: 0.475ms
Current: hits: 12271, mean: 0.081ms, p50: 0.077ms, p90: 0.091ms, p99: 0.117ms, p999: 0.295ms, max: 0.363ms
Current: hits: 9708, mean: 0.102ms, p50: 0.078ms, p90: 0.091ms, p99: 0.123ms, p999: 0.391ms, max: 202.751ms
Current: hits: 9899, mean: 0.100ms, p50: 0.077ms, p90: 0.090ms, p99: 0.112ms, p999: 0.313ms, max: 201.727ms
Current: hits: 5013, mean: 0.241ms, p50: 0.078ms, p90: 0.092ms, p99: 0.122ms, p999: 0.409ms, max: 391.167ms
Current: hits: 4993, mean: 0.202ms, p50: 0.077ms, p90: 0.091ms, p99: 0.124ms, p999: 0.339ms, max: 403.455ms
Current: hits: 12207, mean: 0.096ms, p50: 0.076ms, p90: 0.089ms, p99: 0.112ms, p999: 0.305ms, max: 201.727ms
Current: hits: 7320, mean: 0.136ms, p50: 0.076ms, p90: 0.089ms, p99: 0.123ms, p999: 0.563ms, max: 403.455ms
Current: hits: 9844, mean: 0.101ms, p50: 0.077ms, p90: 0.090ms, p99: 0.118ms, p999: 0.353ms, max: 201.727ms
Current: hits: 7311, mean: 0.136ms, p50: 0.077ms, p90: 0.090ms, p99: 0.116ms, p999: 0.367ms, max: 403.455ms
Current: hits: 2492, mean: 0.489ms, p50: 0.076ms, p90: 0.091ms, p99: 0.128ms, p999: 214.015ms, max: 403.455ms
Current: hits: 12193, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.120ms, p999: 0.291ms, max: 0.453ms
Current: hits: 9634, mean: 0.123ms, p50: 0.078ms, p90: 0.092ms, p99: 0.122ms, p999: 0.323ms, max: 403.455ms
Current: hits: 4751, mean: 0.210ms, p50: 0.078ms, p90: 0.091ms, p99: 0.124ms, p999: 0.345ms, max: 403.455ms
Current: hits: 12388, mean: 0.080ms, p50: 0.076ms, p90: 0.089ms, p99: 0.117ms, p999: 0.323ms, max: 0.539ms
Current: hits: 7519, mean: 0.159ms, p50: 0.076ms, p90: 0.088ms, p99: 0.117ms, p999: 0.395ms, max: 397.311ms
Current: hits: 2458, mean: 0.407ms, p50: 0.078ms, p90: 0.092ms, p99: 0.126ms, p999: 0.477ms, max: 403.455ms
Current: hits: 2376, mean: 0.420ms, p50: 0.078ms, p90: 0.092ms, p99: 0.119ms, p999: 0.763ms, max: 403.455ms
Current: hits: 12073, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.118ms, p999: 0.321ms, max: 0.473ms
Current: hits: 7167, mean: 0.139ms, p50: 0.079ms, p90: 0.093ms, p99: 0.122ms, p999: 0.331ms, max: 403.455ms
Current: hits: 7168, mean: 0.167ms, p50: 0.080ms, p90: 0.094ms, p99: 0.122ms, p999: 0.329ms, max: 397.311ms
Current: hits: 2369, mean: 0.424ms, p50: 0.079ms, p90: 0.093ms, p99: 0.171ms, p999: 0.523ms, max: 403.455ms
Current: hits: 12393, mean: 0.080ms, p50: 0.077ms, p90: 0.089ms, p99: 0.113ms, p999: 0.303ms, max: 0.441ms
Current: hits: 9701, mean: 0.102ms, p50: 0.077ms, p90: 0.091ms, p99: 0.129ms, p999: 0.335ms, max: 202.751ms
Current: hits: 7299, mean: 0.136ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.303ms, max: 403.455ms
Current: hits: 9679, mean: 0.143ms, p50: 0.078ms, p90: 0.092ms, p99: 0.121ms, p999: 0.339ms, max: 389.119ms
Current: hits: 2578, mean: 0.390ms, p50: 0.077ms, p90: 0.090ms, p99: 0.121ms, p999: 0.475ms, max: 403.455ms
Current: hits: 2355, mean: 0.424ms, p50: 0.078ms, p90: 0.092ms, p99: 0.119ms, p999: 0.445ms, max: 403.455ms
Current: hits: 12242, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.115ms, p999: 0.307ms, max: 0.493ms
Current: hits: 7450, mean: 0.133ms, p50: 0.077ms, p90: 0.091ms, p99: 0.124ms, p999: 0.387ms, max: 389.119ms
Current: hits: 2565, mean: 0.546ms, p50: 0.078ms, p90: 0.091ms, p99: 0.122ms, p999: 387.071ms, max: 403.455ms
Current: hits: 9585, mean: 0.104ms, p50: 0.078ms, p90: 0.091ms, p99: 0.124ms, p999: 0.447ms, max: 206.847ms
Current: hits: 9723, mean: 0.102ms, p50: 0.078ms, p90: 0.092ms, p99: 0.119ms, p999: 0.289ms, max: 202.751ms
Current: hits: 7061, mean: 0.170ms, p50: 0.077ms, p90: 0.091ms, p99: 0.118ms, p999: 0.563ms, max: 216.063ms
Current: hits: 2352, mean: 0.424ms, p50: 0.078ms, p90: 0.092ms, p99: 0.122ms, p999: 0.635ms, max: 403.455ms
Current: hits: 12317, mean: 0.080ms, p50: 0.077ms, p90: 0.090ms, p99: 0.111ms, p999: 0.285ms, max: 0.341ms
Current: hits: 9758, mean: 0.142ms, p50: 0.077ms, p90: 0.091ms, p99: 0.117ms, p999: 0.339ms, max: 389.119ms
Current: hits: 7399, mean: 0.163ms, p50: 0.077ms, p90: 0.090ms, p99: 0.120ms, p999: 0.493ms, max: 403.455ms
Current: hits: 7240, mean: 0.137ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.359ms, max: 403.455ms
Current: hits: 7568, mean: 0.131ms, p50: 0.078ms, p90: 0.091ms, p99: 0.115ms, p999: 0.313ms, max: 382.975ms
Current: hits: 4890, mean: 0.204ms, p50: 0.077ms, p90: 0.090ms, p99: 0.111ms, p999: 0.305ms, max: 403.455ms
Current: hits: 12199, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.116ms, p999: 0.299ms, max: 0.343ms
Current: hits: 4884, mean: 0.283ms, p50: 0.076ms, p90: 0.089ms, p99: 0.129ms, p999: 0.387ms, max: 403.455ms
Current: hits: 7293, mean: 0.136ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.307ms, max: 403.455ms
Current: hits: 11978, mean: 0.083ms, p50: 0.078ms, p90: 0.093ms, p99: 0.152ms, p999: 0.315ms, max: 1.071ms
Current: hits: 12223, mean: 0.081ms, p50: 0.078ms, p90: 0.091ms, p99: 0.120ms, p999: 0.297ms, max: 0.519ms
Current: hits: 9, mean: 133.562ms, p50: 0.170ms, p90: 403.455ms, p99: 403.455ms, p999: 403.455ms, max: 403.455ms
Current: hits: 7012, mean: 0.143ms, p50: 0.078ms, p90: 0.092ms, p99: 0.132ms, p999: 0.435ms, max: 212.991ms
Current: hits: 12095, mean: 0.082ms, p50: 0.078ms, p90: 0.092ms, p99: 0.121ms, p999: 0.315ms, max: 0.535ms
Current: hits: 7175, mean: 0.139ms, p50: 0.078ms, p90: 0.092ms, p99: 0.128ms, p999: 0.417ms, max: 403.455ms
Current: hits: 4794, mean: 0.249ms, p50: 0.078ms, p90: 0.091ms, p99: 0.114ms, p999: 0.339ms, max: 403.455ms
Current: hits: 12160, mean: 0.082ms, p50: 0.078ms, p90: 0.091ms, p99: 0.118ms, p999: 0.287ms, max: 0.451ms
Current: hits: 7178, mean: 0.193ms, p50: 0.078ms, p90: 0.091ms, p99: 0.117ms, p999: 0.305ms, max: 403.455ms
Current: hits: 2528, mean: 0.397ms, p50: 0.077ms, p90: 0.090ms, p99: 0.117ms, p999: 0.435ms, max: 403.455ms
Current: hits: 2473, mean: 0.487ms, p50: 0.077ms, p90: 0.088ms, p99: 0.119ms, p999: 203.775ms, max: 403.455ms


 ============== Done! =================

Ok: hits: 426398, mean: 0.140ms, p50: 0.078ms, p90: 0.091ms, p99: 0.120ms, p999: 0.333ms, max: 606.207ms
OverLimit: hits: 0, mean: 0.000ms, p50: 0.000ms, p90: 0.000ms, p99: 0.000ms, p999: 0.000ms, max: 0.000ms
Unknown: hits: 0, mean: 0.000ms, p50: 0.000ms, p90: 0.000ms, p99: 0.000ms, p999: 0.000ms, max: 0.000ms
Overall: hits: 426398, mean: 0.140ms, p50: 0.078ms, p90: 0.091ms, p99: 0.120ms, p999: 0.333ms, max: 606.207ms

0.01ms |                                                                                  |  0.0th %-ile
0.02ms |                                                                                  |  0.0th %-ile
0.04ms |                                                                                  |  0.0th %-ile
0.05ms | *                                                                                |  0.0th %-ile
0.06ms | *                                                                                |  0.1th %-ile
0.07ms | *****                                                                            |  5.6th %-ile
0.08ms | ****************************************************                             | 70.5th %-ile
0.10ms | ********************                                                             | 94.5th %-ile
0.11ms | ***                                                                              | 98.1th %-ile
0.12ms | *                                                                                | 99.0th %-ile
0.13ms | *                                                                                | 99.4th %-ile
0.14ms | *                                                                                | 99.5th %-ile
0.16ms | *                                                                                | 99.6th %-ile
0.17ms | *                                                                                | 99.6th %-ile
0.18ms | *                                                                                | 99.6th %-ile
0.19ms | *                                                                                | 99.7th %-ile
0.20ms | *                                                                                | 99.7th %-ile
0.22ms | *                                                                                | 99.7th %-ile
0.23ms | *                                                                                | 99.7th %-ile
0.24ms | *                                                                                | 99.7th %-ile
0.25ms | *                                                                                | 99.7th %-ile
0.26ms | *                                                                                | 99.7th %-ile
0.28ms | *                                                                                | 99.8th %-ile
0.29ms | *                                                                                | 99.8th %-ile
0.30ms | *                                                                                | 99.8th %-ile
0.31ms | *                                                                                | 99.9th %-ile
0.32ms | *                                                                                | 99.9th %-ile
0.34ms | *                                                                                | 99.9th %-ile
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eguzki eguzki

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@eguzki