-
Notifications
You must be signed in to change notification settings - Fork 2
Experiments
To reproduce Single Datacenter Experiment from SOSP 2015 TAPIR paper
- AWS m5.large EC2 instances: 2 vCPUs, 2.5 GHz, Intel Xeon Platinum 8175, 8 GiB memory, 10 Gb NIB
- All VMs are in one US region - US West (Oregon). 3 availability zones (a, b and c) within the region.
- Ubuntu 16.04 LTS. To compile and run Tapir you will need to install following libraries: libprotobuf-dev, protobuf-compiler, libevent-dev, libssl-dev, make, g++, pkgconf, git (link to VM snapshot to be added).
- 10 shards, 1 million keys, zipf coefficient 0.75
- Retwis Benchmark: retwisClient has 4 txn types with different Get/Put ratio (see table 2, SOSP 2015 paper). Average numbers of Puts and Gets per txn are 2 and 4 respectively.
- Clock sync - NTP, you will need to install ntp and ntpdate on hosts
4 VMs: 1 host (av. zone a) and 3 servers (one per each av.zone), each server contains copy of all 10 shards.
Description: 1 client, different running times: 30 sec, 1 min, 2min, 4 min.
Results: running time within tested range doesn't affect throughput.
Description: 1/2/4 client(s), 10 runs for each number of clients, each run - 10 sec.
Preliminary results here. Each summary table contains following stats - average latency for Put, Get and Commit, Throughput, Abort ratio, average Latency and Number of retries for successful txns, average Latency and Number of retries for failures.
- 1 client - stable performance, average 480 txn/sec.
- 2 clients - unstable, big failure latency for 2 exp (500 ms and 350 ms, see runs 1 and 4), max throughput - 850 txn/sec.
- 4 clients - unstable, big failure latency for 1 exp (100 ms, see run 5), max throughput - 1300 txn/sec.
Questions:
- To get 18000 txn/sec we will need 40 clients if we have linear increase in performance (which is unlikely). Looks like 1 client performance is low. How many clients did Irene have? What was 1 client performance?
- If we run several clients on one VM we do not have clock skew. What might be the reason for observed big failure latency?
Answers:
- For 2104 TAPIR technical report authors used 70 clients to produce throughput ~8800 txn/sec (see fig.9, 2014 Tapir tech. report). Experiment setup was slightly different (5 shards, zipf coef 0.6), but results might be used as an estimate.
- To take fast path IR client should receive super quorum of matching replies, in our case it's 3 out of 3 replicas. If super quorum replies didn't match IR client took timeout (max 500 ms) before it switched to slow path. This behavior was the reason for big failure latency. It was corrected by Irene - IR client switches to slow path immediately if super quorum replies do not match. Changes were merged with our github repo.
Final results here
- Performance: 1 client - 460 txn/sec, 2 clients - 800 txn/sec, 4 clients - 1100 txn/sec
5 VMs: 2 hosts (both in av.zone a) and 3 servers (one per each av.zone), each server contains copy of all 10 shards.
Description: 1/2/4 client(s) per host, 10 runs for each number of clients, each run - 10 sec.
Preliminary results: 1 client, 2 clients and 4 clients per host. Average Latency for Put, Get and Commit, Throughput, Abort rate, Latency and Number of retries are shown for each host. Also provided: skew and total throughput (calculated from merged log).
- Big failure latency (1 client - runs 7 and 9; 2 clients - run 3; 4 clients - 50% of all runs)
- Difference in performance even for close skew values (1 client runs 8 and 9)
- Max total performance: 1 cl per host - 900 txn/sec, 2 cls per host - 1500 txn/sec, 4 cls per host - 2000 txn/sec.
- For bigger clock skew (8-17 ms) clients hang, have to terminate manually.
Questions:
- Similar skew and different performance. What might be a reason? Probably it is related to the same issue for 1 host experiment.
- Very unstable behavior for 4 cls per host. Reasons? To check: CPU usage.
Answers:
- It was the timeout issue (see 1 host experiment).
- Timeout issue as well. Host CPU usage checked: we almost reach 100% of host CPU with 4 clients.
Final results 1 client, 2 clients, 4 clients per host.
- Performance: 2 cls (1cl/h) - 900 txn/sec, 4cls (2cl/h) - 1500 txn/sec, 8cls (4cl/h) - 2200 txn/sec
- Result obtained for 8cls is close to 2104 Tapir tech. report (10cls ~2500 txn/sec). Decided to increase number of clients in order to measure throughput for 20 and 30 cls.
Initial setup for this experiment was 5 hosts - 3 servers, but we reached 100% of server CPU usage with 10 clients. Decided to split shards between 3 servers: 3, 3 and 4 respectively; each shard has 3 copies, one per availability zone.
14 VMs: 5 hosts (2 in av.zone a, 2 in av.zone b, 1 in av.zone c) and 9 servers (three per each av.zone).
Description: 1/2/3/4 client(s) per host, 10 runs for each number of clients, each run - 10 sec. To keep clock skew low (< 3.4ms, see table 1, SOSP 2015 paper) sometimes we had to force clock sync on hosts.
Results 1 client, 2 clients, 3 clients, 4 clients per host. Average Latency for Put, Get and Commit, Throughput, Abort rate, Latency and Number of retries are shown for each host. Skew presented in tables is a skew between host 1 and host 2/3/4/5. Total throughput is calculated from merged log.
- Performance: 5cls - 2550 txn/sec, 10 cls - 4200 txn/sec, 15 cls - 5100 txn/sec, 20 cls - 5700 txn/sec
- Throughput could be increased by assigning less shards per server, preferably 1 shard per server (means that we need at least 30 VMs for servers), and increasing number of hosts.
- Decided to compare TAPIR with CockroachDB that wasn't considered in SOSP 2105 paper.