Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[test-only] Simulate dialogue clients against a DeterministicScheduler #348

Merged
merged 111 commits into from
Feb 17, 2020
Merged
Changes from 1 commit
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
4bdc207
Statistics sink interface
iamdanfox Feb 11, 2020
6b57fd5
WIP
iamdanfox Feb 11, 2020
b38159d
Eliminate one cache for simplicity
iamdanfox Feb 12, 2020
187af81
Seems like it kinda works
iamdanfox Feb 12, 2020
2c0cc5c
One more class
iamdanfox Feb 12, 2020
56a9e6d
Cache the computation on the hot path
iamdanfox Feb 12, 2020
df76f65
Randomness
iamdanfox Feb 12, 2020
1f42a68
Ticker
iamdanfox Feb 12, 2020
14e395e
Wire in ticker and codahale clock
iamdanfox Feb 12, 2020
fdd9b49
Combined ticker and deterministic scheduler
iamdanfox Feb 12, 2020
38d06ea
We can schedule stuff
iamdanfox Feb 12, 2020
1ae2078
Spin up lots of instant tests
iamdanfox Feb 12, 2020
0e8e6f3
format
iamdanfox Feb 12, 2020
5785c19
Pull class out
iamdanfox Feb 12, 2020
6fb6ba3
Initial simulation server
iamdanfox Feb 12, 2020
94b7a78
server simulates response times
iamdanfox Feb 12, 2020
347d67a
Move clock
iamdanfox Feb 12, 2020
fdcc3b5
Can draw graphs
iamdanfox Feb 12, 2020
f6212e7
SimulationMetrics
iamdanfox Feb 12, 2020
fee44df
Dump metrics to csv
iamdanfox Feb 12, 2020
6d8bc0f
Dump png
iamdanfox Feb 12, 2020
448873a
Pull out simulation test
iamdanfox Feb 12, 2020
233c91a
Support composing simulation servers
iamdanfox Feb 13, 2020
95b533e
Can I pass static analysis
iamdanfox Feb 13, 2020
e1e0967
Merge remote-tracking branch 'origin/develop' into dfox/statistics
iamdanfox Feb 13, 2020
118cf8e
Switch all logging to slf4j
iamdanfox Feb 13, 2020
70b04dc
Rename to Simulation
iamdanfox Feb 13, 2020
5e41eee
PreferLowestUpstreamUtilization
iamdanfox Feb 13, 2020
1b39887
Allow making parallel requests in batches
iamdanfox Feb 13, 2020
a44444c
Active requests counter
iamdanfox Feb 13, 2020
ea9c18c
WIP
iamdanfox Feb 13, 2020
6e072d6
ConcurrencyLimitedChannel has a clock
iamdanfox Feb 14, 2020
257e703
Test against concurrency limiters
iamdanfox Feb 14, 2020
7dafc33
Print out the mean from the client-side perspective
iamdanfox Feb 14, 2020
b4d6a79
Move everything to new :simulation project
iamdanfox Feb 14, 2020
392461c
Uncaught exception handler
iamdanfox Feb 14, 2020
b4d987b
Allow filtering metrics to print
iamdanfox Feb 14, 2020
6880f42
PreferLowestUpstreamUtilization implements LimitedChannel
iamdanfox Feb 14, 2020
db4d4a3
PreferLowestUtilization is neater
iamdanfox Feb 14, 2020
7c5e843
Move the code
iamdanfox Feb 15, 2020
577e9da
More move instrumentation
iamdanfox Feb 15, 2020
d655eb8
SimulationTest is now parameterized
iamdanfox Feb 15, 2020
13af858
This is beautiful
iamdanfox Feb 15, 2020
12961a6
prefix names
iamdanfox Feb 15, 2020
e90a29a
track status codes
iamdanfox Feb 15, 2020
369aa85
Track percentage success
iamdanfox Feb 15, 2020
5dcd96a
Random tiebreaking
iamdanfox Feb 15, 2020
481d3e8
linearResponseTime
iamdanfox Feb 15, 2020
d9c94ce
Nicer png dumping
iamdanfox Feb 15, 2020
72f7e9e
refactor PreferLowest to take LimitedChannels
iamdanfox Feb 15, 2020
543e201
PreferLowest takes limited channels
iamdanfox Feb 15, 2020
84a56c5
Combo charts
iamdanfox Feb 15, 2020
264be81
Check in some pngs
iamdanfox Feb 15, 2020
b580e81
Return typed results from benchmark
iamdanfox Feb 15, 2020
6e0b7b2
Charts include som key stats
iamdanfox Feb 15, 2020
d1171c4
Blacklisting channel behaves weirdly when the floodgates re-open
iamdanfox Feb 15, 2020
7b256ca
Blacklisting channel is drastically better
iamdanfox Feb 16, 2020
a009747
emphasise three states
iamdanfox Feb 16, 2020
36b0974
update graphs & pass static analysis
iamdanfox Feb 16, 2020
3c949f0
fast_500s_then_revert initial graphs
iamdanfox Feb 16, 2020
c418523
Blacklist host after receiving a 500
iamdanfox Feb 16, 2020
4220edf
drastic_slowdown graphs
iamdanfox Feb 16, 2020
ff87fbc
Needless generic
iamdanfox Feb 16, 2020
2364460
RetryingChannel retries 503 and 500s
iamdanfox Feb 16, 2020
8cafb90
PreferLowestUtilization was counting wrong!
iamdanfox Feb 16, 2020
2ad087d
Maybe caffeine iteration isn't deterministic?
iamdanfox Feb 16, 2020
31f7a9e
new all_nodes_500 case
iamdanfox Feb 16, 2020
49cf31c
Try QueuedChannel on LOWEST_UTILIZATION
iamdanfox Feb 16, 2020
367bf60
Trivial randomness move
iamdanfox Feb 16, 2020
a9c0547
Lossless metrics
iamdanfox Feb 16, 2020
42f6339
Delete unnecessary abstraction
iamdanfox Feb 16, 2020
128051b
Benchmark has smarter finish conditions
iamdanfox Feb 16, 2020
c164cfa
No change
iamdanfox Feb 16, 2020
6cb43b3
Use stream internally
iamdanfox Feb 16, 2020
5ce1e84
Way nicer API
iamdanfox Feb 16, 2020
be42176
New black_hole scenario
iamdanfox Feb 16, 2020
67a7d75
Use time to bound simulations
iamdanfox Feb 16, 2020
7a18393
Dial up simplest
iamdanfox Feb 16, 2020
d1fa9ad
SimulationServer can have multiple endpoints
iamdanfox Feb 16, 2020
f3c94dd
New scenario: one_endpoint_dies_on_each_server
iamdanfox Feb 16, 2020
11d1162
Simpler API
iamdanfox Feb 16, 2020
32f77fe
Per-endpoint metrics
iamdanfox Feb 16, 2020
5571cc0
Randomize endpoint calls
iamdanfox Feb 16, 2020
821c637
static analysis
iamdanfox Feb 16, 2020
aeda3ed
Merge remote-tracking branch 'origin/develop' into dfox/simulation
iamdanfox Feb 16, 2020
1821e6d
Per-endpoint blacklisting!
iamdanfox Feb 16, 2020
c72ac47
Don't need composed thingy anymore
iamdanfox Feb 16, 2020
994340d
Track cumulative server time
iamdanfox Feb 17, 2020
941bc58
Move all pngs to src/test/resources
iamdanfox Feb 17, 2020
31eed09
Sweet aggregate txt report
iamdanfox Feb 17, 2020
2cabe21
cute markers
iamdanfox Feb 17, 2020
de5fa01
SimulationTest ready to live reload
iamdanfox Feb 17, 2020
dde229a
new scenario: live_reloading
iamdanfox Feb 17, 2020
199e585
Tidying
iamdanfox Feb 17, 2020
2565396
Extra fancy failure threshold
iamdanfox Feb 17, 2020
562f7b2
Debugging the spike
iamdanfox Feb 17, 2020
f323f14
Merge remote-tracking branch 'origin/develop' into dfox/simulation
iamdanfox Feb 17, 2020
a2955fa
static analsysi
iamdanfox Feb 17, 2020
2f954ff
Revert RetryingChannel
iamdanfox Feb 17, 2020
54c5646
Revert BlacklistingChannel
iamdanfox Feb 17, 2020
5ffefea
Delete LOWEST_UTILIZATION for now
iamdanfox Feb 17, 2020
dc43ad4
SimpleLimiter -> Limiter for now
iamdanfox Feb 17, 2020
12a24f1
Combine RefreshingChannel file
iamdanfox Feb 17, 2020
3468b5a
Merge remote-tracking branch 'origin/develop' into dfox/simulation
iamdanfox Feb 17, 2020
90889ee
Supplier<Long> -> Ticker
iamdanfox Feb 17, 2020
cf3acae
use longs everywhere
iamdanfox Feb 17, 2020
888242f
always debug log
iamdanfox Feb 17, 2020
765b6b5
Dead code
iamdanfox Feb 17, 2020
28d9b4b
Improve comment
iamdanfox Feb 17, 2020
c445cd5
Don't clobber X-B3-TraceId
iamdanfox Feb 17, 2020
19ee73a
Extra javadoc & less dead code
iamdanfox Feb 17, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update graphs & pass static analysis
  • Loading branch information
iamdanfox committed Feb 16, 2020
commit 36b0974684ba626d9427baac2a42e57086a149e2
4 changes: 4 additions & 0 deletions simulation/build.gradle
Original file line number Diff line number Diff line change
@@ -18,3 +18,7 @@ dependencies {
annotationProcessor 'org.immutables:value'
compile 'org.immutables:value::annotations'
}

tasks.withType(JavaCompile) {
options.errorprone.errorproneArgs += '-Xep:Slf4jLogsafeArgs:OFF'
}
Binary file modified simulation/simplest_possible_case[LOWEST_UTILIZATION].png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified simulation/slow_503s_then_revert[LOWEST_UTILIZATION].png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -103,7 +103,7 @@ Builder response(Response value) {
return this;
}

/** BEWARE: servers don't actually behave like this! */
/** BEWARE: servers don't actually behave like this. */
Builder responseTimeConstant(Duration duration) {
responseTime = server -> duration;
return this;
Original file line number Diff line number Diff line change
@@ -78,6 +78,7 @@ public BenchmarkResult run() {
return Futures.getUnchecked(result);
}

@SuppressWarnings("FutureReturnValueIgnored")
public SettableFuture<BenchmarkResult> schedule() {
Instant realStart = Instant.now();
SettableFuture<BenchmarkResult> done = SettableFuture.create();
Original file line number Diff line number Diff line change
@@ -39,7 +39,7 @@ final class Simulation {
private final SimulationMetrics metrics = new SimulationMetrics(this);
private final CodahaleClock codahaleClock = new CodahaleClock(ticker);

public Simulation() {
Simulation() {
Thread.currentThread().setUncaughtExceptionHandler((t, e) -> log.error("Uncaught throwable", e));
}

@@ -97,7 +97,7 @@ public long read() {
return nanos;
}

public void advanceTo(Duration duration) {
void advanceTo(Duration duration) {
long newNanos = duration.toNanos();
if (newNanos < nanos) {
log.error("TestTicker time may not go backwards current={} new={}", nanos, newNanos);
Original file line number Diff line number Diff line change
@@ -195,13 +195,12 @@ public XYChart chart(Pattern metricNameRegex) {
return chart;
}

public static final void png(String file, XYChart... charts) {
public static void png(String file, XYChart... charts) {
Stopwatch sw = Stopwatch.createStarted();
try {
int rows = charts.length;
int cols = 1;
BitmapEncoder.saveBitmap(
ImmutableList.copyOf(charts), rows, cols, file, BitmapEncoder.BitmapFormat.PNG);
BitmapEncoder.saveBitmap(ImmutableList.copyOf(charts), rows, cols, file, BitmapEncoder.BitmapFormat.PNG);
log.info("Generated {} ({} ms)", file, sw.elapsed(TimeUnit.MILLISECONDS));
} catch (IOException e) {
throw new RuntimeException(e);
Original file line number Diff line number Diff line change
@@ -64,7 +64,7 @@
*/
@RunWith(Parameterized.class)
public class SimulationTest {
private static final Endpoint endpoint = mock(Endpoint.class);
private static final Endpoint ENDPOINT = mock(Endpoint.class);

@Parameterized.Parameters(name = "{0}")
public static Strategy[] data() {
@@ -80,6 +80,7 @@ public static Strategy[] data() {
private final Simulation simulation = new Simulation();
private Benchmark.BenchmarkResult result;

@SuppressWarnings("ImmutableEnumChecker")
public enum Strategy {
LOWEST_UTILIZATION(SimulationTest::lowestUtilization),
CONCURRENCY_LIMITER(SimulationTest::concurrencyLimiter),
@@ -120,7 +121,7 @@ public void simplest_possible_case() {
result = Benchmark.builder()
.numRequests(2000)
.requestsPerSecond(50)
.channel(i -> channel.execute(endpoint, request("req-" + i)))
.channel(i -> channel.execute(ENDPOINT, request("req-" + i)))
.simulation(simulation)
.run();
}
@@ -156,7 +157,7 @@ public void slow_503s_then_revert() {
result = Benchmark.builder()
.numRequests(3000) // something weird happens at 1811... bug in DeterministicScheduler?
.requestsPerSecond(200)
.channel(i -> channel.execute(endpoint, request("req-" + i)))
.channel(i -> channel.execute(ENDPOINT, request("req-" + i)))
.simulation(simulation)
.run();
}
@@ -190,7 +191,7 @@ private static Channel lowestUtilization(Simulation sim, Channel... channels) {
.map(SimulationTest::noOpLimitedChannel)
.map(c -> new BlacklistingChannel(c, Duration.ofSeconds(1), sim.clock()))
.collect(ImmutableList.toImmutableList());
LimitedChannel idea = new PreferLowestUtilization(chans, sim.clock(), SimulationUtils.DETERMINISTIC);
LimitedChannel idea = new PreferLowestUtilization(chans, sim.clock(), SimulationUtils.newPseudoRandom());
return dontTolerateLimits(idea);
}

@@ -206,7 +207,7 @@ private static Channel concurrencyLimiter(Simulation sim, Channel... channels) {

private static Channel roundRobin(Simulation sim, Channel... channels) {
List<LimitedChannel> limitedChannels = Stream.of(channels)
.map(c -> (LimitedChannel) (endpoint, request) -> Optional.of(c.execute(endpoint, request)))
.map(c -> (LimitedChannel) (e, r) -> Optional.of(c.execute(e, r)))
.collect(Collectors.toList());
LimitedChannel limited = new RoundRobinChannel(limitedChannels);
Channel channel = new QueuedChannel(limited, DispatcherMetrics.of(new DefaultTaggedMetricRegistry()));
Original file line number Diff line number Diff line change
@@ -29,22 +29,22 @@

public final class SimulationUtils {

@SuppressWarnings("UnnecessaryAnonymousClass")
public static final Randomness DETERMINISTIC = new Randomness() {
public static Randomness newPseudoRandom() {
return new Randomness() {
private final Random random = new Random(12345L);

private final Random random = new Random(12345L);
@Override
public <T> List<T> shuffle(List<T> list) {
if (list.size() == 1 || list.isEmpty()) {
return list;
}

@Override
public <T> List<T> shuffle(List<T> list) {
if (list.size() == 1 || list.isEmpty()) {
return list;
List<T> shuffleMe = new ArrayList<>(list);
Collections.shuffle(shuffleMe, random);
return shuffleMe;
}

List<T> shuffleMe = new ArrayList<>(list);
Collections.shuffle(shuffleMe, random);
return shuffleMe;
}
};
};
}

public static Response response(int status, String version) {
return new Response() {