Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jetty-based benchmark for memory usage #329

Merged
merged 9 commits into from
Apr 17, 2020
Merged

Jetty-based benchmark for memory usage #329

merged 9 commits into from
Apr 17, 2020

Conversation

pmm-sumo
Copy link
Contributor

This PR attempts at measuring what is the impact of having the agent on the memory profile of app. Essentially a first part of issue: #312

I added a basic custom memory profiler that measures the amount of memory consumed before and after the series of operations.

The benchmark runs an embedded Jetty server with a basic servlet and runs a series of GET requests. Both parts should be automatically instrumented and impact memory and cpu usage.

Running this locally on macOS and OpenJDK 11.0.6, I got following results:

gc.alloc.rate.norm heap.used.before
No agent 55,927 B/op 9,826,125 B
OT 0.2.3-SNAPSHOT 88,233 B/op 31,947,220 B

Raw benchmark results:

HttpBenchmark.WithOT023ExpAgent.testMakingRequest                                   avgt   10         13.186 ±         0.856   ms/op
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.alloc.rate                    avgt   10          6.074 ±         0.364  MB/sec
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.alloc.rate.norm               avgt   10      88233.833 ±       285.569    B/op
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.churn.G1_Eden_Space           avgt   10          5.337 ±         3.140  MB/sec
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.churn.G1_Eden_Space.norm      avgt   10      77916.125 ±     46270.096    B/op
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.churn.G1_Survivor_Space       avgt   10          0.057 ±         0.139  MB/sec
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.churn.G1_Survivor_Space.norm  avgt   10        822.790 ±      2003.598    B/op
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.count                         avgt   10         14.000                  counts
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·gc.time                          avgt   10        111.000                      ms
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.total.after                 avgt   10  229428428.800 ± 133367802.800   bytes
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.total.before                avgt   10  115133644.800 ±  47721866.371   bytes
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.total.change                avgt   10  114294784.000 ± 177573740.280   bytes
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.used.after                  avgt   10   96286686.400 ±  23026009.240   bytes
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.used.before                 avgt   10   31947220.800 ±  11074380.709   bytes
HttpBenchmark.WithOT023ExpAgent.testMakingRequest:·heap.used.change                 avgt   10   64339465.600 ±  27097305.905   bytes
HttpBenchmark.testMakingRequest                                                     avgt   10         12.438 ±         0.567   ms/op
HttpBenchmark.testMakingRequest:·gc.alloc.rate                                      avgt   10          4.082 ±         0.188  MB/sec
HttpBenchmark.testMakingRequest:·gc.alloc.rate.norm                                 avgt   10      55927.110 ±       220.234    B/op
HttpBenchmark.testMakingRequest:·gc.churn.G1_Eden_Space                             avgt   10          4.732 ±         0.392  MB/sec
HttpBenchmark.testMakingRequest:·gc.churn.G1_Eden_Space.norm                        avgt   10      64861.886 ±      5367.398    B/op
HttpBenchmark.testMakingRequest:·gc.churn.G1_Old_Gen                                avgt   10          1.356 ±         0.230  MB/sec
HttpBenchmark.testMakingRequest:·gc.churn.G1_Old_Gen.norm                           avgt   10      18629.961 ±      3734.859    B/op
HttpBenchmark.testMakingRequest:·gc.count                                           avgt   10        119.000                  counts
HttpBenchmark.testMakingRequest:·gc.time                                            avgt   10        191.000                      ms
HttpBenchmark.testMakingRequest:·heap.total.after                                   avgt   10   41313894.400 ±   3007890.117   bytes
HttpBenchmark.testMakingRequest:·heap.total.before                                  avgt   10   41313894.400 ±   3007890.117   bytes
HttpBenchmark.testMakingRequest:·heap.total.change                                  avgt   10            ≈ 0                   bytes
HttpBenchmark.testMakingRequest:·heap.used.after                                    avgt   10   19197616.800 ±   3515717.907   bytes
HttpBenchmark.testMakingRequest:·heap.used.before                                   avgt   10    9826125.600 ±    553095.971   bytes
HttpBenchmark.testMakingRequest:·heap.used.change                                   avgt   10    9371491.200 ±   3630360.420   bytes

trask
trask reviewed Apr 16, 2020
View reviewed changes
Copy link
Member

@trask trask left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! A couple of comments below.

benchmark/src/jmh/java/io/opentelemetry/benchmark/HttpBenchmark.java Outdated Show resolved Hide resolved
benchmark/benchmark.gradle
@@ -25,6 +30,7 @@ jmh {

// profilers = ['stack:lines=5;detailLine=true;period=5;excludePackages=true']
// Use profilers to collect additional data. Supported profilers: [cl, comp, gc, stack, perf, perfnorm, perfasm, xperf, xperfasm, hs_cl, hs_comp, hs_gc, hs_rt, hs_thr]
profilers = ['io.opentelemetry.benchmark.UsedMemoryProfiler', 'gc']
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is cool, I hadn't seen custom profiler before.

benchmark/src/jmh/java/io/opentelemetry/benchmark/UsedMemoryProfiler.java Outdated Show resolved Hide resolved
benchmark/src/jmh/java/io/opentelemetry/benchmark/UsedMemoryProfiler.java Outdated Show resolved Hide resolved
benchmark/src/jmh/java/io/opentelemetry/benchmark/classes/HttpClass.java Outdated Show resolved Hide resolved
benchmark/src/jmh/java/io/opentelemetry/benchmark/classes/HttpClass.java Outdated Show resolved Hide resolved
benchmark/src/jmh/java/io/opentelemetry/benchmark/classes/HttpClass.java Outdated Show resolved Hide resolved
pmm-sumo and others added 6 commits April 17, 2020 11:55
@pmm-sumo @trask
Remove unnecessary code
561e44e 
Co-Authored-By: Trask Stalnaker <trask.stalnaker@gmail.com>
@pmm-sumo @trask
Remove unnecessary code
1312abc 
Co-Authored-By: Trask Stalnaker <trask.stalnaker@gmail.com>
@pmm-sumo @trask
Remove unnecessary code
83e023a 
Co-Authored-By: Trask Stalnaker <trask.stalnaker@gmail.com>
@pmm-sumo
Formatting and code cleanup
2fb6811
@pmm-sumo
Formatting fix
6344a6b
@trask
Merge branch 'master' into memory-churn-benchmark
fae9094
trask
trask approved these changes Apr 17, 2020
View reviewed changes
Copy link
Member

@trask trask left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@trask trask merged commit ddb9988 into open-telemetry:master Apr 17, 2020
@pmm-sumo pmm-sumo mentioned this pull request Jun 6, 2021
Closed
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@trask trask trask approved these changes

@jkwatson jkwatson Awaiting requested review from jkwatson

@safris safris Awaiting requested review from safris

@tylerbenson tylerbenson Awaiting requested review from tylerbenson

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pmm-sumo @trask