Profiling Memory Usage and Object Creation

[benjchristensen]

We need to spend time profiling memory and object allocation and finding places where we can improve.

I would really appreciate help diving into this and finding problem areas. Even if you don't fix them, but just identity use cases, operators, etc that would be very valuable.

This is partly a result of the fact that in Netflix production we have seen an increase in YoungGen GCs since 0.17.x.

The areas to start should probably be:

• Observable.create
• Observable.lift
• Subscriber
• CompositeSubscription
• map
• flatMap

If you can or want to get involved in this please comment here so we all can collaborate together.

[akarnokd]

The most likely source of garbage is the CompositeSubscription: whenever a task is scheduled or a new merge-source appears, the state transition creates a new State object and copies the subscription array. PR #1145 reduces the amount of garbage by switching to HashSet on a larger composite. Since flatMap->mergeMap->merge needs to track the active inner subscriptions, it uses a composite as well and would gain the same benefits.

[benjchristensen]

The HashSet change shouldn't kick in though because these aren't generally large lists (handful at most, not hundreds). I expect it's just far too many of them being created.

[daschl]

I'll also like to help out here, since its crucial for our uses as well.

[benjchristensen]

Thanks @daschl I'd appreciate your help. Profiling and identifying hot spots is what we need most right now.

[daschl]

I did some GC profiling of my test workloads and I'd also like to nominate:

• BlockingObservable (it seems that especially in the .single() case its not optimized)
• Observable.subscribe (for both Observer and Subscriber) - in my GC logs they take up 40% of the overall GC pressure.

The bad news is that i had to fall back out of Observables to plain execution on the hot code path (aside from the overall wrapping observable), because also using it in the path produces way too much garbage (moving away from Rx in the hot code path got my throughput from 20% to 80% according to the GC logs) and it correlates with my findings since I could not sustain constant IO throughput because of full GCs happening way too frequently.

[benjchristensen]

Observable.subscribe (for both Observer and Subscriber)

Not surprised on this. Were you able to identify what the garbage is?

[daschl]

I did some changes in my code and will re-profile and post the results here so we get better measurements.

Here is a slightly older profile run which might give you a hint or two.

[benjchristensen]

We can definitely improve on the AtomicReference by using AtomicFieldUpdater. The others will require more effort to analyze along with sample code to see what's triggering them.

[akarnokd]

I think that many AtomicReference come from mostly the CompositeSubscription instances.

[benjchristensen]

I went back in history to 0.16.1 to compare performance of the basic map/flatMap behavior and found that current master is faster. This fits what I had expected of the work of the past couple months, which means it's not an obvious glaring issue (unless my test is completely flawed).

Here is the code for the test:

• Branch 0.16 => https://github.com/benjchristensen/RxJava/blob/0.16-perf/rxjava-core/src/perf/java/rx/usecases/PerfTransforms.java
• Master => https://github.com/benjchristensen/RxJava/blob/performance/rxjava-core/src/perf/java/rx/usecases/PerfTransforms.java

Results

0.16

Benchmark                                       (size)   Mode   Samples         Mean   Mean error    Units
r.u.PerfTransforms.flatMapTransformsUsingFrom        1  thrpt         5  2421210.583    68845.966    ops/s
r.u.PerfTransforms.flatMapTransformsUsingFrom     1024  thrpt         5     1017.787       73.597    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust        1  thrpt         5  2398541.067    90703.197    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust     1024  thrpt         5      990.623      100.735    ops/s
r.u.PerfTransforms.mapTransformation                 1  thrpt         5  4020548.060   262841.500    ops/s
r.u.PerfTransforms.mapTransformation              1024  thrpt         5    16205.747      352.618    ops/s

Master

Benchmark                                       (size)   Mode   Samples         Mean   Mean error    Units
r.u.PerfTransforms.flatMapTransformsUsingFrom        1  thrpt         5  3184873.133   172320.420    ops/s
r.u.PerfTransforms.flatMapTransformsUsingFrom     1024  thrpt         5     9079.937      343.905    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust        1  thrpt         5  3411785.677    73767.161    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust     1024  thrpt         5    10860.963      294.309    ops/s
r.u.PerfTransforms.mapTransformation                 1  thrpt         5  7208334.997   703327.745    ops/s
r.u.PerfTransforms.mapTransformation              1024  thrpt         5    18720.797      278.529    ops/s

GC

On the master branch test I'm seeing GC results like this:

Iteration   5: 3189218.350 ops/s
          GC | wall time = 5.001 secs,  GC time = 0.047 secs, GC% = 0.94%, GC count = +98
Iteration   5: 9198.700 ops/s
          GC | wall time = 5.002 secs,  GC time = 0.048 secs, GC% = 0.96%, GC count = +98

versus 0.16

Iteration   5: 2420099.017 ops/s
          GC | wall time = 5.000 secs,  GC time = 0.046 secs, GC% = 0.92%, GC count = +96
Iteration   5: 993.867 ops/s
          GC | wall time = 5.001 secs,  GC time = 0.100 secs, GC% = 2.00%, GC count = +212

Summary

Unless I'm mistaken, current code is better:

• similar GC behavior on Observable with 1 item, better GC behavior with 1024 items
• higher ops/second on all of the map/flatMap tests

I'll start profiling this and improve ... but this does not reveal the source of the problems seen. Possibly it's related to schedulers, or it's a specific operator. I exercised map, flatMap (and thus merge), Observable and Subscribe here to get the most fundamental ones.

[benjchristensen]

The observeOn test shows the cost of scheduling ... particularly Observables of a single item.

    @GenerateMicroBenchmark
    public void observeOn(UseCaseInput input) throws InterruptedException {
        input.observable.observeOn(Schedulers.computation()).subscribe(input.observer);
        input.awaitCompletion();
    }

r.u.PerfObserveOn.observeOn        1  thrpt         5   457830.180     9943.221    ops/s
r.u.PerfObserveOn.observeOn     1024  thrpt         5    14496.430      518.463    ops/s

Thus, with an Observable of 1 item we can do 457k onNext/second, with an Observable of 1024 items we can do 14.8million onNext/second (14.5k * 1024).

[benjchristensen]

By the way, all testing is just being done on my Mac laptop ... so these numbers are all relative and not representative of proper server hardware.

[benjchristensen]

Converting from AtomicReference to AtomicReferenceFieldUpdater improved performance from this:

Benchmark                                       (size)   Mode   Samples         Mean   Mean error    Units
r.u.PerfTransforms.flatMapTransformsUsingFrom        1  thrpt         5  3184873.133   172320.420    ops/s
r.u.PerfTransforms.flatMapTransformsUsingFrom     1024  thrpt         5     9079.937      343.905    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust        1  thrpt         5  3411785.677    73767.161    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust     1024  thrpt         5    10860.963      294.309    ops/s
r.u.PerfTransforms.mapTransformation                 1  thrpt         5  7208334.997   703327.745    ops/s
r.u.PerfTransforms.mapTransformation              1024  thrpt         5    18720.797      278.529    ops/s

to this:

Benchmark                                       (size)   Mode   Samples         Mean   Mean error    Units
r.u.PerfTransforms.flatMapTransformsUsingFrom        1  thrpt         5  3459205.110   124790.906    ops/s
r.u.PerfTransforms.flatMapTransformsUsingFrom     1024  thrpt         5     9225.037      604.720    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust        1  thrpt         5  3639603.060   225599.038    ops/s
r.u.PerfTransforms.flatMapTransformsUsingJust     1024  thrpt         5    11135.613      337.022    ops/s
r.u.PerfTransforms.mapTransformation                 1  thrpt         5  7264202.633   214787.109    ops/s
r.u.PerfTransforms.mapTransformation              1024  thrpt         5    18795.790      713.668    ops/s