STORM-1038: Upgrade netty to 4.x in 1.x-branch

[ooasis]

This is to add the feature to 1.x-branch. The original PR for master branch is #728.

[ooasis]

Performance test results are attached here.
perf_compare_netty_3vs4_1.x-branch.zip

Baseline: current head of 1.x-branch (e55684b)
Netty4: 57fbccc

In addition to default configuration,
io.netty.noPreferDirect=false
io.netty.allocator.type=pooled

following configurations are also tested for netty 4.x

-unpool
io.netty.allocator.type=unpooled. Ask Netty to not use pooled allocation for buffer

-nodirect
io.netty.noPreferDirect=true. Ask Netty to not use off-heap buffer allocation

-unpool-nodirect
io.netty.noPreferDirect=true
io.netty.allocator.type=unpooled

Due to the capacity of my laptop, I tested following loads: 2k, 5k, 10k. At 10k, the CPU usage reaches above 85% and failures started to appear at startup.

The performance data seems to suggest

• no big difference is observed comparing before/after netty upgrades
  I suspect only a small percentage of traffic will cross worker instances (jvm) so Netty does not play a big role in performance measured
• netty 4.x consumes a little more CPU and memory
  The small memory increment may be simply b/c netty 4.x pulled in more classes.
• netty 4.x has lower CPU consumption on GC
  This is especially visible with the default configuration where off-heap buffer allocation is used.

[ooasis]

Seems the Travis CI build will fail at random places which are not related to this PR.

For example the last build failed due to

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.2.1:process (default) on project flux-examples: Error resolving project artifact: Could not transfer artifact io.confluent:kafka-schema-registry-client:pom:1.0 from/to sonatype-apache (https://repository.apache.org/releases/): Connect to repository.apache.org:443 [repository.apache.org/207.244.88.143] failed: Connection timed out for project io.confluent:kafka-schema-registry-client:jar:1.0 -> [Help 1]

Any way to manually trigger a new CI build w/o code change?

-thanks

[HeartSaVioR]

You can close and reopen this PR to retrigger. Btw, apache repository seems to be unstable recent days. I saw other build failures due to repository connectivity issue.

[satishd]

Why is it changed to use writeAndFlush?

[satishd]

@hsun-cnnxty Channel#write is modified to Channel#writeAndFlush at multiple places. Does not that flush to the underlying stream without buffering and flush it when buffers are full?

[ooasis]

@satishd nett 4.x has removed auto-flush in write() and it requires user to explicitly call flush() or writeAndFlush(), so we need to flush for the last message in the logic flow. Fortunately, the storm netty client does batching of messages itself and it does not flush for every message.

[knusbaum]

These config descriptions are not descriptive. Without reading the code, I have no idea what this does.

[ooasis]

Will take a look at how to improve. Any suggestions?

-thanks

[HeartSaVioR]

Since Netty made 3.x version line EOL, I'd like to bump Netty to 4.0.x or even 4.1.x if there's no performance / resource usage issue.

@hsun-cnnxty
Please upmerge this. If you really don't mind, could you do performance tests again on two latest versions: 4.0.41.Final and 4.1.5.Final, too?

[ooasis]

I will find some time this weekend to work on it.

-thanks

[HeartSaVioR]

Please rebase and force push accordingly. Your PR now has all of the commits from other side.

You can give it an another try: 1) create new branch based on recent master or 1.x 2) cherry-pick your commits 3) checkout to your PR branch 4) reset your PR branch to the branch which you just worked 4) force push.

[ooasis]

@HeartSaVioR
I rebased to 1.x-branch and also squashed the commits. Will attached the performance test results soon.

[ooasis]

Tests are done to compare following two code bases:

• baseline: current head of 1.x-branch (62476f5)
• netty 4.1.x: 1.x-branch-netty4 (dd96d9f)

In summary:

• netty 4.1.x is able to sustain higher throughput (20k/sec) than baseline (between 15k -20k/sec)
• netty 4.1.x has lower GC activity which could be the reason that it has higher throughput**

Also tested different netty configurations regarding buffer allocation

• use direct buffer or heap
• use pooled or unpooled

Result shows the default setting works best.

Performance tests are done on my laptop with following config:

Model Name: MacBook Pro
Processor Name: Intel Core i7
Processor Speed: 2.3 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 6 MB
Memory: 16 GB
System Version: OS X 10.10.5 (14F27)
Kernel Version: Darwin 14.5.0

The data collected is attached.
performance_compare_netty_1.x-branch.zip

[HeartSaVioR]

@hsun-cnnxty
Sorry I lost tracking this.
From the last time I tested on my local machine (via ThroughputVsLatency), before patching this works better than after patching this. My test environment seems not stable.
Someone might need to test on this, and if he/she can run benchmark with multi machines that should be great.