-
Notifications
You must be signed in to change notification settings - Fork 594
Improve concurrency for needed parts. #3107
Improve concurrency for needed parts. #3107
Conversation
That is the root cause of the exception? Nice find! |
I created a Bolt with MoultiThread (finagle-http) async and ack all at once. I think NPE is happening often. heron-0.18.7 I can not create a maven package file, so I can not test it in my project. |
I see. Make sense then when you use multi threading. In this case you might be better to synchronized {} to make sure ack() is not called many times at the same time. OutputCollector has a queue in it and I am not sure the queue is thread-safe. In Heron engine, the core assumes single thread environment. Internally, we build library and use it directly, instead of the ones in Maven. bazel build --compilation_mode=dbg --config=darwin scripts/packages:tarpkgs |
heron/api/src/java/org/apache/heron/api/metric/MultiCountMetric.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
otherwise lgtm.
heron/api/src/java/org/apache/heron/api/metric/CountStatAndMetric.java
Outdated
Show resolved
Hide resolved
heron/api/src/java/org/apache/heron/api/metric/LatencyStatAndMetric.java
Outdated
Show resolved
Hide resolved
storm-compatibility/src/java/org/apache/storm/metric/api/MultiCountMetric.java
Outdated
Show resolved
Hide resolved
storm-compatibility/src/java/org/apache/storm/metric/api/MultiReducedMetric.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see above comments. I think these changes should be made before merging this PR to master. Thanks in advance.
2198208
to
2e2702d
Compare
(cherry picked from commit 8d6d506)
Please review the PR. @nwangtw |
The changes after Ning's comments look fine to me. I'll leave it to @nwangtw to approve. |
Made the OutputCollector thread-safe.
works fine without it. |
@joshfischer1108 PR Check ^^ |
* master: Improve concurrency for needed parts. (apache#3107) Add documents for setting up a docker based development environment (apache#3475) Patch to fix cppcheck with newer glibc (apache#3471) Make log/sink/consume Streamlet component support setName and setNumPartitions (apache#3459)
* Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506)
…hinker0/heron into feature/support-custom-metrics-rules-for-prometheus-sink * 'feature/fix-prometheus-metrics' of https://github.com/thinker0/heron: Add Rules Update kafkaOffset metrics Improve concurrency for needed parts. (apache#3107) Add documents for setting up a docker based development environment (apache#3475) Patch to fix cppcheck with newer glibc (apache#3471) Make log/sink/consume Streamlet component support setName and setNumPartitions (apache#3459)
* Support Java 11 * config travis to use oracle jdk 11 * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * Add missing parameter * typo * Add pause time * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Update kafkaOffset metrics * Add Rules * Make log/sink/consume Streamlet component support setName and setNumPartitions (#3459) * Patch to fix cppcheck with newer glibc (#3471) * Add documents for setting up a docker based development environment (#3475) * Improve concurrency for needed parts. (#3107) * Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506) * Update kafkaOffset metrics * Add Rules * Update line is longer than 100 characters * Update line is longer than 100 characters * Add attrNameSnakeCase or other metrics fix * fix checkstyle Co-authored-by: Ning Wang <wangninggm@gmail.com> Co-authored-by: Ning Wang <nwang@twitter.com> Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
* Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506)
* Support Java 11 * config travis to use oracle jdk 11 * Java 11 support (apache#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * Add missing parameter * typo * Add pause time * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Java 11 support (apache#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Update kafkaOffset metrics * Add Rules * Make log/sink/consume Streamlet component support setName and setNumPartitions (apache#3459) * Patch to fix cppcheck with newer glibc (apache#3471) * Add documents for setting up a docker based development environment (apache#3475) * Improve concurrency for needed parts. (apache#3107) * Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506) * Update kafkaOffset metrics * Add Rules * Update line is longer than 100 characters * Update line is longer than 100 characters * Add attrNameSnakeCase or other metrics fix * fix checkstyle Co-authored-by: Ning Wang <wangninggm@gmail.com> Co-authored-by: Ning Wang <nwang@twitter.com> Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
* Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506)
* Support Java 11 * config travis to use oracle jdk 11 * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * Add missing parameter * typo * Add pause time * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Java 11 support (#3399) * Support Java 11 * config travis to use oracle jdk 11 * Add check jdk version * Fix command arguments. Change insert gc_options Update list Fix gc-logging * wip * Support jmx_exporter format configuration. * Fix checkstyle * Remove unused * Update kafkaOffset metrics * Add Rules * Make log/sink/consume Streamlet component support setName and setNumPartitions (#3459) * Patch to fix cppcheck with newer glibc (#3471) * Add documents for setting up a docker based development environment (#3475) * Improve concurrency for needed parts. (#3107) * Change concurrent Map * Change concurrent Map * HashMap changes for unneeded parts. * HashMap changes for unneeded parts. * Review changes * Changes HashMap for unneeded parts. * Improve concurrency for needed parts. * Remove unused imports. * Remove unused imports. * Remove unused imports. * Fix NPE (cherry picked from commit 545d381) * Fix WhitespaceAround * Add dummy Object * Fix ConstantName (cherry picked from commit 8d6d506) * Update kafkaOffset metrics * Add Rules * Update line is longer than 100 characters * Update line is longer than 100 characters * Add attrNameSnakeCase or other metrics fix * fix checkstyle Co-authored-by: Ning Wang <wangninggm@gmail.com> Co-authored-by: Ning Wang <nwang@twitter.com> Co-authored-by: Nicholas Nezis <nicholas.nezis@gmail.com>
Improve concurrency for needed parts.
#3104