Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle unknown runtime exception while reading entries #2993

Merged
merged 2 commits into from
Nov 16, 2018
Merged

Handle unknown runtime exception while reading entries #2993

merged 2 commits into from
Nov 16, 2018

Conversation

rdhabalia
Copy link
Contributor

Motivation

Recently we have seen below runtime-exception while reading entries at broker and broker is not handling it correctly and not completing read callback so, dispatch get stuck for the subscription.

22:16:58.424 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.notifyEntriesAvailable(ManagedCursorImpl.java:2221) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$notifyCursors$27(ManagedLedgerImpl.java:1580) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.7.2.jar:4.7.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.22.Final.jar:4.1.22.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

3:22:26.425 [bookkeeper-ml-workers-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.OpReadEntry.lambda$checkReadCompletion$76(OpReadEntry.java:136) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]


        06:58:57.334 [bookkeeper-ml-workers-OrderedExecutor-19-0] ERROR org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught 
java.lang.NullPointerException: null
        at org.apache.bookkeeper.mledger.impl.EntryImpl.create(EntryImpl.java:89) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.asyncReadEntry(EntryCacheImpl.java:225) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.internalReadFromLedger(ManagedLedgerImpl.java:1509) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.asyncReadEntries(ManagedLedgerImpl.java:1359) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.notifyEntriesAvailable(ManagedCursorImpl.java:2221) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$notifyCursors$27(ManagedLedgerImpl.java:1580) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) ~[managed-ledger-original-2.2.2-yahoo.jar:2.2.2-yahoo]
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [bookkeeper-common-4.7.2.jar:4.7.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pulsar-functions-metrics-2.2.2-yahoo.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Modifications

Handle NPE and complete the callback with failure.

Result

Dispatch will not be blocked if there is runtime exception while reading entries.

@rdhabalia rdhabalia added this to the 2.2.1 milestone Nov 15, 2018
@rdhabalia rdhabalia self-assigned this Nov 15, 2018
@rdhabalia rdhabalia requested a review from merlimat November 15, 2018 21:47
@merlimat merlimat added the type/bug The PR fixed a bug or issue reported a bug label Nov 15, 2018
@merlimat
Copy link
Contributor

The suspicion is that this might be due to the scenario described in netty/netty#8563

merlimat
merlimat approved these changes Nov 15, 2018
View reviewed changes
Copy link
Contributor

@merlimat merlimat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@merlimat merlimat mentioned this pull request Nov 16, 2018
Merged
@rdhabalia rdhabalia merged commit 2c8c288 into apache:master Nov 16, 2018
merlimat pushed a commit that referenced this pull request Dec 13, 2018
@rdhabalia @merlimat
Handle unknown runtime exception while reading entries (#2993)
61ce27d 
* Handle unknown runtime exception while reading entries

* make asyncReadEntry0 private
@lhotari lhotari mentioned this pull request May 3, 2021
Closed
@mattisonchao mattisonchao mentioned this pull request Nov 1, 2022
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@massakam massakam massakam approved these changes

@merlimat merlimat merlimat approved these changes

Assignees

@rdhabalia rdhabalia

Labels
area/broker type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
2.2.1
Development

Successfully merging this pull request may close these issues.
3 participants
@rdhabalia @merlimat @massakam