HBASE-25556 Frequent replication "Encountered a malformed edit" warnings #2965
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
apurtell
force-pushed
the
HBASE-25556
branch
2 times, most recently
from
c226007 to
5195981
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
busbey
reviewed
Feb 20, 2021
| @@ -400,22 +400,22 @@ protected boolean readNext(Entry entry) throws IOException { | |||
| initCause(realEofEx != null ? realEofEx : ex); | |||
| } | |||
| if (trailerPresent && this.inputStream.getPos() > this.walEditsStopOffset) { | |||
| LOG.error( | |||
There was a problem hiding this comment.
This particular message should only happen if the read was expected to succeed because the trailer presence means the wal was closed. Is this specific message showing up in the false alarm?
There was a problem hiding this comment.
No, this one did not
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
apurtell
commented
Feb 20, 2021
| @@ -424,7 +424,7 @@ protected boolean readNext(Entry entry) throws IOException { | |||
| } else { | |||
| // Else restore our position to original location in hope that next time through we will | |||
| // read successfully. | |||
| LOG.warn("Encountered a malformed edit, seeking back to last good position in file, " | |||
| LOG.debug("Encountered a malformed edit, seeking back to last good position in file, " | |||
There was a problem hiding this comment.
@busbey This is the frequent message. The particular EOFexception it prints originates from
There was a problem hiding this comment.
Yeah I agree that's not useful.
|
I'm +1 with the one log statement I commented on kept at error and the rest moved to debug. |
ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below.
apurtell
force-pushed
the
HBASE-25556
branch
from
5195981 to
b9a075f
Compare
|
🎊 +1 overall
This message was automatically generated. |
|
If I am reading Sean's comment correctly with the change I pushed we are good to go. Will merge in a few hours unless objection. |
|
💔 -1 overall
This message was automatically generated. |
asfgit
pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit
pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit
pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
asfgit
pushed a commit
that referenced
this pull request
Feb 23, 2021
…ngs (#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
|
🎊 +1 overall
This message was automatically generated. |
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
Feb 24, 2021
…ngs (apache#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org>
wchevreuil
pushed a commit
to wchevreuil/hbase
that referenced
this pull request
May 24, 2021
…ngs (apache#2965) ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as several per minute, and this will cause false positive assessment from operators that they are experiencing a bug. Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below. Signed-off-by: Sean Busbey <busbey@apache.org> (cherry picked from commit ecdc942) Change-Id: I5181c60fc8d97586e2be77909cd923df37c94325
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ProtobufLogReader#readNext may be called by code that attempts to advance the reader but does not necessarily expect to succeed, for example WALEntryStream#tryAdvanceEntry. Much of the logging in this method is at TRACE level. Other logging at WARN level will be frequently emitted, as often as a few times per minute, and this will cause false positive assessment from operators that they are experiencing a bug.
Fix the mixed intent with respect to log levels in readNext. Log at only DEBUG level or below.