-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Log entry checksum validation #123
Log entry checksum validation #123
Conversation
…e/SnapshotStorage etc. #77
冲突暂时不需要处理,等 #110 测试通过。 |
jraft-core/src/main/java/com/alipay/sofa/jraft/core/IteratorImpl.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/error/LogEntryCorrupteddException.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/util/CrcUtil.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/entity/LogEntry.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/entity/codec/v2/V2Encoder.java
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/entity/PeerId.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/entity/codec/v2/V2Decoder.java
Show resolved
Hide resolved
这个 PR 有点大,还没看完,我先歇会 |
jraft-core/src/main/java/com/alipay/sofa/jraft/core/IteratorImpl.java
Outdated
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/storage/impl/LogManagerImpl.java
Show resolved
Hide resolved
jraft-core/src/main/java/com/alipay/sofa/jraft/storage/impl/LogManagerImpl.java
Outdated
Show resolved
Hide resolved
@killme2008 看完了,提了几个意见你先看下 |
@fengjiachun 基本都改完了,你可以再看下,但是不着急合并。 V2 codec 这块性能还有改善的空间,我还是坚持用 protobuf,但是可以再想办法降低下损耗。 |
OK |
jraft-core/src/main/java/com/alipay/sofa/jraft/entity/PeerId.java
Outdated
Show resolved
Hide resolved
@fengjiachun 可以看下最新两个 commit ,优化后, 200 万次(并发20个线程)的 encode/decode 512 个字节数据的 log entry,三次结果为:
这个差距我觉的暂时是可以接受的,大小缩小在 1%~2% 之间, data 是完全随机的数据。 |
赞👍 节省了一次 toBytesArray 的 memory copy |
* (feat) optimize encodeV2, avoid some memory copy * (feat) typo and clear code * (feat) back to old code style with 'for' * (feat) back to old code style with 'for' * (feat) rename method
* (feat) optimize decode * (feat) optimize decode * (feat) rm unsafe dependency
这个先合并掉? @fengjiachun |
* (feat) Adds JRaftServiceFactory for custom services such as LogStorage/SnapshotStorage etc. #77 * (feat) update toString in NodeOptions * (feat) Supports custom log entry serialization, #106 (#111) * (feat) Adds JRaftServiceFactory for custom services such as LogStorage/SnapshotStorage etc. #77 * (feat) update toString in NodeOptions * (feat) Supports custom log entry serialization, #106 * (fix) testSaveFail * (feat) Adds serviceFactory to BootstrapOptions and fix tests * (fix) by code review comments * Log entry checksum validation (#123) * (feat) Adds JRaftServiceFactory for custom services such as LogStorage/SnapshotStorage etc. #77 * (feat) update toString in NodeOptions * (feat) Supports custom log entry serialization, #106 * (fix) testSaveFail * (feat) Adds serviceFactory to BootstrapOptions and fix tests * (fix) by code review comments * (feat) Impl log entry checksum * (feat) remove RaftGroupService default constructor * (feat) Checksum in iterator and readUserCommitLog * (feat) Refactor old version 1 log entry codec * (feat) Impl v2 codec factory supports checksum * (feat) format code * (feat) Adds reserved flag * (feat) Adds test for v2 codec * (feat) Adds log entry checksum test * (fix) header comments * (fix) comment * (fix) by code review comments * (feat) tweak v2 codec performance * (feat) Use AsciiStringUtil.unsafeEncode replacs Utils.getBytes in peer checksum * (feat) Use ThreadLocal to replace FastThreadLocal * (feat) forgot CrcUtilTest * (feat) remove unnecessary method handler (#124) * Feat/optimize encode (#126) * (feat) optimize encodeV2, avoid some memory copy * (feat) typo and clear code * (feat) back to old code style with 'for' * (feat) back to old code style with 'for' * (feat) rename method * feat/optimize v2 decode (#127) * (feat) optimize decode * (feat) optimize decode * (feat) rm unsafe dependency
Motivation:
LogEntry doesn't have any data checksum validation right now, it's a data integrity risk here. We should check log data's integrity after reading it from network or disk.
Modification:
LogEntryV2CodecFactory
that encoding/decoding log entry by protobuf, and it adds a header to the message:Test the performance between v1 and v2 codec in
LogEntryCodecPerfTest
, do about 200k times to encoding/decoding log entry in 512 bytes:The encoded log size is reduced about 1%~2% but cost more time, i think it's acceptable.
LogEntry
:hasChecksum()
return true when has a checksum.checksum()
caculate the checksum of the log entry.isCorrupted()
returns true when the log entry has checksum and mismatched.The checksum is caculated by CRC64 and xor.
The
LogEntryV2CodecFactory
usesAutoDetectDecoder
to detect the codec version automatically and chooses the right decoder to decode the log entry. We keep log codec compatibility in such way.Adds checksum validation in
NodeImpl#handleAppendEntriesRequest
,LogManagerImpl#getEntry
andLogManagerImpl#getTermFromLogStorage
, i introduce a new raft optionRaftOptions#enableLogEntryChecksum
to control the behaviour, it's false(disabled) by default.Deprecate
LogEntry#encode
andLogEntry#decode
, you should useLogEntryEncoder
andLogEntryDecoder
instead. Also , deprecateLogStorage#getTerm(index)
,it can be replaced bygetEntry(index).getId().getTerm()
, because it don't want to do checksum validation in log storage but do it all inLogManagerImpl
.Result:
Fixes #105