Manage file key by the file to encrypt itself - incremental changes #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: There are some comments on subclasses in EncryptedEnv module which are duplicate to their parent classes, it would be nice to remove the duplication and keep the consistency if the comments on parent classes updated in someday. Pull Request resolved: facebook#11549 Reviewed By: akankshamahajan15 Differential Revision: D47007061 Pulled By: ajkr fbshipit-source-id: 8bfdaf9f2418a24ca951c30bb88e90ac861d9016
apache/incubator-pegasus#1575 Cherry-pick from tikv@113b363 Summary: Introduce `KeyManagedEncryptedEnv` which wraps around `EncryptedEnv` but provides an `KeyManager` API to enable key management per file. Also implements `AESBlockCipher` with OpenSSL. Test Plan: not tested yet. will update. Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@3d44a33 Summary: Instead of using openssl's raw `AES_encrypt` and `AES_decrypt` API, which is a low level call to encrypt or decrypt exact one block (16 bytes), we change to use the `EVP_*` API. The former is deprecated, and will use the default C implementation without AES-NI support. Also the EVP API is capable of handing CTR mode on its own. Test Plan: will add tests Signed-off-by: Yi Wu <yiwu@pingcap.com> --------- Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@2360562 Summary: Fix NewRandomRWFile and ReuseWritableFile misuse of `GetFile()` and `NewFile()`. See inline comments. Test Plan: manual test with tikv Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@93e89a5 fix bug: tikv/tikv#9115 Summary: we need to update encryption metadata via encryption::DataKeyManager, which cannot combine with the actual file operation into one atomic operation. In RenameFile, when the src_file has been removed, power is off, then we may lost the file info of src_file next restart. Signed-off-by: Xintao [hunterlxt@live.com](mailto:hunterlxt@live.com) Signed-off-by: Xintao <hunterlxt@live.com> Co-authored-by: Xintao <hunterlxt@live.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@bbd27cf used LinkFile instead of RenameFile api of key manager. But LinkFile needs check the dst file information, in RenameFile logic, we don't care about that. So just skip encryption for current file. Signed-off-by: Xintao [hunterlxt@live.com](mailto:hunterlxt@live.com)
acelyc111
force-pushed
the
my_enc6
branch
2 times, most recently
from
1f88779
to
66abf18
Compare
…cebook#9) apache/incubator-pegasus#1575 Cherry-pick from tikv@1868d12 Signed-off-by: Xintao <hunterlxt@live.com> Signed-off-by: tabokie <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@4cebfc1 * Add SM4-CTR encryption algorithm * Adjust block size for sm4 encryption * Add UT for SM4 encryption * Adjust macros indentation for sm4 * Fix format for adding sm4 Signed-off-by: Jarvis Zheng <jiayang@hust.edu.cn>
apache/incubator-pegasus#1575 Cherry-pick from tikv@9464766 In some env, user installed openssl by yum install, and the openssl software may compiled with OPENSSL_NO_SM4 flag, so although the version is >= 1.1.1, but we still could not use sm4 in that situation. Signed-off-by: Jarvis Zheng <jiayang@hust.edu.cn>
apache/incubator-pegasus#1575 Cherry-pick from tikv@acc624f * hook delete dir in encrypted env * add a comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@14f36f8 (without compaction related code) * fix renaming encrypted directory Signed-off-by: tabokie <xy.tao@outlook.com>
acelyc111
changed the title
acelyc111
added a commit
that referenced
this pull request
Aug 16, 2023
Summary: There are some comments on subclasses in EncryptedEnv module which are duplicate to their parent classes, it would be nice to remove the duplication and keep the consistency if the comments on parent classes updated in someday. Pull Request resolved: facebook#11549 Reviewed By: akankshamahajan15 Differential Revision: D47007061 Pulled By: ajkr fbshipit-source-id: 8bfdaf9f2418a24ca951c30bb88e90ac861d9016
acelyc111
added a commit
that referenced
this pull request
Aug 16, 2023
…book#15) Summary: There are some comments on subclasses in EncryptedEnv module which are duplicate to their parent classes, it would be nice to remove the duplication and keep the consistency if the comments on parent classes updated in someday. Pull Request resolved: facebook#11549 Reviewed By: akankshamahajan15 Differential Revision: D47007061 Pulled By: ajkr fbshipit-source-id: 8bfdaf9f2418a24ca951c30bb88e90ac861d9016
acelyc111
force-pushed
the
v8.3.2-pegasus-encrypt
branch
from
1d09030
to
05d042a
Compare
acelyc111
pushed a commit
that referenced
this pull request
Feb 27, 2024
Summary: I must have chosen trimming before frame 8 based on assertion failures, but that trims too many frame for a general segfault. So this changes to start printing at frame 4, as in this example where I've seeded a null deref: ``` Received signal 11 (Segmentation fault) Invoking LLDB for stack trace... Process 873208 stopped * thread #1, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007fb1fe8f1033 libc.so.6`__GI___wait4(pid=873478, stat_loc=0x00007fb1fb114030, options=0, usage=0x0000000000000000) at wait4.c:30:10 thread #2, name = 'rocksdb:low', stop reason = signal SIGSTOP frame #0: 0x00007fb1fe8972a1 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12 Executable module set to "/data/users/peterd/rocksdb/db_stress". Architecture set to: x86_64-unknown-linux-gnu. True frame #4: 0x00007fb1fe844540 libc.so.6`__restore_rt at libc_sigaction.c:13 frame #5: 0x0000000000608514 db_stress`rocksdb::StressTest::InitDb(rocksdb::SharedState*) at db_stress_test_base.cc:345:18 frame #6: 0x0000000000585d62 db_stress`rocksdb::RunStressTestImpl(rocksdb::SharedState*) at db_stress_driver.cc:84:17 frame facebook#7: 0x000000000058dd69 db_stress`rocksdb::RunStressTest(shared=0x00006120000001c0) at db_stress_driver.cc:266:34 frame facebook#8: 0x0000000000453b34 db_stress`rocksdb::db_stress_tool(int, char**) at db_stress_tool.cc:370:20 ... ``` Pull Request resolved: facebook#12101 Test Plan: manual (see above) Reviewed By: ajkr Differential Revision: D51593217 Pulled By: pdillinger fbshipit-source-id: 4a71eb8e516edbc32e682f9537bc77d073a7b4ed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.