[ALL CHANGES] Fix enc #5
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Do not run actions on `push` to save resources. - Skip to run `build-macos-java-static` and `build-macos-java-static-universal` jobs because they will cause link errors when OpenSSL introduced and they are not the Pegasus use cases. We can resolve the issue later. - Fix the error flags used in `build-linux-encrypted_env-no_compression`.
apache/incubator-pegasus#1575 Cherry-pick from tikv@113b363 Summary: Introduce `KeyManagedEncryptedEnv` which wraps around `EncryptedEnv` but provides an `KeyManager` API to enable key management per file. Also implements `AESBlockCipher` with OpenSSL. Test Plan: not tested yet. will update. Signed-off-by: Yi Wu <yiwu@pingcap.com> Signed-off-by: tabokie <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@3d44a33 Summary: Instead of using openssl's raw `AES_encrypt` and `AES_decrypt` API, which is a low level call to encrypt or decrypt exact one block (16 bytes), we change to use the `EVP_*` API. The former is deprecated, and will use the default C implementation without AES-NI support. Also the EVP API is capable of handing CTR mode on its own. Test Plan: will add tests Signed-off-by: Yi Wu <yiwu@pingcap.com> --------- Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@2360562 Summary: Fix NewRandomRWFile and ReuseWritableFile misuse of `GetFile()` and `NewFile()`. See inline comments. Test Plan: manual test with tikv Signed-off-by: Yi Wu <yiwu@pingcap.com> Co-authored-by: yiwu-arbug <yiwu@pingcap.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@93e89a5 fix bug: tikv/tikv#9115 Summary: we need to update encryption metadata via encryption::DataKeyManager, which cannot combine with the actual file operation into one atomic operation. In RenameFile, when the src_file has been removed, power is off, then we may lost the file info of src_file next restart. Signed-off-by: Xintao [hunterlxt@live.com](mailto:hunterlxt@live.com) Signed-off-by: Xintao <hunterlxt@live.com> Co-authored-by: Xintao <hunterlxt@live.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@bbd27cf used LinkFile instead of RenameFile api of key manager. But LinkFile needs check the dst file information, in RenameFile logic, we don't care about that. So just skip encryption for current file. Signed-off-by: Xintao [hunterlxt@live.com](mailto:hunterlxt@live.com)
…cebook#9) apache/incubator-pegasus#1575 Cherry-pick from tikv@1868d12 Signed-off-by: Xintao <hunterlxt@live.com> Signed-off-by: tabokie <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@4cebfc1 * Add SM4-CTR encryption algorithm * Adjust block size for sm4 encryption * Add UT for SM4 encryption * Adjust macros indentation for sm4 * Fix format for adding sm4 Signed-off-by: Jarvis Zheng <jiayang@hust.edu.cn>
apache/incubator-pegasus#1575 Cherry-pick from tikv@9464766 In some env, user installed openssl by yum install, and the openssl software may compiled with OPENSSL_NO_SM4 flag, so although the version is >= 1.1.1, but we still could not use sm4 in that situation. Signed-off-by: Jarvis Zheng <jiayang@hust.edu.cn>
apache/incubator-pegasus#1575 Cherry-pick from tikv@acc624f * hook delete dir in encrypted env * add a comment Signed-off-by: tabokie <xy.tao@outlook.com> Co-authored-by: Xinye Tao <xy.tao@outlook.com>
apache/incubator-pegasus#1575 Cherry-pick from tikv@14f36f8 (without compaction related code) * fix renaming encrypted directory Signed-off-by: tabokie <xy.tao@outlook.com>
apache/incubator-pegasus#1575 After all encryption related patches been cherry-picked from [tikv](https://github.com/tikv/rocksdb/commits/6.29.tikv) and merged, now we will improve the encrytion, including: - Fix action job `build-linux-encrypted_env-no_compression-no_openssl` to build binaries without openssl and compression libs correctly. - Fix action job `build-linux-encrypted_env-openssl` to export the `ENCRYPTED_ENV` enviroment variable correctly. - Don not skip tests which are skipped by TiKV. - Refactor `AESCTRCipherStream` and `AESEncryptionProvider` to support manage file key by the file itself, according to the design docs in [Data at rest encryption](apache/incubator-pegasus#1575). - Remove all KeyManager related codes. - Replace KeyManager tests by AES encryption tests. - Refactor encryption/encryption_test.cc and add more tests. - Make it possible to construct AESEncryptionProvider object via `EncryptionProvider::CreateFromString()` by registering a factory in "encryption" library. It's possible to construct an object by URI: `AES`, `AES://test` or `AES:<instance_key>,<EncryptionMethod>`. - `ldb` tool support to parse `--fs_uri` flags as the URI mentioned above. - Add tests to create AESEncryptionProvider object in `CreateEncryptedEnvTest.CreateEncryptedFileSystem` - `db_bench` support to run benchmark with encryption enabled, by adding new flags for `db_bench`, they are `encryption_method` and `encryption_instance_key`. - Move code from the exported header directory (i.e. include/rocksdb/encryption.h) to rocksdb internal (i.e. encryption/encryption.h), do not expose them to users. - Code format. Review hint: pegasus-kv#17 shows all the code changes from the base branch (i.e. `pegasus-kv:v8.3.2-pegasus`), you can review it together to make sure the request branch `acelyc111:pk_enc_new` doesn't have vice effect on the base. Manual test: ``` // Generate some data. ./db_bench --encryption_method=AES128CTR --encryption_instance_key=test_instance_key --num=10000 // Dump WAL OK ./tools/ldb --fs_uri="provider=AES; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES://test; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log // Dump WAL failed. Pass bad provider parameters to --fs_uri, e.g. ./tools/ldb --fs_uri="provider=AES1:test_instance_key,1AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:bad_test_instance_key,AES128CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log ./tools/ldb --fs_uri="provider=AES:test_instance_key,AES192CTR; id=EncryptedFileSystem" dump_wal --walfile=/tmp/rocksdbtest-1000/dbbench/000004.log // The same to other ldb tools. ```
apache/incubator-pegasus#1575 1. Update the status badge to pegasus-kv/rocksdb's own site. 2. Also aim to check whether all tests could pass after cherry-picking encryption related patches to 8.5.3 branch.
Return a pointer rather than Status for GetEVPCipher() when SM4 not supported.
acelyc111
pushed a commit
that referenced
this pull request
Feb 27, 2024
Summary: I must have chosen trimming before frame 8 based on assertion failures, but that trims too many frame for a general segfault. So this changes to start printing at frame 4, as in this example where I've seeded a null deref: ``` Received signal 11 (Segmentation fault) Invoking LLDB for stack trace... Process 873208 stopped * thread #1, name = 'db_stress', stop reason = signal SIGSTOP frame #0: 0x00007fb1fe8f1033 libc.so.6`__GI___wait4(pid=873478, stat_loc=0x00007fb1fb114030, options=0, usage=0x0000000000000000) at wait4.c:30:10 thread #2, name = 'rocksdb:low', stop reason = signal SIGSTOP frame #0: 0x00007fb1fe8972a1 libc.so.6`__GI___futex_abstimed_wait_cancelable64 at futex-internal.c:57:12 Executable module set to "/data/users/peterd/rocksdb/db_stress". Architecture set to: x86_64-unknown-linux-gnu. True frame #4: 0x00007fb1fe844540 libc.so.6`__restore_rt at libc_sigaction.c:13 frame #5: 0x0000000000608514 db_stress`rocksdb::StressTest::InitDb(rocksdb::SharedState*) at db_stress_test_base.cc:345:18 frame #6: 0x0000000000585d62 db_stress`rocksdb::RunStressTestImpl(rocksdb::SharedState*) at db_stress_driver.cc:84:17 frame facebook#7: 0x000000000058dd69 db_stress`rocksdb::RunStressTest(shared=0x00006120000001c0) at db_stress_driver.cc:266:34 frame facebook#8: 0x0000000000453b34 db_stress`rocksdb::db_stress_tool(int, char**) at db_stress_tool.cc:370:20 ... ``` Pull Request resolved: facebook#12101 Test Plan: manual (see above) Reviewed By: ajkr Differential Revision: D51593217 Pulled By: pdillinger fbshipit-source-id: 4a71eb8e516edbc32e682f9537bc77d073a7b4ed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.