Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

security: Encrypt region boundary keys, Part 1 - helper functions #2931

Merged
merged 22 commits into from
Sep 16, 2020
Merged

security: Encrypt region boundary keys, Part 1 - helper functions #2931

merged 22 commits into from
Sep 16, 2020

Conversation

yiwu-arbug
Copy link

Signed-off-by: Yi Wu yiwu@pingcap.com

What problem does this PR solve?

This is part 1 for adding TDE support to PD. pingcap/tidb#18262 It contains helper methods for encryption.

What is changed and how it works?

This PR adds utility types and functions:

  • encrypt/decrypt data using aes-ctr (for encrypting region boundary keys) and aes-gcm (for encrypting data encryption keys)
  • helper methods to encrypt/decrypt region boundary keys in-place
  • wrapper of master key, and helper method to read a master key (must be a 256 bit encryption key, stored as hex-string) from a local file

Check List

Tests

  • Unit test

Related changes

Release note

  • No release note

Encrypt region boundary keys, part 1
5d4aa1f 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
yiwu-arbug
yiwu-arbug commented Sep 9, 2020
View reviewed changes
go.mod Outdated Show resolved Hide resolved
@yiwu-arbug yiwu-arbug changed the title Encrypt region boundary keys, part 1 Encrypt region boundary keys, Part 1 Sep 9, 2020
Yi Wu added 3 commits September 9, 2020 12:30
update kvproto
28fb6d4 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
Merge branch 'master' into enc_util
c8d8f30
fix lint
1176ea9 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
@nolouch nolouch added the component/security Security logic. label Sep 9, 2020
HunDunDM
HunDunDM reviewed Sep 9, 2020
View reviewed changes
go.mod Outdated Show resolved Hide resolved
Yisaer
Yisaer reviewed Sep 9, 2020
View reviewed changes
pkg/encryption/region_crypter_test.go Outdated Show resolved Hide resolved
pkg/encryption/region_crypter.go Outdated Show resolved Hide resolved
pkg/encryption/region_crypter.go Show resolved Hide resolved
pkg/encryption/master_key_test.go Show resolved Hide resolved
pkg/encryption/master_key_test.go Show resolved Hide resolved
pkg/encryption/crypter.go Outdated Show resolved Hide resolved
Yi Wu added 6 commits September 15, 2020 12:18
Merge branch 'master' into enc_util
bff9c8c
update kvproto
913d654 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
fix lint
a315f3b 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
remove IsEncrypted field
cc42c9e 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
address comments
9e6f0d3 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
add errno
e25f99a 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
@yiwu-arbug
Copy link
Author

comments addressed or answered. PTAL.

@yiwu-arbug yiwu-arbug changed the title Encrypt region boundary keys, Part 1 Encrypt region boundary keys, Part 1 - helper functions Sep 15, 2020
Yi Wu added 9 commits September 15, 2020 14:07
fix lint
eca7afa 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
fix lint
2b4592f 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
Merge branch 'master' into enc_util
0693249
fix lint
02b7b4a 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
update kvproto
1c150fb 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
fix lint
f651283 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
Merge branch 'master' into enc_util
0fb074e
go mod tidy
8651f07 
Signed-off-by: Yi Wu <yiwu@pingcap.com>
Merge branch 'master' into enc_util
5d9b209
Yisaer
Yisaer approved these changes Sep 16, 2020
View reviewed changes
Copy link
Contributor

@Yisaer Yisaer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-srebot ti-srebot added the status/LGT1 Indicates that a PR has LGTM 1. label Sep 16, 2020
@yiwu-arbug yiwu-arbug merged commit 0a15643 into tikv:master Sep 16, 2020
@yiwu-arbug yiwu-arbug deleted the enc_util branch September 16, 2020 21:29
@yiwu-arbug yiwu-arbug changed the title Encrypt region boundary keys, Part 1 - helper functions security: Encrypt region boundary keys, Part 1 - helper functions Sep 16, 2020
@yiwu-arbug yiwu-arbug mentioned this pull request Sep 16, 2020
Merged
HunDunDM
HunDunDM reviewed Sep 17, 2020
View reviewed changes
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/crypter.go Show resolved Hide resolved
pkg/encryption/master_key.go Show resolved Hide resolved
pkg/encryption/master_key.go Show resolved Hide resolved
@yiwu-arbug yiwu-arbug mentioned this pull request Oct 12, 2020
Merged
@yiwu-arbug yiwu-arbug mentioned this pull request Nov 3, 2020
Merged
ti-chi-bot pushed a commit that referenced this pull request Nov 13, 2020
security: Encrypt region boundary keys, Part 4 - KMS (#3141)
3c62563 
* Encrypt region boundary keys, Part 2 - server changes

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* update errno and config template

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix typo

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix tests

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix tests

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix tests

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix tests

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* address comment in #2931

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix loadRegion

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* rename encryption_key_manager package

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix lint

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix lint

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix comments

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix comment

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* use option pattern

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* move loadRegion and saveRegion

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* revert changes

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix doc

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* address comment

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* key manager

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* add test and refactor

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* make EncryptionKeysPath a const

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix region_crypter key manager nil check

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* save conflict test

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* sanity check keys revision

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* kms

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* test set ciphertextKey

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* test set ciphertextKey

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* clone region only when needed

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* add test for config

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* update errors

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix lint

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix lint

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* make errdoc

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* address comment

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix error type

Signed-off-by: Yi Wu <yiwu@pingcap.com>

* fix test

Signed-off-by: Yi Wu <yiwu@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HunDunDM HunDunDM HunDunDM left review comments

@Yisaer Yisaer Yisaer approved these changes

@nolouch nolouch Awaiting requested review from nolouch

Assignees
No one assigned
Labels
component/security Security logic. status/LGT1 Indicates that a PR has LGTM 1.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@yiwu-arbug @Yisaer @HunDunDM @nolouch @ti-srebot