Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

security(logging): fix aes implementation in audit logging #8323

Merged
merged 66 commits into from
Mar 30, 2023
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
3f5fb30
push changes to branch
joshua-goldstein Sep 20, 2022
d2d06ca
update log_writer to fix aes issue
joshua-goldstein Sep 20, 2022
cc0f613
update docker-compose.yml in contrib/local-test to use encrypted audi…
joshua-goldstein Sep 20, 2022
14308ee
update run_ee.go to decrypt updated audit logs
joshua-goldstein Sep 20, 2022
77a3ca3
revert x/logger.go
joshua-goldstein Sep 20, 2022
b80723f
dgraph/dgraph:local was not getting tagged
joshua-goldstein Sep 20, 2022
f2721b0
revert test message in audit_ee.go
joshua-goldstein Sep 20, 2022
709ed5b
changes to log_writer.go
joshua-goldstein Sep 20, 2022
3fdd06b
modify run_ee.go
joshua-goldstein Sep 20, 2022
f075085
update log writer test
joshua-goldstein Sep 20, 2022
18ec9c3
update log writer test
joshua-goldstein Sep 20, 2022
d5a8081
revert changes to Makefile
joshua-goldstein Sep 20, 2022
d05989b
update comment
joshua-goldstein Sep 20, 2022
d6d81e1
rebase changes from main
joshua-goldstein Feb 23, 2023
9c25f5c
add old functions as deprecated
joshua-goldstein Feb 23, 2023
9e596ae
refactor
joshua-goldstein Feb 23, 2023
97cc6d6
minor comments
joshua-goldstein Feb 23, 2023
c9c53c1
remove todo
joshua-goldstein Feb 23, 2023
354db49
refactoring
joshua-goldstein Feb 23, 2023
ea7d8d7
add todo
joshua-goldstein Feb 23, 2023
2863841
test file
joshua-goldstein Feb 23, 2023
349e7dc
audit run fix
joshua-goldstein Feb 23, 2023
f95d0e2
formatting
joshua-goldstein Feb 23, 2023
22c875c
small fix
joshua-goldstein Feb 23, 2023
4e5d02a
cleanup
joshua-goldstein Feb 23, 2023
88e4ad3
final
joshua-goldstein Feb 23, 2023
e3abcec
final
joshua-goldstein Feb 23, 2023
59164b2
remove comment
joshua-goldstein Feb 23, 2023
af907ec
add new test packages to test encrypted audit logs
Feb 24, 2023
3b1874e
clean crufts
Feb 24, 2023
ef0e75b
change test name
Feb 24, 2023
e7fd984
remove enc files
Feb 24, 2023
2089043
final
Feb 24, 2023
18c0933
remove deprecated test
Feb 24, 2023
4c24657
clean up
joshua-goldstein Feb 25, 2023
4b1e033
cleanup and remove decrypt method from open (unused)
joshua-goldstein Feb 25, 2023
1c6bcb0
remove fatal error check
joshua-goldstein Feb 25, 2023
5efd7da
lint issues
joshua-goldstein Feb 25, 2023
68d2875
fix typo iv to ct
Feb 25, 2023
32cf0a0
add debugging for aman
Mar 2, 2023
d303f1e
dont remove zero audit logs after test
Mar 2, 2023
05a622f
aman's debugging
Mar 2, 2023
d3dd3c7
remove cruft
Mar 2, 2023
abe1206
dont panic if cannot read all audit log
Mar 15, 2023
50a00c9
make tests indepdendent
Mar 15, 2023
69b4b99
rebase on latest main
joshua-goldstein Mar 16, 2023
cf56c96
add error to glog
joshua-goldstein Mar 16, 2023
963ae87
make info a warning and fix typos
joshua-goldstein Mar 16, 2023
1bd2bd0
cleanup alpha and zero comands for test
joshua-goldstein Mar 16, 2023
b0bbaf1
use bytes read, and add breaks
Mar 17, 2023
eb275cd
remove comments
Mar 17, 2023
d227b13
add decrypt function to use io.readat
Mar 17, 2023
2f7cffa
add unit test for decrypt
Mar 17, 2023
a91c3f8
add more error message wrappers
Mar 20, 2023
01cd716
prepare truncate test
Mar 21, 2023
91bafdc
add truncate test
Mar 21, 2023
75a1070
add testutil/audit directory
Mar 21, 2023
a74dd58
use verifyLogs from testutil/audit
Mar 21, 2023
4a7e37f
rename testutil/audit to testutil/testaudit
Mar 21, 2023
b36a9b5
satisfy errcheck linter
Mar 21, 2023
cbc434c
more error checks
Mar 21, 2023
4b7c5ba
error check
Mar 21, 2023
6264fe1
cannot truncate higher than 790
Mar 21, 2023
15e6cba
nits
Mar 22, 2023
dc07b7c
use require
Mar 22, 2023
4e5ba7f
remove extra file
Mar 23, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 146 additions & 22 deletions ee/audit/run_ee.go
Original file line number Diff line number Diff line change
Expand Up @@ -107,37 +107,161 @@ func run() error {
glog.Info("audit file is empty")
return nil
}
var iterator int64 = 0

iv := make([]byte, aes.BlockSize)
x.Check2(file.ReadAt(iv, iterator))
iterator = iterator + aes.BlockSize
// decrypt header in audit log to verify encryption key
// [16]byte IV + [4]byte len(x.VerificationText) + [11]byte x.VerificationText
decryptHeader := func() ([]byte, int64, error) {
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
var iterator int64 = 0
iv := make([]byte, aes.BlockSize)
_, err := file.ReadAt(iv, iterator) // get first iv
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, 0, err
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
}
iterator = iterator + aes.BlockSize + 4 // length of verification text encoded in uint32

t := make([]byte, len(x.VerificationText))
x.Check2(file.ReadAt(t, iterator))
iterator = iterator + int64(len(x.VerificationText))
ct := make([]byte, len(x.VerificationText))
_, err = file.ReadAt(ct, iterator)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
return nil, 0, err
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
}
iterator = iterator + int64(len(x.VerificationText))

stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(t, t)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
if string(t) != x.VerificationText {
return errors.New("invalid encryption key provided. Please check your encryption key")
text := make([]byte, len(x.VerificationText))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(text, ct)
if string(text) != x.VerificationText {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we definer the VerificationText of type []byte and use bytes.Compare instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we declare a string constant in x/log_writer.go . I think this is reasonable since this should be a constant, and we can't declare an array/slice to be a constant. We could cast the x.VerificationText as a byte slice and compare the bytes but I don't think this makes a difference at all

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize that byte array/slice can't be declared as a constant. This article has a few ideas https://blog.boot.dev/golang/golang-constant-maps-slices/. I don't mind if we declare a function that returns the constant value. I think it is useful to avoid a allocation that converts byte slice to string.

return nil, 0, errors.New("invalid encryption key provided. Please check your encryption key")
}
return iv, iterator, nil
}

for {
// if its the end of data. finish decrypting
if iterator >= stat.Size() {
break
// [12]byte baseIV + [4]byte len(x.VerificationTextDeprecated) + [11]byte x.VerificationTextDeprecated
decryptHeaderDeprecated := func() ([]byte, int64, error) {
var iterator int64 = 0

iv := make([]byte, aes.BlockSize)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
_, err := file.ReadAt(iv, iterator)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can move this inside the if too

Copy link
Contributor Author

@joshua-goldstein joshua-goldstein Mar 17, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we now return the number of bytes written n and use this to increment the iterator

if err != nil {
return nil, 0, err
}
x.Check2(file.ReadAt(iv[12:], iterator))
iterator = iterator + 4
iterator = iterator + aes.BlockSize

content := make([]byte, binary.BigEndian.Uint32(iv[12:]))
x.Check2(file.ReadAt(content, iterator))
iterator = iterator + int64(binary.BigEndian.Uint32(iv[12:]))
ct := make([]byte, len(x.VerificationTextDeprecated))
_, err = file.ReadAt(ct, iterator)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we now return the number of bytes written n and use this to increment the iterator

if err != nil {
return nil, 0, err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more context in the error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, what kind of context? this will be propagated up and return the invalid encryption key error to the user

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added error wrapper

}
iterator = iterator + int64(len(x.VerificationTextDeprecated))

text := make([]byte, len(x.VerificationTextDeprecated))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(content, content)
x.Check2(outfile.Write(content))
stream.XORKeyStream(text, ct)
if string(text) != x.VerificationTextDeprecated {
return nil, 0, errors.New("invalid encryption key provided. Please check your encryption key")
}
return iv, iterator, nil
}

useDeprecated := false
iv, iterator, err := decryptHeader()
if err != nil {
// might have an old audit log
iv2, iterator2, err := decryptHeaderDeprecated()
if err != nil {
return errors.New("invalid encryption key provided. Please check your encryption key")
}
// found old audit log
useDeprecated = true
iv, iterator = iv2, iterator2
}

var count int

// encrypted writes each have the form below
// IV generated for each write
// #################################################################
// ##### [16]byte IV + [4]byte uint32(len(p)) + [:]byte p #####
// #################################################################
decryptBody := func() {
for {
count++
// if its the end of data. finish decrypting
if iterator >= stat.Size() {
break
}
n, err := file.ReadAt(iv, iterator)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
glog.Warningf("received %v while decrypting audit log\n", err)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
glog.Warningf("read %v bytes, expected %v\n", n, len(iv))
}
iterator = iterator + 16
length := make([]byte, 4)
n, err = file.ReadAt(length, iterator)
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
mangalaman93 marked this conversation as resolved.
Show resolved Hide resolved
glog.Warningf("received %v while decrypting audit log\n", err)
glog.Warningf("read %v bytes, expected %v\n", n, len(length))
}
iterator = iterator + 4

content := make([]byte, binary.BigEndian.Uint32(length))
n, err = file.ReadAt(content, iterator)
if err != nil {
glog.Warningf("received %v while decrypting audit log\n", err)
glog.Warningf("read %v bytes, expected %v\n", n, len(content))
}
iterator = iterator + int64(binary.BigEndian.Uint32(length))

stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(content, content)
n, err = outfile.Write(content)
if err != nil {
glog.Warningf("received %v while writing decrypted audit log\n", err)
glog.Warningf("wrote %v bytes, expected to write %v\n", n, len(content))
}
}
}

// encrypted writes in body have the form
// baseIV is constant, last 4 bytes vary
// ########################################################
// ##### [4]byte uint32(len(p)) + [:]byte p #####
// ########################################################
decryptBodyDeprecated := func() {
for {
// if its the end of data. finish decrypting
if iterator >= stat.Size() {
break
}
n, err := file.ReadAt(iv[12:], iterator)
if err != nil {
joshua-goldstein marked this conversation as resolved.
Show resolved Hide resolved
glog.Warningf("received %v while decrypting audit log\n", err)
glog.Warningf("read %v bytes, expected %v\n", n, len(iv[12:]))
}
iterator = iterator + 4

content := make([]byte, binary.BigEndian.Uint32(iv[12:]))
n, err = file.ReadAt(content, iterator)
if err != nil {
glog.Warningf("received %v while decrypting audit log\n", err)
glog.Warningf("read %v bytes, expected %v\n", n, len(content))
}
iterator = iterator + int64(binary.BigEndian.Uint32(iv[12:]))
stream := cipher.NewCTR(block, iv)
stream.XORKeyStream(content, content)
n, err = outfile.Write(content)
if err != nil {
glog.Warningf("received %v while writing decrypted audit log\n", err)
glog.Warningf("wrote %v bytes, expected to write %v\n", n, len(content))
}
}
}

if useDeprecated {
decryptBodyDeprecated()
} else {
decryptBody()
}

glog.Infof("Decryption of Audit file %s is Done. Decrypted file is %s",
decryptCmd.Conf.GetString("in"),
decryptCmd.Conf.GetString("out"))
Expand Down
Empty file.
Empty file.
Empty file.
Binary file not shown.
Binary file not shown.
Loading