actual_lrps internal_routes field does not get re-encrypted when rotating bbs_encryption_key #626
Comments
|
邮件已收到，我会尽快阅读并回复，谢谢
|
|
Hello, As you can see, on the ActualLRP table, only the net_info is being reEncrypted. Yesterday I managed to reproduce the issue on a dev environment by rotating out of existence the key, that was used to encrypt the internal_routes column in the first place and today I added internal_routes to the Columns, cleared the diegodb from the broken records and rotated the keys again. This time the ActualLRP's endpoint worked fine and they were no broken records in the database. So yeah, seems like a simple fix, if I am not missing something and please correct me if I am. |
|
@sleepychild thank you for reporting this. Seems like a bug indeed. @klapkov this looks like the right place to make a fix. We need to make sure we also have a test for this. @klapkov would you be willing to make a PR for this fix? Please let me know if you have any more questions or need help. |
|
Hey @mariash, thanks for the response, I will make a PR with the fix and tests, but we will probably leave it for the next sprint, meaning in about 2 weeks. I hope that's okay with you. |
|
@klapkov thank you for this information. We consider this bug too important to wait for 2 weeks and we will be working on a fix. |
When encryption key is rotated internal_routes field was not getting re-encrypted and thus BBS was failing to read it from database. * Add a test to verify all text fields are being re-encrypted. cloudfoundry/diego-release#626
|
We pushed the fix, thank you for reporting this issue. |
|
@mariash , thank you for the quick response and fix. |
actual_lrps internal_routes field does not get re-encrypted when rotating bbs_encryption_key
Summary
Rotation of bbs_encryption_key makes the internal_routes field of the actual_lrps table unreadable.
Steps to Reproduce
Push an application. Rotate the bbs_encryption_key so that the one used to encrypt the internal_routes of your app is no longer in the encryption_keys list in the bbs config. Run
cfdot actual-lrpsand observe the following error.Error: BBS error Type 0: UnknownError Message: Key with label "{label of the key used to encrypt the internal_routes field when the lrp was created}" was not foundDiego repo
It's in the current bbs release. Probably introduced when internal_routes were introduced to the actual_lrps table.
cloudfoundry/bbs@2cfb94f
Environment Details
diego-release ~= 2.61.0 is where the issue manifested on our deployments
It could have been introduced prior to that.
Possible Causes or Fixes (optional)
None that I can think of.
Additional Text Output, Screenshots, contextual information (optional)
From the diegodb we take a list of actual-lrps that would be suspicious. They have crashed 200 times and are still CRASHED and have no cell_id. We order them from oldest to newest.
For all of the entries we execute
cfdot actual-lrps -p {process_guid}. Showing only the one before the two broken and the one after for brevity.diego-cell/859ab2d2-9b6a-4aa2-8a2b-c143a9576796:/var/vcap/bosh_ssh/bosh_aad697f4924f492# cfdot actual-lrps -p 68cc1eb8-1c02-4e09-8e97-d18d8d13522e-8b6941ae-c158-4c6f-99ed-ebfcfc0cb5ce {"process_guid":"68cc1eb8-1c02-4e09-8e97-d18d8d13522e-8b6941ae-c158-4c6f-99ed-ebfcfc0cb5ce","index":0,"domain":"cf-apps","instance_guid":"","cell_id":"","address":"","ports":null,"preferred_address":"UNKNOWN","crash_count":200,"crash_reason":"APP/PROC/WEB: Exited with status 1","state":"CRASHED","since":1649880643393329116,"modification_tag":{"epoch":"fcf272b5-8d11-4497-7c90-2bf4c842ee96","index":597},"presence":"ORDINARY"} diego-cell/859ab2d2-9b6a-4aa2-8a2b-c143a9576796:/var/vcap/bosh_ssh/bosh_aad697f4924f492# cfdot actual-lrps -p 27c91377-edbc-4a41-bbf4-139dbea989ec-0059b083-a987-411c-8c44-54528a7e8594 Error: BBS error Type 0: UnknownError Message: Key with label "bbs_encryption_key_2022T04b_0" was not found diego-cell/859ab2d2-9b6a-4aa2-8a2b-c143a9576796:/var/vcap/bosh_ssh/bosh_aad697f4924f492# cfdot actual-lrps -p b98f8918-1c10-49ea-9280-c609f9784f19-dcbb183c-75f5-4792-9f3e-f29781dab80c Error: BBS error Type 0: UnknownError Message: Key with label "bbs_encryption_key_2022T04b_0" was not found diego-cell/859ab2d2-9b6a-4aa2-8a2b-c143a9576796:/var/vcap/bosh_ssh/bosh_aad697f4924f492# cfdot actual-lrps -p a58dee1c-d65a-444f-95e1-4b29f40052dc-15526680-dfed-46c5-bff0-829c425fe724 {"process_guid":"a58dee1c-d65a-444f-95e1-4b29f40052dc-15526680-dfed-46c5-bff0-829c425fe724","index":0,"domain":"cf-apps","instance_guid":"","cell_id":"","address":"","ports":null,"preferred_address":"UNKNOWN","crash_count":200,"crash_reason":"APP/PROC/WEB: Exited with status 1","state":"CRASHED","since":1654805738286954714,"modification_tag":{"epoch":"47ab4452-ed34-49c8-5b7e-00c0ea0bf017","index":597},"presence":"ORDINARY"}As we can see the actual-lrps old enough to have their initial encryption keys are failing. The even older entries that don't have the
internal_routesfield populated and those whose encrypting key is still available in the bbs config work.The text was updated successfully, but these errors were encountered: