Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle EKP Tenant Not Found Errors #9261

Merged
merged 39 commits into from
Feb 2, 2023
Merged

Handle EKP Tenant Not Found Errors #9261

merged 39 commits into from
Feb 2, 2023

Conversation

sfc-gh-nwijetunga
Copy link
Collaborator

@sfc-gh-nwijetunga sfc-gh-nwijetunga commented Jan 27, 2023 •
edited
Loading

It is possible for the EKP to throw tenant not found (key fetch) errors during the restore process. Ensure that we handle these errors to allow the restore to make progress (both snapshot and mutation log restores)

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

  • The PR has a description, explaining both the problem and the solution.
  • The description mentions which forms of testing were done and the testing seems reasonable.
  • Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

  • This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
  • There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

@sfc-gh-nwijetunga
inital
ba1fd49
@sfc-gh-nwijetunga
inital commit
1ad0d37
@sfc-gh-nwijetunga
mutation log restore with tenant deletion
3687dc2
@sfc-gh-nwijetunga
use tenant map instead of cache
accb2a5
@sfc-gh-nwijetunga
mutation log tenant deletion working
22b1e68
@sfc-gh-nwijetunga
Merge branch 'main' into nim/restore-optional-to-required
47b584b 
* main:
  remove client info encryption state (apple#9096)
  Fix /redwood/correctness/EnforceEncodingType unit test (apple#9095)
  Disconnection to satellite TLog should trigger recovery in gray failure detection
  Use const& for functor arguments
  Address review comments
  Update generated.go
  Also make database option visible
  Update design doc, and make AUTOMATIC_IDEMPOTENCY visible
  Use std::decay_t instead of std::remove_reference_t and std::remove_const_t
  Add support for flatMap in optional and erroror. Allow calling map with a lambda without specifying the return type template parameter.
  Fix example code
  Initial automatic idempotency docs
  Add overloads of the Optional::map and ErrorOr::map functions that allow you to more easily convert to a value's members. Add a similar mapRef function if the wrapped type is a pointer-type.
@sfc-gh-nwijetunga
fixes
658ebe6
@sfc-gh-nwijetunga
fix test
c5f8e6e
@sfc-gh-nwijetunga
address pr comments
29c918b
@sfc-gh-nwijetunga
modify
d5e23ff
@sfc-gh-nwijetunga
fixes
a4aedab
@sfc-gh-nwijetunga
inital commit
799364a
@sfc-gh-nwijetunga
merge main
450bd0c
@sfc-gh-nwijetunga
Merge branch 'nim/restore-optional-to-required' into nim/snapshot-ten…
f5dce13 
…ant-deletion

* nim/restore-optional-to-required: (40 commits)
  Use existing function to read database configuration
  Ignore `g_simulator` when testing on a real cluster
  More ss cf perf fixes main (apple#9109)
  The metacluster consistency check didn't account for the possibility that a partially applied operation could leave the set of tenant groups different between the management cluster and a data cluster. Also update metacluster consistency to use comparison based asserts, where appropriate.
  fix the no tenant check failure
  Trigger a commit if none happens within some amount of time when a tenant lookup is performed
  Change TLog pull async data warning timeout
  clearify the return type
  Blob Worker Encryption doesn't use BG_METADATA_SOURCE (apple#9121)
  Restart joshua
  Add tokensign dependency for Windows
  Trace data hall id in MachineMetrics events
  Add event for txn server initialization and a warning for TLog slow catching up
  fix assertion error
  check SetVersionstampedKey offset
  toml file format
  Resolver uses Encryption DB Config (apple#9002)
  Apply suggestions from code review
  Add data verification at the end of BlobRestoreBasic.toml
  Clean up cluster controller's wait on recoveredDiskFiles (apple#9105)
  ...
@sfc-gh-nwijetunga
tmp
cbf4f64
@sfc-gh-nwijetunga
workload modifications
d78a304
@sfc-gh-nwijetunga
address pr comments
3631b25
@sfc-gh-nwijetunga
snapshot tenant deletion support
2844e64
@sfc-gh-nwijetunga
merge main
4014e1e
@sfc-gh-nwijetunga
Trigger Build
7ec238d
@sfc-gh-nwijetunga
fix compile issues
8d48616
@sfc-gh-nwijetunga
temp commit
2e3f156
@sfc-gh-nwijetunga
handle ekp key not found
00c53aa
@sfc-gh-nwijetunga
changes
350a9ee
@sfc-gh-nwijetunga
revert
8e4aaa3
@sfc-gh-nwijetunga
fix bug in readwrite
c8afe9b
@sfc-gh-nwijetunga
merge main
591922a
@fdb-windows-ci
Copy link
Collaborator

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 591922a
  • Duration 0:16:47
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 591922a
  • Duration 0:50:54
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: bd17ba5
  • Duration 0:17:31
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: bd17ba5
  • Duration 0:40:32
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: bd17ba5
  • Duration 0:42:23
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: cebd70f
  • Duration 0:17:06
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

  • Commit ID: cebd70f
  • Duration 0:33:01
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: bd17ba5
  • Duration 1:20:40
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: cebd70f
  • Duration 0:40:56
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: cebd70f
  • Duration 0:43:40
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-macos on macOS Monterey 12.x

  • Commit ID: cebd70f
  • Duration 0:44:32
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@sfc-gh-nwijetunga sfc-gh-nwijetunga mentioned this pull request Feb 1, 2023
Merged
5 tasks
@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 977d8d2
  • Duration 0:20:17
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang-ide on Linux CentOS 7

  • Commit ID: 4c70382
  • Duration 0:17:23
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-macos on macOS Monterey 12.x

  • Commit ID: 977d8d2
  • Duration 0:44:59
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@fdb-windows-ci
Copy link
Collaborator

Doxense CI Report for Windows 10

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-macos-m1 on macOS Monterey 12.x

  • Commit ID: 4c70382
  • Duration 0:33:00
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 977d8d2
  • Duration 0:57:18
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 977d8d2
  • Duration 0:57:19
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-clang on Linux CentOS 7

  • Commit ID: 4c70382
  • Duration 0:36:42
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 4c70382
  • Duration 0:43:05
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

sfc-gh-ahusain
sfc-gh-ahusain approved these changes Feb 2, 2023
View reviewed changes
Copy link
Collaborator

@sfc-gh-ahusain sfc-gh-ahusain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 977d8d2
  • Duration 1:58:26
  • Result: ❌ FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: 4c70382
  • Duration 1:57:27
  • Result: ❌ FAILED
  • Error: Error while executing command: if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/fdb-kubernetes-tests/logs/*.log; exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

  • Commit ID: cebd70f
  • Duration 4:36:12
  • Result: ✅ SUCCEEDED
  • Error: N/A
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)
  • Cluster Test Logs zip file of the test logs (available for 30 days)

@sfc-gh-nwijetunga sfc-gh-nwijetunga merged commit 86f3665 into apple:main Feb 2, 2023
sfc-gh-nwijetunga added a commit to sfc-gh-nwijetunga/foundationdb that referenced this pull request Feb 2, 2023
@sfc-gh-nwijetunga
Merge branch 'main' into nim/tenant-backup-mutation-log-fix
4c0fa15 
* main: (23 commits)
  Handle EKP Tenant Not Found Errors (apple#9261)
  move feed cleanup check to after data is guaranteed to be available for granule (apple#9283)
  remove test timeout
  Reduce logging level for verbose events
  Added documentation for consistencyscan CLI command.
  Fix audit_storage issues (apple#9265)
  Update bindings/bindingtester/spec/tenantTester.md
  Update bindings/bindingtester/spec/tenantTester.md
  update bindingtester spec
  Fixing SkewedReadWrite to load its metadata in a transactionally consistent way (apple#9274)
  push string onto stack when active tenant is set
  Add comments on why custom encoding is needed
  patch to fix some existing bindingtester issues
  add arg and return type to the c_api for impl.py
  Fix includes
  Add from_7.0.0_until_7.2.0 for UpgradeAndBackupRestore tests
  Change UpgradeAndBackupRestore to from_7.2.4
  Add a new toml option to disable failure injection workload
  Change SubmitBackup to only reboot in Attrition
  add method to return idfuture
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-ahusain sfc-gh-ahusain sfc-gh-ahusain approved these changes

@sfc-gh-yiwu sfc-gh-yiwu Awaiting requested review from sfc-gh-yiwu

Assignees

@sfc-gh-ahusain sfc-gh-ahusain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sfc-gh-nwijetunga @fdb-windows-ci @foundationdb-ci @sfc-gh-ahusain