HBASE-29495: Integrate key management with existing encryption #7297

haridsv · 2025-09-12T05:22:32Z

This PR addresses HBASE-29495 to allow the new managed keys being detected and used by the existing encryption

Squashing the below changes for the convenience of rebasing: 2f6c381a71 Write path integration 9befa74658 Provision to instantiate ManagedKeyAccessor without server for the sake of tools 81540a0ca8 Test coverage with L2 disabled a16313d1c1 Improved test coverage 332f4e756b null check for server on write path

…extensively

…po in shell command format and a few renames

- Also includes a couple of fixes for typos - Refactored some of the key management test code so that it is reusable from a shell test.

…y search in global namespace - Also fixes an issue with point lookups for active keys.

…gion operations - The approach involves creating a new KeyManagementSerice interface and passing it along the Server in as many code paths as possible. - Also improved the assertions in the encryption E2E test.

Also added test coverage to read the HFile.

Also reverted some prior changes and added overloaded methods to reduce impact on tests following existing pattern.

…ation

…e dropped With mockito-inline enabled, at least the below 2 tests are failing: - org.apache.hadoop.hbase.TestHBaseTestingUtil Could not initialize plugin: interface org.mockito.plugins.MockMaker (alternate: null) - org.apache.hadoop.hbase.security.access.TestRpcAccessChecks java.lang.NullPointerException: Cannot invoke "org.apache.hbase.thirdparty.com.google.protobuf.Descriptors$ServiceDescriptor.getFullName()" because "service" is null at org.apache.hadoop.hbase.ipc.CoprocessorRpcUtils.getServiceName(CoprocessorRpcUtils.java:77) at org.apache.hadoop.hbase.master.HMaster.registerService(HMaster.java:3479) The later is especially weird. The NPE should be ocuring even now as the mock is not stubbed, but it is somehow working.

haridsv · 2025-09-22T04:10:28Z

Ran the tests locally using the mvn command from nightly master and got a clean run:

$ time mvn --batch-mode --threads=2 -DHBasePatchProcess -Dhadoop-three.version=3.3.5 -PrunAllTests -Dtest.exclude.pattern=**/snapshot.TestMobSecureExportSnapshot.java,**/quotas.TestDefaultHandlerUsageQuota.java,**/regionserver.TestCompactionState.java,**/master.TestMasterRegionMutation1.java,**/replication.TestSyncReplicationStandbyAsyncFSWAL.java,**/snapshot.TestSnapshotClientRetries.java,**/master.procedure.TestSCPWithMetaWithReplicas.java,**/util.TestProcDispatcher.java,**/security.TestMutualTlsServerSide.java,**/client.TestMobCloneSnapshotFromClientAfterSplittingRegion.java,**/backup.TestIncrementalBackup.java,**/master.procedure.TestHBCKSCP.java,**/regionserver.wal.TestLogRolling.java,**/master.balancer.TestStochasticLoadBalancerRegionReplicaHighReplication.java,**/client.TestAdminShell2.java,**/regionserver.TestSplitTransactionOnCluster.java,**/regionserver.TestRegionServerReportForDuty.java,**/namequeues.TestTooLargeLog.java,**/io.hfile.bucket.TestPrefetchPersistence.java,**/replication.TestNamespaceReplication.java,**/master.TestDLSFSHLog.java,**/master.assignment.TestRollbackSCP.java,**/replication.regionserver.TestBasicWALEntryStreamFSHLog.java,**/replication.TestBidirectionSerialReplicationStuck.java,**/io.TestFileChangeWatcher.java,**/snapshot.TestRestoreSnapshotHelper.java,**/io.hfile.bucket.TestBucketCache.java,**/client.TestBufferedMutator.java,**/regionserver.TestFailedAppendAndSync.java,**/regionserver.wal.TestHBaseWalOnEC.java,**/monitoring.TestTaskMonitor.java,**/quotas.TestClusterScopeQuotaThrottle.java,**/replication.regionserver.TestMetaRegionReplicaReplication.java,**/security.TestNettyTLSIPCFileWatcher.java,**/regionserver.wal.TestFSHLog.java,**/regionserver.TestTags.java,**/rsgroup.TestRSGroupsAdmin1.java,**/master.region.TestMasterRegionCompaction.java,**/namequeues.TestWALEventTracker.java,**/snapshot.TestSecureExportSnapshot.java,**/regionserver.TestBytesReadServerSideScanMetrics.java,**/master.region.TestMasterRegionWALCleaner.java,**/io.hfile.bucket.TestRecoveryPersistentBucketCache.java,**/io.asyncfs.TestSaslFanOutOneBlockAsyncDFSOutput.java,**/security.access.TestZKPermissionWatcher.java,**/quotas.TestThreadHandlerUsageQuota.java,**/io.hfile.TestPrefetchRSClose.java,**/security.access.TestAccessController.java,**/client.TestClientScannerTimeouts.java,**/replication.multiwal.TestReplicationEndpointWithMultipleAsyncWAL.java,**/io.hfile.bucket.TestBucketCachePersister.java,**/master.balancer.TestBalancerDecision.java,**/master.procedure.TestSnapshotProcedureRSCrashes.java clean test -fae -Dsurefire.testFailureIgnore=true -Dmaven.test.failure.ignore=true
...
[INFO] Reactor Summary for Apache HBase 4.0.0-alpha-1-SNAPSHOT:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [  1.116 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [  0.210 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [  0.629 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [  0.088 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [  0.774 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 24.792 s]
[INFO] Apache HBase - Common .............................. SUCCESS [01:54 min]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [  3.042 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [  6.786 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 17.231 s]
[INFO] Apache HBase - Client .............................. SUCCESS [01:08 min]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 27.474 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [  7.986 s]
[INFO] Apache HBase - Balancer ............................ SUCCESS [12:07 min]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [  0.101 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 30.334 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 28.486 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 59.235 s]
[INFO] Apache HBase - Server .............................. SUCCESS [  03:07 h]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [22:14 min]
[INFO] Apache HBase - Diagnostics Tools ................... SUCCESS [03:17 min]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [01:19 min]
[INFO] Apache HBase - Thrift .............................. SUCCESS [06:07 min]
[INFO] Apache HBase - Shell ............................... SUCCESS [06:30 min]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [02:57 min]
[INFO] Apache HBase - Backup .............................. SUCCESS [14:18 min]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 29.585 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [03:06 min]
[INFO] Apache HBase - Examples ............................ SUCCESS [01:22 min]
[INFO] Apache HBase - Shaded .............................. SUCCESS [  0.361 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [  1.040 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [  1.116 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [  3.394 s]
[INFO] Apache HBase - Extensions .......................... SUCCESS [  0.051 s]
[INFO] Apache HBase - OpenSSL support for TLS RPC ......... SUCCESS [  0.136 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [  6.907 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 11.495 s]
[INFO] Apache HBase - Compression ......................... SUCCESS [  0.156 s]
[INFO] Apache HBase - Compression - Aircompressor ......... SUCCESS [ 40.375 s]
[INFO] Apache HBase - Compression - Brotli ................ SUCCESS [ 33.545 s]
[INFO] Apache HBase - Compression - LZ4 ................... SUCCESS [ 22.442 s]
[INFO] Apache HBase - Compression - Snappy ................ SUCCESS [ 23.443 s]
[INFO] Apache HBase - Compression - ZStandard ............. SUCCESS [06:45 min]
[INFO] Apache HBase - Assembly ............................ SUCCESS [  4.141 s]
[INFO] Apache HBase - Assembly without Hadoop and its transitive dependencies SUCCESS [  4.176 s]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [ 10.873 s]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 23.381 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [  2.463 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [  1.069 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [  0.227 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 31.074 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 33.723 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [  0.415 s]
[INFO] Apache HBase - Generate Dev Classpath .............. SUCCESS [  3.723 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:04 h (Wall Clock)
[INFO] Finished at: 2025-09-21T13:27:51-07:00
[INFO] ------------------------------------------------------------------------
...
real    244m43.832s
user    1368m30.870s
sys     171m3.674s

…ation

haridsv · 2025-09-22T08:06:05Z

Looks like mvn surefire-report:report-only is broken, so I got a count of total tests run this way:

$ sed -En 's/.* Tests run: ([0-9]+).*/\1/p' /tmp/mvn-test.out | paste -s -d+ | bc
23678

The last green nightly build had 33,242 tests run, however, when I ran the above command on the test output file from the nightly build, I got a similar number (my branch must be having higher count because of new tests):

$ gsed -En 's/.* Tests run: ([0-9]+).*/\1/p' /tmp/patch-unit-root.txt | gpaste -s -d+ | bc
22793

The count is higher at the overall build level probably because of tests run in different profiles, such as hadoop3-backwards-3.3.5 etc.

Apache-HBase · 2025-09-23T05:17:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 33s		Docker mode activated.
-0 ⚠️	yetus	0m 4s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 29s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 26s		HBASE-29368-key-management-feature passed
+1 💚	compile	2m 14s		HBASE-29368-key-management-feature passed
+1 💚	javadoc	3m 21s		HBASE-29368-key-management-feature passed
+1 💚	shadedjars	6m 20s		branch has no errors when building our shaded downstream artifacts.
-0 ⚠️	patch	7m 1s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 12s		the patch passed
+1 💚	compile	2m 14s		the patch passed
+1 💚	javac	2m 14s		the patch passed
-0 ⚠️	javadoc	0m 27s	/results-javadoc-javadoc-hbase-server.txt	hbase-server generated 2 new + 63 unchanged - 0 fixed = 65 total (was 63)
-0 ⚠️	javadoc	1m 58s	/results-javadoc-javadoc-root.txt	root generated 2 new + 210 unchanged - 0 fixed = 212 total (was 210)
+1 💚	shadedjars	6m 15s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
-1 ❌	unit	3m 20s	/patch-unit-root.txt	root in the patch failed.
		36m 38s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#7297
JIRA Issue	HBASE-29495
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux e3161587d2f7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `f52ca4f`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/testReport/
Max. process+thread count	317 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-shell . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2025-09-23T06:02:59Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29368-key-management-feature Compile Tests _
+0 🆗	mvndep	0m 31s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 31s		HBASE-29368-key-management-feature passed
+1 💚	compile	8m 30s		HBASE-29368-key-management-feature passed
+1 💚	checkstyle	1m 12s		HBASE-29368-key-management-feature passed
-1 ❌	spotbugs	1m 35s	/branch-spotbugs-hbase-server-warnings.html	hbase-server in HBASE-29368-key-management-feature has 2 extant spotbugs warnings.
-1 ❌	spotbugs	7m 33s	/branch-spotbugs-root-warnings.html	root in HBASE-29368-key-management-feature has 2 extant spotbugs warnings.
-1 ❌	spotless	0m 42s		branch has 43 errors when running spotless:check, run spotless:apply to fix.
-0 ⚠️	patch	1m 17s		Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 8s		the patch passed
+1 💚	compile	8m 26s		the patch passed
+1 💚	cc	8m 26s		the patch passed
-0 ⚠️	javac	8m 26s	/results-compile-javac-root.txt	root generated 3 new + 1895 unchanged - 1 fixed = 1898 total (was 1896)
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 12s	/results-checkstyle-root.txt	root: The patch generated 5 new + 31 unchanged - 0 fixed = 36 total (was 31)
-0 ⚠️	rubocop	0m 9s	/results-rubocop.txt	The patch generated 13 new + 1 unchanged - 13 fixed = 14 total (was 14)
+1 💚	xmllint	0m 0s		No new issues.
+1 💚	spotbugs	13m 17s		the patch passed
+1 💚	hadoopcheck	12m 21s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	hbaseprotoc	4m 52s		the patch passed
+1 💚	spotless	0m 45s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 53s		The patch does not generate ASF License warnings.
		81m 44s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#7297
JIRA Issue	HBASE-29495
Optional Tests	dupname asflicense codespell detsecrets spotless javac spotbugs checkstyle compile hadoopcheck hbaseanti cc buflint bufcompat hbaseprotoc xmllint rubocop
uname	Linux 66e7ad1809ce 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29368-key-management-feature / `f52ca4f`
Default Java	Eclipse Adoptium-17.0.11+9
spotless	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/artifact/yetus-general-check/output/branch-spotless.txt
Max. process+thread count	190 (vs. ulimit of 30000)
modules	C: hbase-protocol-shaded hbase-common hbase-client hbase-server hbase-shell . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-7297/13/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3 rubocop=1.37.1 xmllint=20913
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

haridsv added 16 commits September 4, 2025 19:37

First attempt to get key management integrated into read path

0c7d277

Copy caches in ReaderContext copy constructor

901be98

Accommodate null server for offline. Also some quick refactoring.

e8fe61c

Handle scenarios that create HStoreFile directly

802416b

Moved key namespace generation utils out of SecurityUtil

0652890

Initial test coverage for SecurityUtil, used Cursor and then tweaked …

8c5aefe

…extensively

Extensive refactoring of TestSecurityUtil

202059e

Fix an issue with master region initialization failing. Also fix a ty…

8a2e79b

…po in shell command format and a few renames

Basic integration test for key management enablement.

80e610a

- Also includes a couple of fixes for typos - Refactored some of the key management test code so that it is reusable from a shell test.

Fix failing tests due to previous changes and also make the active ke…

aff8aef

…y search in global namespace - Also fixes an issue with point lookups for active keys.

Basic E2E encryption test

911b500

Refactoring to facilitate sharing key management caches across all re…

5022703

…gion operations - The approach involves creating a new KeyManagementSerice interface and passing it along the Server in as many code paths as possible. - Also improved the assertions in the encryption E2E test.

Added support to read from encrypted HFile

6e20700

Also added test coverage to read the HFile.

Added some test coverage for recent changes

b7eac2a

Fixed broken tests and some compilation failures

b01090c

Also reverted some prior changes and added overloaded methods to reduce impact on tests following existing pattern.