HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted #5927

fjvbn2003 · 2024-05-19T12:17:22Z

Issue Link: https://issues.apache.org/jira/browse/HBASE-28599 ( reported by @Robiee17 )
Below is written by @Robiee17

Issue:
`RowTooBigException` is thrown when a duplicate increment RPC call is attempted.

Expected Behavior:
1. The initial RPC increment call should time out for some reason.
2. The duplicate RPC call should be converted to a GET request and fetch the result that I am trying to increment.
3. The result should contain only the qualifier that I am attempting to increment.

Actual Behavior:
1. The initial RPC increment call timed out, which is expected.
2. The duplicate RPC call is converted to a GET request but fails to clone the qualifier into the GET request.
3. Hence, the GET request attempts to retrieve all qualifiers for the given row and columnfamily, resulting in a `RowTooBigException`.

Steps to Reproduce:
1. Ensure a row with a total value size exceeding `hbase.table.max.rowsize` (default = 1073741824) exists.
2. Nonce property should be enabled `hbase.client.nonces.enabled` which is actually defaulted to true.
3. Attempt to increment a qualifier against the same row.
4. In my case, I am using a postIncrement co-processor which may cause a delay (longer than the RPC timeout property).
5. A duplicate increment call should be triggered, which tries to get the value rather than increment it.
6. The GET request actually tries to retrieve all the qualifiers for the row, resulting in a `RowTooBigException`

Test

When I print out this get object, there are no Column Family and Column Qualifier information in the second retry.

{"targetReplicaId":-1,"totalColumns":0,"families":{},"priority":-1,"consistency":"STRONG","maxVersions":1,"storeOffset":0,"checkExistenceOnly":false,"storeLimit":-1,"cacheBlocks":true,"row":"row1","timeRange":["0","9223372036854775807"]}

Because logic does not run this while loop,The GET request actually tries to retrieve all the qualifiers for the row, resulting in a RowTooBigException

After applying this change, I found that Column Family and Column Information is synced well and RowTooBigException doesn't occur when Incrementing that row in the second retry .

{"targetReplicaId":-1,"totalColumns":1,"families":{"cf":["counter"]},"priority":-1,"consistency":"STRONG","maxVersions":1,"storeOffset":0,"checkExistenceOnly":false,"storeLimit":-1,"cacheBlocks":true,"row":"row1","timeRange":["0","9223372036854775807"]}

Sugestion

To resolve this issue I deleted exclamation mark in while loop condition. Please review it and let me know if this have any problems. Thank you.

Apache-HBase · 2024-05-19T12:48:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 27s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	2m 37s	master passed
+1 💚	checkstyle	0m 35s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 41s	the patch passed
+1 💚	compile	2m 38s	the patch passed
+1 💚	javac	2m 38s	the patch passed
+1 💚	checkstyle	0m 31s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 2s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 39s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 28s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 9s	The patch does not generate ASF License warnings.
		28m 17s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5927
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 93e510c2318f 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6b3f5ae`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9

Seems like a typo, let's check the commit history and also check why we can not find this typo by our current tests...

Apache9 · 2024-05-19T13:25:11Z

Let's wait for the pre commit result.

injae-kim · 2024-05-19T14:10:38Z

while (!cellScanner.advance()) is only used here so seems typo?
there's no test that cover toGet() so we can't find this until now 😅

Apache-HBase · 2024-05-19T16:40:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 9s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 5s	master passed
+1 💚	compile	0m 51s	master passed
+1 💚	shadedjars	5m 19s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 28s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 50s	the patch passed
+1 💚	compile	0m 49s	the patch passed
+1 💚	javac	0m 49s	the patch passed
+1 💚	shadedjars	5m 14s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	the patch passed
		_ Other Tests _
+1 💚	unit	235m 4s	hbase-server in the patch passed.
		260m 28s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5927
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ec642c0da72d 5.4.0-174-generic #193-Ubuntu SMP Thu Mar 7 14:29:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6b3f5ae`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/testReport/
Max. process+thread count	4765 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-05-19T17:14:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 58s	master passed
+1 💚	compile	1m 12s	master passed
+1 💚	shadedjars	5m 53s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 31s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 15s	the patch passed
+1 💚	compile	1m 9s	the patch passed
+1 💚	javac	1m 9s	the patch passed
+1 💚	shadedjars	6m 17s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 33s	the patch passed
		_ Other Tests _
-1 ❌	unit	265m 42s	hbase-server in the patch failed.
		293m 51s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5927
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux f516d78ac54c 5.4.0-182-generic #202-Ubuntu SMP Fri Apr 26 12:29:36 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6b3f5ae`
Default Java	Eclipse Adoptium-17.0.10+7
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/testReport/
Max. process+thread count	5549 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-05-19T17:16:14Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 18s	master passed
+1 💚	compile	0m 51s	master passed
+1 💚	shadedjars	6m 25s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 32s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 2s	the patch passed
+1 💚	compile	0m 53s	the patch passed
+1 💚	javac	0m 53s	the patch passed
+1 💚	shadedjars	6m 19s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 29s	the patch passed
		_ Other Tests _
-1 ❌	unit	268m 11s	hbase-server in the patch failed.
		295m 31s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5927
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9e35ebca19e7 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `6b3f5ae`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/testReport/
Max. process+thread count	4731 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5927/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

fjvbn2003 · 2024-05-20T00:36:02Z

@Apache9 It's my first time contributing to HBase, so I'm not very familiar with the process. Could you help me to resolve this build failed situation?

In the above pre-commit result, there are a few failed unit tests.
Do I need to fix the tests listed below

[ERROR] Errors: 
[ERROR] org.apache.hadoop.hbase.regionserver.TestRegionServerReportForDuty.null
[ERROR]   Run 1: TestRegionServerReportForDuty.testReportForDutyWithMasterChange:211->waitForClusterOnline:348 Â» TestTimedOut test timed out after 780 seconds
[ERROR]   Run 2: TestRegionServerReportForDuty Â»  Appears to be stuck in thread HRegionServer-EventLoopGroup-5-1

[ERROR] Errors: 
[ERROR] org.apache.hadoop.hbase.TestJMXConnectorServer.null
[ERROR]   Run 1: TestJMXConnectorServer.setUpBeforeClass:74 Â» IO Shutting down
[ERROR]   Run 2: TestJMXConnectorServer.tearDownAfterClass:80 NullPointer

Apache9 · 2024-05-20T01:13:58Z

The UTs of HBase are kinda heavy, more like a integration test, so it is common that there are some flaky ones.

The failed ones are likely not related to this PR here.

… call is attempted (#5927) The root cause is analyzed by Robin Infant A(robiee17) Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit dba7dcc)

… call is attempted (apache#5927) The root cause is analyzed by Robin Infant A(robiee17) Signed-off-by: Duo Zhang <zhangduo@apache.org>

HBASE-28599 Delete exclamation mark

af9c2a5

fjvbn2003 changed the title ~~HBASE-28599 Delete exclamation mark~~ HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted May 19, 2024

Apache9 reviewed May 19, 2024

View reviewed changes

Apache9 approved these changes May 19, 2024

View reviewed changes

Apache9 merged commit dba7dcc into apache:master May 20, 2024
1 check failed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted #5927

HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted #5927

fjvbn2003 commented May 19, 2024 •

edited

Loading

Apache-HBase commented May 19, 2024

Apache9 left a comment

Apache9 commented May 19, 2024

injae-kim commented May 19, 2024

Apache-HBase commented May 19, 2024

Apache-HBase commented May 19, 2024

Apache-HBase commented May 19, 2024

fjvbn2003 commented May 20, 2024 •

edited

Loading

Apache9 commented May 20, 2024

HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted #5927

HBASE-28599 RowTooBigException is thrown when duplicate increment RPC call is attempted #5927

Conversation

fjvbn2003 commented May 19, 2024 • edited Loading

Test

Sugestion

Apache-HBase commented May 19, 2024

Apache9 left a comment

Choose a reason for hiding this comment

Apache9 commented May 19, 2024

injae-kim commented May 19, 2024

Apache-HBase commented May 19, 2024

Apache-HBase commented May 19, 2024

Apache-HBase commented May 19, 2024

fjvbn2003 commented May 20, 2024 • edited Loading

Apache9 commented May 20, 2024

fjvbn2003 commented May 19, 2024 •

edited

Loading

fjvbn2003 commented May 20, 2024 •

edited

Loading