HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching #5660

bbeaudreault · 2024-01-31T14:55:02Z

This also simplifies the code quite a bit, since we no longer need reflection for old unsupported versions. Updated the test to account for the new logic and cover this case.

…witching

Apache-HBase · 2024-01-31T16:54:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 43s	master passed
+1 💚	compile	2m 28s	master passed
+1 💚	checkstyle	0m 36s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 29s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 43s	the patch passed
+1 💚	compile	2m 26s	the patch passed
+1 💚	javac	2m 26s	the patch passed
-0 ⚠️	checkstyle	0m 36s	hbase-server: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 42s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 41s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 39s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		27m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux d7623d3c8220 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count	81 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-01-31T20:36:26Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 24s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 2s	master passed
+1 💚	compile	0m 48s	master passed
+1 💚	shadedjars	5m 48s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 58s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	5m 47s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	224m 45s	hbase-server in the patch passed.
		249m 13s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b08c38b8a95d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/testReport/
Max. process+thread count	4712 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-01-31T20:42:32Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 36s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 31s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	5m 1s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	4m 57s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
+1 💚	unit	233m 2s	hbase-server in the patch passed.
		255m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux bf08ae9f76fb 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/testReport/
Max. process+thread count	5717 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-01-31T21:10:31Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 11s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+1 💚	mvninstall	2m 43s	master passed
+1 💚	compile	2m 28s	master passed
+1 💚	checkstyle	0m 37s	master passed
+1 💚	spotless	0m 41s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 30s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	2m 26s	the patch passed
+1 💚	javac	2m 26s	the patch passed
+1 💚	checkstyle	0m 36s	the patch passed
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 43s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 38s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		26m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux f680d3126af9 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Eclipse Adoptium-11.0.17+8
Max. process+thread count	81 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-01T00:57:28Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 26s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	3m 21s	master passed
+1 💚	compile	0m 47s	master passed
+1 💚	shadedjars	5m 46s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 59s	the patch passed
+1 💚	compile	0m 48s	the patch passed
+1 💚	javac	0m 48s	the patch passed
+1 💚	shadedjars	5m 48s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 8s	hbase-server in the patch passed.
		253m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 06bc5a6bf7d1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/testReport/
Max. process+thread count	4585 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-02-01T01:07:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+1 💚	mvninstall	2m 24s	master passed
+1 💚	compile	0m 40s	master passed
+1 💚	shadedjars	5m 1s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	master passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 25s	the patch passed
+1 💚	compile	0m 40s	the patch passed
+1 💚	javac	0m 40s	the patch passed
+1 💚	shadedjars	5m 4s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 36s	the patch passed
		_ Other Tests _
-1 ❌	unit	241m 48s	hbase-server in the patch failed.
		264m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5660
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a1a7e1e17036 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `c4e332a`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/testReport/
Max. process+thread count	5357 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Humbedooh · 2024-02-01T09:14:15Z

testing jira connections here, ignore this please.

…witching (#5660) Signed-off-by: Duo Zhang <zhangduo@apache.org>

…m checksum switching (apache#5660) (apache#91) Signed-off-by: Duo Zhang <zhangduo@apache.org> (cherry picked from commit c28e285) Co-authored-by: Bryan Beaudreault <bbeaudreault@apache.org>

…m checksum switching (apache#5660) (cherry picked from commit c28e285) Change-Id: I1e644c145d0a93466b1d9e259a565d19e4ec88d1 Signed-off-by: Duo Zhang <zhangduo@apache.org> Co-authored-by: Bryan Beaudreault <bbeaudreault@apache.org> (cherry picked from commit bbd848a00eedd0129ac7711e620fe4ec2d903e65)

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum s…

94536ad

…witching

bbeaudreault requested a review from Apache9 January 31, 2024 14:55

This comment was marked as outdated.

Sign in to view

spotless/checkstyle

9ed72ff

This comment was marked as outdated.

Sign in to view

fix/add test

132ad6b

This comment was marked as outdated.

Sign in to view

checkstyle

087e425

Apache9 approved these changes Feb 1, 2024

View reviewed changes

bbeaudreault merged commit 256f10b into apache:master Feb 5, 2024
1 check failed

bbeaudreault deleted the HBASE-28338 branch February 5, 2024 13:26

bbeaudreault added a commit that referenced this pull request Feb 5, 2024

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum s…

ee6332a

…witching (#5660) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit that referenced this pull request Feb 5, 2024

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum s…

c28e285

…witching (#5660) Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit that referenced this pull request Feb 5, 2024

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum s…

db8ae95

…witching (#5660) Signed-off-by: Duo Zhang <zhangduo@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching #5660

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching #5660

bbeaudreault commented Jan 31, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Feb 1, 2024

Apache-HBase commented Feb 1, 2024

Humbedooh commented Feb 1, 2024

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching #5660

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching #5660

Conversation

bbeaudreault commented Jan 31, 2024 • edited Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Jan 31, 2024

Apache-HBase commented Feb 1, 2024

Apache-HBase commented Feb 1, 2024

Humbedooh commented Feb 1, 2024

bbeaudreault commented Jan 31, 2024 •

edited

Loading