HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin #6010

stoty · 2024-06-20T14:57:51Z

No description provided.

Apache9

I do not fully understand what does the streams mean here...

All requests and responses are fully kept in memory here I think?

Apache9 · 2024-07-03T12:04:36Z

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ProtobufHttpEntity.java

+ * For sending a Protobuf encoded object via Apache HttpClient efficiently without an interim byte
+ * array. This exposes the underlying Apache HttpClient types, but so do the other client classes.
+ */
+@InterfaceAudience.Public


Does this need to be IA.Public?

Yes, we expect the user to explicitly create these entities.

i have not updated the tests to use them, but the intentended usage is similar to how
org.apache.hadoop.hbase.rest.client.RemoteHTable.put(Put) is implemented.

At least have a section in the javadoc to show how to use this class?

And since we have not reach an agreement on whether to move RemoteAdmin and RemoteHTable to src/main, I'm not sure whether we should make this class IA.Public...

I only mentined RemoHTable as example where this is used, but this
is not limited to RemotHTable, it should also be used directly in user code.

I will add a standalone test case that uses this class as an illustration.

At least have a section in the javadoc to show how to use this class?

And since we have not reach an agreement on whether to move RemoteAdmin and RemoteHTable to src/main, I'm not sure whether we should make this class IA.Public...

This is not tied to RemoteAdmin / RemoteHTable.
This is just a more efficient way to use the existing public Client class (or any custom client based on Apache HttpClient)

The RemoteAdmin / RemoteHTable API does not even change, this new class only used internally in the RemoteHTable implementation.

Apache9 · 2024-07-03T12:05:07Z

hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/ProtobufHttpEntity.java

+
+  @Override
+  public boolean isStreaming() {
+    // TODO Auto-generated method stub


Remove this line?

stoty · 2024-07-08T14:36:33Z

I do not fully understand what does the streams mean here...

All requests and responses are fully kept in memory here I think?

Currently they are.
The current code calls the Apache HttpClient getResponseBody() method, which will cause the client to wait untill all data is received, and load it into a byte array.

However, the goal is to avoid having to do that.

Protobuf primarily works on streams, so for a large resultset, we may reduce both processing (wall clock) time and memory consumption by not buffering the whole response into memory, but reading directly from the stream, so that

We do not have to wait for the full response to arrive before starting to process it.
We do not have to copy the whole response into a single byte array.
The processed response segments can be GCd while we are processing the rest of the message.

The Cell/Cellset structures are still kept in memory, but we avoid having to explicitly store them twice during processing (once the serialized byte array and once the java POJOs)

Apache9 · 2024-07-12T12:53:21Z

Please fix the javac and javadoc warnings if possible?

stoty · 2024-07-15T10:24:17Z

I have added org.apache.hadoop.hbase.rest.TestGetAndPutResource.testMultipleCellPutPBEntity() to illustrate how ProtobufEntity can be used to more efficiently execute PUT operations.

stoty · 2024-07-15T12:39:09Z

Fixed the warnings (though most were in code that I haven't touched)
and added a test that can serve as an example on using ProtobufEntity.
PTAL @Apache9 .

stoty · 2024-08-16T07:08:16Z

Can you take a look @Apache9 ?

stoty · 2024-09-10T05:49:35Z

Now that you're back from vacation and hopefully settled can you take a look at this again, @Apache9 ?

stoty · 2024-10-28T11:28:54Z

hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestGetAndPutResource.java

+    path.append('/');
+    path.append("dummy_row");
+    HttpPut httpPut = new HttpPut(path.toString());
+    httpPut.setEntity(new ProtobufHttpEntity(model));


This is how a normal client can use ProtobufHttpEntity, @Apache9 .

Apache-HBase · 2024-10-28T13:32:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 16s		Docker mode activated.
-0 ⚠️	yetus	0m 3s		Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
			_ Prechecks _
			_ master Compile Tests _
+1 💚	mvninstall	3m 7s		master passed
+1 💚	compile	0m 23s		master passed
+1 💚	javadoc	0m 17s		master passed
+1 💚	shadedjars	5m 24s		branch has no errors when building our shaded downstream artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 52s		the patch passed
+1 💚	compile	0m 21s		the patch passed
+1 💚	javac	0m 21s		the patch passed
+1 💚	javadoc	0m 16s		the patch passed
+1 💚	shadedjars	5m 23s		patch has no errors when building our shaded downstream artifacts.
			_ Other Tests _
+1 💚	unit	3m 27s		hbase-rest in the patch passed.
		22m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6010/8/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#6010
Optional Tests	javac javadoc unit compile shadedjars
uname	Linux 24174a730d70 5.4.0-192-generic #212-Ubuntu SMP Fri Jul 5 09:47:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5b0906d`
Default Java	Eclipse Adoptium-17.0.11+9
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6010/8/testReport/
Max. process+thread count	1602 (vs. ulimit of 30000)
modules	C: hbase-rest U: hbase-rest
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6010/8/console
versions	git=2.34.1 maven=3.9.8
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-10-28T13:36:59Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ master Compile Tests _
+1 💚	mvninstall	3m 3s		master passed
+1 💚	compile	0m 34s		master passed
+1 💚	checkstyle	0m 12s		master passed
+1 💚	spotbugs	0m 32s		master passed
+1 💚	spotless	0m 45s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+1 💚	mvninstall	2m 48s		the patch passed
+1 💚	compile	0m 33s		the patch passed
+1 💚	javac	0m 33s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 12s		the patch passed
+1 💚	spotbugs	0m 37s		the patch passed
+1 💚	hadoopcheck	10m 16s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 43s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 11s		The patch does not generate ASF License warnings.
		27m 19s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6010/8/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6010
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux beef180c5b0d 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `5b0906d`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	83 (vs. ulimit of 30000)
modules	C: hbase-rest U: hbase-rest
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6010/8/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

This comment has been minimized.

Sign in to view

stoty requested a review from Apache9 June 21, 2024 11:17

Apache9 reviewed Jul 3, 2024

View reviewed changes

stoty requested a review from Apache9 July 8, 2024 14:54

This comment has been minimized.

Sign in to view

stoty force-pushed the HBASE-28647 branch from 4fd2161 to 05adfad Compare July 10, 2024 07:13

This comment has been minimized.

Sign in to view

stoty force-pushed the HBASE-28647 branch from 3f2010b to 1ccfecb Compare July 15, 2024 10:53

This comment has been minimized.

Sign in to view

stoty added 3 commits October 28, 2024 12:27

HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin

710a3e8

set isStreaming to true.

3d9eb5c

add test for ProtobufHttpEntity

b3fdaad

stoty added 2 commits October 28, 2024 12:27

fix warnings

881f5b1

fix moar warnings

5b0906d

stoty force-pushed the HBASE-28647 branch from 3f9243d to 5b0906d Compare October 28, 2024 11:27

stoty commented Oct 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin #6010

HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin #6010

stoty commented Jun 20, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 left a comment

Apache9 Jul 3, 2024

stoty Jul 8, 2024

Apache9 Jul 12, 2024

stoty Jul 12, 2024

stoty Oct 16, 2024 •

edited

Loading

Apache9 Jul 3, 2024

stoty Jul 8, 2024

stoty commented Jul 8, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 commented Jul 12, 2024

stoty commented Jul 15, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty commented Jul 15, 2024

stoty commented Aug 16, 2024

stoty commented Sep 10, 2024

stoty Oct 28, 2024

Apache-HBase commented Oct 28, 2024

Apache-HBase commented Oct 28, 2024

HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin #6010

Are you sure you want to change the base?

HBASE-28647 Support streams in REST Client, RemoteHTable and RemoteAdmin #6010

Conversation

stoty commented Jun 20, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 left a comment

Choose a reason for hiding this comment

Apache9 Jul 3, 2024

Choose a reason for hiding this comment

stoty Jul 8, 2024

Choose a reason for hiding this comment

Apache9 Jul 12, 2024

Choose a reason for hiding this comment

stoty Jul 12, 2024

Choose a reason for hiding this comment

stoty Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Apache9 Jul 3, 2024

Choose a reason for hiding this comment

stoty Jul 8, 2024

Choose a reason for hiding this comment

stoty commented Jul 8, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Apache9 commented Jul 12, 2024

stoty commented Jul 15, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

stoty commented Jul 15, 2024

stoty commented Aug 16, 2024

stoty commented Sep 10, 2024

stoty Oct 28, 2024

Choose a reason for hiding this comment

Apache-HBase commented Oct 28, 2024

Apache-HBase commented Oct 28, 2024

stoty Oct 16, 2024 •

edited

Loading