HBASE-27217 Revisit the DumpReplicationQueues tool #4810

2005hithlj · 2022-10-02T15:36:21Z

https://issues.apache.org/jira/browse/HBASE-27217

Apache-HBase · 2022-10-02T16:14:18Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 5s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 33s	HBASE-27109/table_based_rqs passed
+1 💚	compile	2m 20s	HBASE-27109/table_based_rqs passed
+1 💚	checkstyle	0m 33s	HBASE-27109/table_based_rqs passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 22s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 13s	the patch passed
+1 💚	compile	2m 18s	the patch passed
+1 💚	javac	2m 18s	the patch passed
+1 💚	checkstyle	0m 31s	hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	7m 56s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 39s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		29m 6s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux d406b8ac8afd 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `de2d20a`
Default Java	Temurin-1.8.0_345-b01
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.7.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-10-02T19:35:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	4m 51s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 43s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 59s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 29s	the patch passed
+1 💚	compile	0m 42s	the patch passed
+1 💚	javac	0m 42s	the patch passed
+1 💚	shadedjars	3m 59s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	the patch passed
		_ Other Tests _
-1 ❌	unit	210m 14s	hbase-server in the patch failed.
		230m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 814721bae111 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `de2d20a`
Default Java	Eclipse Adoptium-11.0.16.1+1
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/testReport/
Max. process+thread count	2693 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-10-02T19:40:01Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 22s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 33s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 39s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 50s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 26s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 15s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	3m 49s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	217m 56s	hbase-server in the patch passed.
		234m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 102a8cb800a3 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `de2d20a`
Default Java	Temurin-1.8.0_345-b01
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/testReport/
Max. process+thread count	2506 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/1/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

2005hithlj · 2022-10-03T05:03:22Z

@Apache9 sir. Could you take a look? Thanks.
And the relevant UT of DumpReplicationQueues will be added later.

Apache9

Left some comments, PTAL.

Thanks @2005hithlj !

Apache9 · 2022-10-04T12:17:08Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

    }
  }

+  public String dumpReplicationViaTable(Connection connection) throws ReplicationException {


Does this need to be public?

Apache9 · 2022-10-04T12:18:46Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+        if (queueId.isRecovered()) {
+          wals = AbstractFSWALProvider
+            .getArchivedWALFiles(connection.getConfiguration(), queueId.getSourceServerName().get(),
+              URLEncoder.encode(queueId.getSourceServerName().get().toString(),


Do we need encode here?

Apache9 · 2022-10-04T12:20:24Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+    sb.append("    PeerID: " + queueId.getPeerId() + "\n");
+    sb.append("    Recovered: " + queueId.isRecovered() + "\n");
+    if (queueId.getSourceServerName().isPresent()) {
+      sb.append("    Dead RegionServers: " + queueId.getSourceServerName().get() + "\n");


Actually here we only record the first region server instead of the all the region servers, so maybe we should change the wording here in the future...

Apache9 · 2022-10-04T12:23:02Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

-      // long position = queueStorage.getWALPosition(regionserver, queueInfo.getPeerId(), wal);
-      // sb.append(" Replication position for " + wal + ": "
-      // + (position > 0 ? position : "0" + " (not started or nothing to replicate)") + "\n");
+      long position = -1;


I think here we should filter out the wals before the offset?

Apache9 · 2022-10-04T12:23:35Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+          position = offset.getOffset();
+        }
+      }
+      sb.append(" Replication position for " + (walGroup != null ? walGroup + "/" + wal : wal)


Notice that, -1 in ReplicationGroupOffset means this file has already been fully replicated.

Apache9 · 2022-10-19T15:49:37Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

@@ -229,21 +238,45 @@ private int dumpReplicationQueues(DumpOptions opts) throws Exception {
        LOG.info("Found [--distributed], will poll each RegionServer.");
        Set<String> peerIds =
          peers.stream().map((peer) -> peer.getPeerId()).collect(Collectors.toSet());
-        System.out.println(dumpQueues(zkw, peerIds, opts.isHdfs()));
+        System.out.println(dumpQueues(zkw, connection, peerIds, opts.isHdfs()));


We still need zkw?

Apache9 · 2022-10-19T15:53:47Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+      for (ReplicationQueueId queueId : queueIds) {
+        List<String> wals = null;
+        if (queueId.isRecovered()) {
+          wals = AbstractFSWALProvider


Just use queueId.getServerWALsBelongTo?

This method can really solve my problem.

Apache9 · 2022-10-19T15:57:13Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+              queueId.getSourceServerName().get().toString())
+            .stream().map(Path::toString).collect(Collectors.toList());
+        } else {
+          wals = AbstractFSWALProvider


Is this enough to get all the wals for a live region server?

Apache9 · 2022-10-19T15:57:50Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

    StringBuilder sb = new StringBuilder();
+    ReplicationQueueStorage queueStorage =
+      ReplicationStorageFactory.getReplicationQueueStorage(connection, getConf());
+    Set<ServerName> liveRegionServers = ZKUtil.listChildrenNoWatch(zkw, zkw.getZNodePaths().rsZNode)


This could also be done through Admin interface, so we do not need to depend on zk?

Apache9 · 2022-10-19T16:00:34Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+              queueId.getServerName().toString())
+            .stream().map(Path::toString).collect(Collectors.toList());
+        }
+        Collections.sort(wals);


String compare is enough for sorting? Not sure...

Better use TS_COMPARATOR?

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

Apache-HBase · 2022-10-19T16:19:34Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 12s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 30s	HBASE-27109/table_based_rqs passed
+1 💚	compile	2m 20s	HBASE-27109/table_based_rqs passed
+1 💚	checkstyle	0m 32s	HBASE-27109/table_based_rqs passed
+1 💚	spotless	0m 40s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 22s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 11s	the patch passed
+1 💚	compile	2m 15s	the patch passed
+1 💚	javac	2m 15s	the patch passed
+1 💚	checkstyle	0m 31s	hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	7m 59s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 38s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 12s	The patch does not generate ASF License warnings.
		29m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux cb55febcf8b2 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `67a94a6`
Default Java	Temurin-1.8.0_345-b01
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.7.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-10-19T19:37:24Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 43s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 43s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 42s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 58s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 24s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 31s	the patch passed
+1 💚	compile	0m 42s	the patch passed
+1 💚	javac	0m 42s	the patch passed
+1 💚	shadedjars	3m 57s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	208m 31s	hbase-server in the patch passed.
		226m 30s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7413adc0a5e5 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `67a94a6`
Default Java	Eclipse Adoptium-11.0.16.1+1
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/testReport/
Max. process+thread count	2962 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-10-19T19:42:27Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 32s	Docker mode activated.
-0 ⚠️	yetus	0m 9s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 16s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 39s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 46s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 14s	the patch passed
+1 💚	compile	0m 39s	the patch passed
+1 💚	javac	0m 39s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	215m 9s	hbase-server in the patch passed.
		231m 53s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 9fd67d6aa1e0 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `67a94a6`
Default Java	Temurin-1.8.0_345-b01
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/testReport/
Max. process+thread count	3887 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/2/console
versions	git=2.17.1 maven=3.6.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-10-22T16:23:43Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 15s	HBASE-27109/table_based_rqs passed
+1 💚	compile	2m 18s	HBASE-27109/table_based_rqs passed
+1 💚	checkstyle	0m 32s	HBASE-27109/table_based_rqs passed
+1 💚	spotless	0m 39s	branch has no errors when running spotless:check.
+1 💚	spotbugs	1m 19s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 10s	the patch passed
+1 💚	compile	2m 14s	the patch passed
+1 💚	javac	2m 14s	the patch passed
+1 💚	checkstyle	0m 31s	hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	8m 5s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 38s	patch has no errors when running spotless:check.
+1 💚	spotbugs	1m 25s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 10s	The patch does not generate ASF License warnings.
		28m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux e0fc3c649b2b 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `67a94a6`
Default Java	Temurin-1.8.0_345-b01
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/3/console
versions	git=2.17.1 maven=3.6.3 spotbugs=4.7.2
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2022-10-24T09:52:45Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

    }
  }

+  private String dumpReplicationViaTable(Connection connection) throws ReplicationException {


Checked the code in ZKDump.getReplicationZnodesDump, I do not think this is a direct mapping of the code there?

In ZKDump.getReplicationZnodesDump, first we dump the replication peer data, while here we do not dump it.
And when dumping the replication queue data, we will iterate all rsZNode and print them all, but here we first print all the peers and region servers, and then print all the wal files? Seems a bit strange...
And for dumping hfile refs, we will print hfile refs for each peer in ZKDump.getReplicationZnodesDump, but here we just print them all, without the peer information...

Apache9 · 2022-10-24T10:04:37Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+              queueId.getServerName().toString())
+            .stream().map(Path::toString).collect(Collectors.toList());
+        }
+        Collections.sort(wals);


Better use TS_COMPARATOR?

Apache9 · 2022-10-24T10:06:31Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+            " Replication position for " + (walGroup != null ? walGroup + "/" + wal : wal) + ": ");
+          // Position is -1, which means that the file has already been fully replicated,
+          // the logic here is different from the previous version.
+          if (position == -1) {


Is it possible that we arrive here but position is -1? I think it will be filtered out in the above ReplicationOffsetUtil.shouldReplicate check?

Apache9 · 2022-10-24T10:18:22Z

...rc/test/java/org/apache/hadoop/hbase/replication/regionserver/TestDumpReplicationQueues.java

@@ -83,7 +83,7 @@ public void testDumpReplicationReturnsWalSorted() throws Exception {
    Set<String> peerIds = new HashSet<>();
    peerIds.add("1");
    dumpQueues.setConf(config);
-    String dump = dumpQueues.dumpQueues(zkWatcherMock, peerIds, false);
+    String dump = dumpQueues.dumpQueues(null, peerIds, false);


Parsing null is enough here? We do not need to pass a Connection instance?

Apache9 · 2022-11-01T15:01:40Z

I've done a rebase on HBASE-27109, please also rebase here.

Thanks. @2005hithlj

2005hithlj · 2022-11-02T15:20:16Z

I've done a rebase on HBASE-27109, please also rebase here.

Thanks. @2005hithlj

@Apache9 OK sir, and I will have a new commit for the comments.

Apache-HBase · 2022-11-04T19:29:04Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	9m 2s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	3m 14s	HBASE-27109/table_based_rqs passed
+1 💚	compile	3m 43s	HBASE-27109/table_based_rqs passed
+1 💚	checkstyle	0m 41s	HBASE-27109/table_based_rqs passed
+1 💚	spotless	0m 58s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 3s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	3m 19s	the patch passed
+1 💚	compile	3m 19s	the patch passed
+1 💚	javac	3m 19s	the patch passed
+1 💚	checkstyle	0m 38s	hbase-server: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	11m 13s	Patch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚	spotless	0m 59s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 14s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 14s	The patch does not generate ASF License warnings.
		48m 33s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux d01e9569f619 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `56f0241`
Default Java	Temurin-1.8.0_352-b08
Max. process+thread count	64 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-04T22:40:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 46s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 41s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 57s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 25s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	3m 56s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
-1 ❌	unit	216m 57s	hbase-server in the patch failed.
		239m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux e8a6d98d1dfc 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `56f0241`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/testReport/
Max. process+thread count	2799 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2022-11-04T22:42:47Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	5m 14s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 29s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 38s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 45s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 25s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 8s	the patch passed
+1 💚	compile	0m 41s	the patch passed
+1 💚	javac	0m 41s	the patch passed
+1 💚	shadedjars	3m 46s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 23s	the patch passed
		_ Other Tests _
+1 💚	unit	218m 36s	hbase-server in the patch passed.
		242m 25s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 53ccac7769f5 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `56f0241`
Default Java	Temurin-1.8.0_352-b08
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/testReport/
Max. process+thread count	2860 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/4/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

Apache9 · 2022-11-05T15:38:07Z

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

+            }
+          }
+          // only for test
+          if (tmpWals.isEmpty() && !wals.contains(offset.getWal())) {


What does this mean?

...er/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DumpReplicationQueues.java

Apache9 · 2022-11-06T15:45:06Z

HBASE-27218 has been merged, so you need a rebase here...

2005hithlj · 2022-11-07T04:41:04Z

HBASE-27218 has been merged, so you need a rebase here...

OK, sir.

Apache-HBase · 2022-11-12T19:22:19Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ HBASE-27109/table_based_rqs Compile Tests _
+1 💚	mvninstall	2m 56s	HBASE-27109/table_based_rqs passed
+1 💚	compile	0m 43s	HBASE-27109/table_based_rqs passed
+1 💚	shadedjars	3m 58s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 22s	HBASE-27109/table_based_rqs passed
		_ Patch Compile Tests _
+1 💚	mvninstall	2m 26s	the patch passed
+1 💚	compile	0m 43s	the patch passed
+1 💚	javac	0m 43s	the patch passed
+1 💚	shadedjars	3m 58s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 21s	the patch passed
		_ Other Tests _
+1 💚	unit	216m 31s	hbase-server in the patch passed.
		236m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/7/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#4810
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 94f07ae5fe6d 5.4.0-1088-aws #96~18.04.1-Ubuntu SMP Mon Oct 17 02:57:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-27109/table_based_rqs / `c5196d7`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/7/testReport/
Max. process+thread count	2863 (vs. ulimit of 30000)
modules	C: hbase-server U: hbase-server
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4810/7/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 reviewed Oct 4, 2022

View reviewed changes

Apache9 force-pushed the HBASE-27109/table_based_rqs branch 2 times, most recently from 82a795a to 67a94a6 Compare October 12, 2022 07:55

2005hithlj force-pushed the HBASE-27217 branch from 60f3ae0 to 77e8bcb Compare October 19, 2022 15:41

Apache9 reviewed Oct 19, 2022

View reviewed changes

Apache9 reviewed Oct 24, 2022

View reviewed changes

Apache9 force-pushed the HBASE-27109/table_based_rqs branch from 67a94a6 to 6d4e7fd Compare October 24, 2022 11:44

Apache9 force-pushed the HBASE-27109/table_based_rqs branch from 6d4e7fd to 56f0241 Compare November 1, 2022 04:29

2005hithlj force-pushed the HBASE-27217 branch from dee333c to 36ed2ca Compare November 4, 2022 18:34

Apache9 reviewed Nov 5, 2022

View reviewed changes

Apache9 force-pushed the HBASE-27109/table_based_rqs branch 2 times, most recently from 7d1cbb3 to c5196d7 Compare November 6, 2022 09:13

2005hithlj force-pushed the HBASE-27217 branch 2 times, most recently from e90d2c9 to e9cbb69 Compare November 7, 2022 16:39

HBASE-27217 Revisit the DumpReplicationQueues tool

37f56b8

Apache9 merged commit 1412a33 into apache:HBASE-27109/table_based_rqs Nov 13, 2022

Apache9 pushed a commit that referenced this pull request Nov 13, 2022

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

dd5fedb

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Nov 27, 2022

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

209865d

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Nov 30, 2022

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

8ae42f9

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Dec 3, 2022

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

a78aac5

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Jan 10, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

8ebbc67

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Jan 18, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

643e5fc

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Jan 19, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

d2c82c7

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Jan 31, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

9e8699d

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Feb 3, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

947cbb5

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Mar 6, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

2e2ec6c

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Mar 9, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

e5d10b0

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Mar 13, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

14e24cf

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Mar 15, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

10e73d3

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Mar 18, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

96f15b3

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 5, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

45c1aa1

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 18, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

4a57d32

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 21, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

2b1538d

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 23, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

9c9f261

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 24, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

69a8b5c

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 25, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

48bd2fd

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 27, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

cd49b84

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request Apr 28, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

f31d756

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 1, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

a0822a2

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 4, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

ec8136c

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 6, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

d5bbf0f

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 8, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

015c110

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 13, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

487f0c6

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache9 pushed a commit that referenced this pull request May 15, 2023

HBASE-27217 Revisit the DumpReplicationQueues tool (#4810)

ffad1ff

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HBASE-27217 Revisit the DumpReplicationQueues tool #4810

HBASE-27217 Revisit the DumpReplicationQueues tool #4810

2005hithlj commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

2005hithlj commented Oct 3, 2022

Apache9 left a comment

Apache9 Oct 4, 2022

Apache9 Oct 4, 2022

Apache9 Oct 4, 2022

Apache9 Oct 4, 2022

Apache9 Oct 4, 2022

Apache9 Oct 19, 2022

Apache9 Oct 19, 2022

2005hithlj Oct 20, 2022

Apache9 Oct 19, 2022

Apache9 Oct 19, 2022

Apache9 Oct 19, 2022

Apache9 Oct 24, 2022

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 22, 2022

Apache9 Oct 24, 2022

Apache9 Oct 24, 2022

Apache9 Oct 24, 2022

Apache9 Oct 24, 2022

Apache9 commented Nov 1, 2022

2005hithlj commented Nov 2, 2022

Apache-HBase commented Nov 4, 2022

Apache-HBase commented Nov 4, 2022

Apache-HBase commented Nov 4, 2022

Apache9 Nov 5, 2022

Apache9 commented Nov 6, 2022

2005hithlj commented Nov 7, 2022

Apache-HBase commented Nov 12, 2022

HBASE-27217 Revisit the DumpReplicationQueues tool #4810

HBASE-27217 Revisit the DumpReplicationQueues tool #4810

Conversation

2005hithlj commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

Apache-HBase commented Oct 2, 2022

2005hithlj commented Oct 3, 2022

Apache9 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 19, 2022

Apache-HBase commented Oct 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Apache9 commented Nov 1, 2022

2005hithlj commented Nov 2, 2022

Apache-HBase commented Nov 4, 2022

Apache-HBase commented Nov 4, 2022

Apache-HBase commented Nov 4, 2022

Choose a reason for hiding this comment

Apache9 commented Nov 6, 2022

2005hithlj commented Nov 7, 2022

Apache-HBase commented Nov 12, 2022