Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

HBASE-28705 backup log cleaner cleans required walls when using multiple backuproots #6040

Merged
merged 3 commits into from
Sep 25, 2024
Merged

HBASE-28705 backup log cleaner cleans required walls when using multiple backuproots #6040

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f371d51
HBASE-28705 BackupLogCleaner cleans required WALs when using multiple…
DieterDP-ng Jul 2, 2024
7d76842
HBASE-28705 BackupLogCleaner cleans required WALs when using multiple…
DieterDP-ng Jul 2, 2024
a31a19c
Process review comments, bugfix
DieterDP-ng Sep 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 42 additions & 25 deletions hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/master/BackupLogCleaner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,39 +81,55 @@ public void init(Map<String, Object> params) {
}
}

private Map<Address, Long> getServerToNewestBackupTs(List<BackupInfo> backups)
/**
* Calculates the timestamp boundary up to which all backup roots have already included the WAL.
* I.e. WALs with a lower (= older) or equal timestamp are no longer needed for future incremental
* backups.
*/
private Map<Address, Long> serverToPreservationBoundaryTs(List<BackupInfo> backups)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this method have a precondition asserting that all BackupInfos in the list have an identical table set?

Otherwise I think this could cause data loss in an incremental backup. Let's say we have one-table and another-table on our cluster, and we back up each one independently. Let's say we first took a backup for one-table, then a backup for another-table. Then we passed the corresponding BackupInfos into this method, this would yield preservation boundaries which would remove WALs necessary for the next incremental backup of one-table

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see an issue. Trying to follow your example:

  • Imagine 3 servers S1, S2, S3
  • table T1 has regions on S1 and S2, table T2 has regions on S2 and S3
  • we have 2 backup roots R1 that backs up T1, and R2 that backs up T2

At time:

  • t=0, we backup T1 in R1 => backup B1
  • t=0, we backup T2 in R2 => backup B2
  • t=10, we backup T1 in R1 => backup B3
  • t=20, we backup T2 in R2. => backup B4

Following the logic in this method:

  • newestBackupPerRootDir will contain: (R1: B3, R2: B4)
  • boundaries will contain: (S1: 10, S2: 10, S3: 20)

So for T1, all WALs up to t=10 can be deleted, for T2, WALs will be preserved from t=10 or t=20, depending whether other tables are present.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the explanation, both this and below. I think I follow you. Let me rephrase.

newestBackupPerRootDir is a "race to the top" in terms of identifying the most recent timestamp across backups, while boundaries is a "race to the bottom" in terms of identifying the least recent backup present on each server. By selecting the "oldest of the newest" you determine the minimum timestamp that must be preserved for each server. Any WALs older than this minimum timestamp can be safely discarded.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, I think the missing piece for me was that a backup root was specific to a given table

we have 2 backup roots R1 that backs up T1, and R2 that backs up T2

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, I think the missing piece for me was that a backup root was specific to a given table

It isn't. Each backup root contains one or more tables, i.e. a set of tables, these table sets may or may not have tables in common.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, okay so reframing your example in a single root:

Imagine
3 servers S1, S2, S3
table T1 has regions on S1 and S2, table T2 has regions on S2 and S3
we have 1 backup root R1 that backs up T1 and T2

At time:
t=0, we backup T1 in R1 => backup B1
t=0, we backup T2 in R1 => backup B2
t=10, we backup T1 in R1 => backup B3
t=20, we backup T2 in R1 => backup B4

Following the logic in this method:

newestBackupPerRootDir will contain: (R1: B4)

We would calculate the preservation boundaries here:

    // This map tracks, for every address, the least recent (= oldest / lowest timestamp) inclusion
    // in any backup. In other words, it is the timestamp boundary up to which all backups roots
    // have included the WAL in their backup.
    Map<Address, Long> boundaries = new HashMap<>();
    for (BackupInfo backupInfo : newestBackupPerRootDir.values()) {
      for (TableName table : backupInfo.getTables()) {
        for (Map.Entry<String, Long> entry : backupInfo.getTableSetTimestampMap().get(table)
          .entrySet()) {
          Address address = Address.fromString(entry.getKey());
          Long storedTs = boundaries.get(address);
          if (storedTs == null || entry.getValue() < storedTs) {
            boundaries.put(address, entry.getValue());
          }
        }
      }
    }

Since the newest backup in R1 is B4, serverToPreservationBoundaryTs will contain (S1: 20, S2: 20, S3: 20)

In this situation, we must not delete WALs from S1 or S2 between times 10 and 20 because a subsequent incremental backup of T1 will require those WALs.

In the BackupLogCleaner, though, we will end up calling canDeleteFile and hitting this code with that serverToPreservationBoundaryTs (S1: 20, S2: 20, S3: 20) renamed as addressToBoundaryTs:

      if (!addressToBoundaryTs.containsKey(walServerAddress)) { // for S1 & S2, this will be false
         // irrelevant ...
      }

      Long backupBoundary = addressToBoundaryTs.get(walServerAddress); // for S1 & S2, this will be 20
      if (backupBoundary >= walTimestamp) { // For WALs between times 10 and 20, this will be true
        if (LOG.isDebugEnabled()) {
          LOG.debug(
            "Backup found for server: {}. All backups from {} are newer than file, so deleting: {}",
            walServerAddress.getHostName(), backupBoundary, path);
        }
        // so we will erroneously delete WALs <20, despite the next inc backup of T1 requiring them
        return true;
      }

I guess this depends on how ancestry is defined? If we consider all backups in a root to be ancestors regardless of their table set, then maybe it is okay to delete these WALs. But, if not, then I don't see how it is okay?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newestBackupPerRootDir will contain: (R1: B4)

Agreed

Since the newest backup in R1 is B4, serverToPreservationBoundaryTs will contain (S1: 20, S2: 20, S3: 20)

I think this is wrong.

To make things more concrete, I'm going to assume all backups were full backups, and B1 == B2 (since it has the same timestamp). I.e. the backups were:

  • B1: at timestamp 0, containing tables T1 & T2
  • B3: at timestamp 10, containing T1
  • B4: at timestamp 20, containing T2

At this point, the data in backupInfo.getTableSetTimestampMap() will be:

T1:
  S1: 10
  S2: 10
T2:
  S2: 20
  S3: 20

and serverToPreservationBoundaryTs will be (S1: 10, S2: 10, S3: 20).

The reason that the BackupInfo of B4 contains the log timestamps of tables not included in B4 is due to how the backup client updates these:

// From: FullTableBackupClient

// Updates only the rows for the tables included in B4
backupManager.writeRegionServerLogTimestamp(backupInfo.getTables(), newTimestamps);

// Reads the rows for all tables once included in a backup in this backup root
Map<TableName, Map<String, Long>> newTableSetTimestampMap = backupManager.readLogTimestampMap();

Copy link
Contributor

@rmdmattingly rmdmattingly Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for being a pain here, but I'm not sure I agree. When building serverToPreservationBoundaryTs we loop through:

Map<Address, Long> boundaries = new HashMap<>();
for (BackupInfo backupInfo : newestBackupPerRootDir.values()) {
  // build boundaries via tableSetTimestampMap, like you said
}
// logging omitted
return boundaries;

and we agree that newestBackupPerRootDir.values() will only contain B4. So our boundaries would end up only being based on the timestamps from B4? How does the second newest backup, and beyond, in the root come into play?

Copy link
Contributor

@rmdmattingly rmdmattingly Sep 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does the second newest backup, and beyond, in the root come into play?

Answering my own questions a bit here, I think I needed to look deeper into how the BackupInfo gets constructed. When the tableSetTimestampMap is populated, it will contain entries for every table in the root, not just every table in the given backup. So while B4 might only pertain to T2, its corresponding BackupInfo will be aware of the timestamps for T1. That's a confusing design imo, but if true then I think this logic is sound

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's indeed like that. Doing it that way is a good way to avoid some special cases where backups in the same root backup do not contain the same tables, but the docs surrounding those BackupInfo concepts is indeed lacking.

throws IOException {
if (LOG.isDebugEnabled()) {
LOG.debug(
"Cleaning WALs if they are older than the newest backups. "
"Cleaning WALs if they are older than the newest backups (for all roots). "
+ "Checking WALs against {} backups: {}",
backups.size(),
backups.stream().map(BackupInfo::getBackupId).sorted().collect(Collectors.joining(", ")));
}
Map<Address, Long> serverAddressToNewestBackupMap = new HashMap<>();

Map<TableName, Long> tableNameBackupInfoMap = new HashMap<>();
for (BackupInfo backupInfo : backups) {
// This map tracks, for every backup root, the most recent created backup (= highest timestamp)
Map<String, BackupInfo> newestBackupPerRootDir = new HashMap<>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can there be multiple BackupInfo objects with the same root? (what is a "root" anyway? a FileSystem + Path?) Should this be a Map<String, List<BackupInfo>>?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A backup root corresponds to the path you specify when doing hbase backup create <type> <backup_path>. Within a single root, you'll have a series of backups. Having multiple roots allows you to manage backups independently (eg: if you delete one root, backups in other roots are unaffected). Incremental backups are always built on top of previous backups from the same root.

Eg, (F = full backup, I = incremental backup):

Time --------------------------------------------->
Root1: F    I     I     I     F    I    I    I
Root2: F          F  I    I   I              F

So to answer your questions:

  • There are multiple BackupInfos per root: 1 per backup created in that root. A root is a Path.
  • In this piece of logic, we only need the most recent (= newest) backup, so Map<String, BackupInfo>, where the root is represented as a String.

for (BackupInfo backup : backups) {
BackupInfo existingEntry = newestBackupPerRootDir.get(backup.getBackupRootDir());
if (existingEntry == null || existingEntry.getStartTs() < backup.getStartTs()) {
newestBackupPerRootDir.put(backup.getBackupRootDir(), backup);
}
}

// This map tracks, for every address, the least recent (= oldest / lowest timestamp) inclusion
// in any backup. In other words, it is the timestamp boundary up to which all backups roots
// have included the WAL in their backup.
Map<Address, Long> boundaries = new HashMap<>();
for (BackupInfo backupInfo : newestBackupPerRootDir.values()) {
for (TableName table : backupInfo.getTables()) {
tableNameBackupInfoMap.putIfAbsent(table, backupInfo.getStartTs());
if (tableNameBackupInfoMap.get(table) <= backupInfo.getStartTs()) {
tableNameBackupInfoMap.put(table, backupInfo.getStartTs());
for (Map.Entry<String, Long> entry : backupInfo.getTableSetTimestampMap().get(table)
.entrySet()) {
serverAddressToNewestBackupMap.put(Address.fromString(entry.getKey()),
entry.getValue());
for (Map.Entry<String, Long> entry : backupInfo.getTableSetTimestampMap().get(table)
.entrySet()) {
Address address = Address.fromString(entry.getKey());
Long storedTs = boundaries.get(address);
if (storedTs == null || entry.getValue() < storedTs) {
boundaries.put(address, entry.getValue());
}
}
}
}

if (LOG.isDebugEnabled()) {
for (Map.Entry<Address, Long> entry : serverAddressToNewestBackupMap.entrySet()) {
LOG.debug("Server: {}, Newest Backup: {}", entry.getKey().getHostName(), entry.getValue());
for (Map.Entry<Address, Long> entry : boundaries.entrySet()) {
LOG.debug("Server: {}, WAL cleanup boundary: {}", entry.getKey().getHostName(),
entry.getValue());
}
}

return serverAddressToNewestBackupMap;
return boundaries;
}

@Override
Expand All @@ -128,18 +144,19 @@ public Iterable<FileStatus> getDeletableFiles(Iterable<FileStatus> files) {
return files;
}

Map<Address, Long> addressToNewestBackupMap;
Map<Address, Long> serverToPreservationBoundaryTs;
try {
try (BackupManager backupManager = new BackupManager(conn, getConf())) {
addressToNewestBackupMap = getServerToNewestBackupTs(backupManager.getBackupHistory(true));
serverToPreservationBoundaryTs =
serverToPreservationBoundaryTs(backupManager.getBackupHistory(true));
}
} catch (IOException ex) {
LOG.error("Failed to analyse backup history with exception: {}. Retaining all logs",
ex.getMessage(), ex);
return Collections.emptyList();
}
for (FileStatus file : files) {
if (canDeleteFile(addressToNewestBackupMap, file.getPath())) {
if (canDeleteFile(serverToPreservationBoundaryTs, file.getPath())) {
filteredFiles.add(file);
}
}
Expand Down Expand Up @@ -174,7 +191,7 @@ public boolean isStopped() {
return this.stopped;
}

protected static boolean canDeleteFile(Map<Address, Long> addressToNewestBackupMap, Path path) {
protected static boolean canDeleteFile(Map<Address, Long> addressToBoundaryTs, Path path) {
if (isHMasterWAL(path)) {
return true;
}
Expand All @@ -190,28 +207,28 @@ protected static boolean canDeleteFile(Map<Address, Long> addressToNewestBackupM
Address walServerAddress = Address.fromString(hostname);
long walTimestamp = AbstractFSWALProvider.getTimestamp(path.getName());

if (!addressToNewestBackupMap.containsKey(walServerAddress)) {
if (!addressToBoundaryTs.containsKey(walServerAddress)) {
if (LOG.isDebugEnabled()) {
LOG.debug("No backup found for server: {}. Deleting file: {}",
walServerAddress.getHostName(), path);
}
return true;
}

Long lastBackupTs = addressToNewestBackupMap.get(walServerAddress);
if (lastBackupTs >= walTimestamp) {
Long backupBoundary = addressToBoundaryTs.get(walServerAddress);
if (backupBoundary >= walTimestamp) {
if (LOG.isDebugEnabled()) {
LOG.debug(
"Backup found for server: {}. Backup from {} is newer than file, so deleting: {}",
walServerAddress.getHostName(), lastBackupTs, path);
"Backup found for server: {}. All backups from {} are newer than file, so deleting: {}",
walServerAddress.getHostName(), backupBoundary, path);
}
return true;
}

if (LOG.isDebugEnabled()) {
LOG.debug(
"Backup found for server: {}. Backup from {} is older than the file, so keeping: {}",
walServerAddress.getHostName(), lastBackupTs, path);
walServerAddress.getHostName(), backupBoundary, path);
}
} catch (Exception ex) {
LOG.warn("Error occurred while filtering file: {}. Ignoring cleanup of this log", path, ex);
Expand Down
81 changes: 56 additions & 25 deletions hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/master/TestBackupLogCleaner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,17 @@
*/
package org.apache.hadoop.hbase.backup.master;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertTrue;

import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseClassTestRule;
Expand Down Expand Up @@ -61,8 +65,10 @@ public class TestBackupLogCleaner extends TestBackupBase {

@Test
public void testBackupLogCleaner() throws Exception {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is not quite what you described in your earlier comment, and I'm pretty sure that it doesn't test the criteria in the same way either. It's a valid test case, but I think it's missing some cases that we would like to see covered.

In this test, we have a single region server, which serves as S2 from your above example.

  1. At T0 you take a full backup B1 of all four tables at R1. boundary for our server is the timestamp of oldest of the wals of all the tables.
  2. Data is inserted into tables 1 and 2.
  3. At T1 you take incremental backup B1 of tables 1, 2, 3 to R1. boundary for our server doesn't change because the oldest wal is still from B1 and table 4.
  4. At T2 you take a full backup B2 of tables 1, 2, 3 to R2. boundary does not change because the oldest wal is still from B1 and table 4.

Based on my understanding, this test never sees a point when a wal can be discarded because wals from table4 are never rolled and so the wals identified for table4 in B1 are always the "oldest of the newest" for this server.

I think that we'd like to see a test that does what you described in your initial comment: observes the "rising tide" of wal timestamp on the server as backups across roots progress.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've majorly refactored & extended the testcase, and found a bug along the way.
@rmdmattingly @ndimiduk please take a look.

Path backupRoot1 = new Path(BACKUP_ROOT_DIR, "root1");
Path backupRoot2 = new Path(BACKUP_ROOT_DIR, "root2");

// #1 - create full backup for all tables
// Create full backup for all tables
LOG.info("create full backup image for all tables");

List<TableName> tableSetFullList = Lists.newArrayList(table1, table2, table3, table4);
Expand All @@ -71,44 +77,43 @@ public void testBackupLogCleaner() throws Exception {
// Verify that we have no backup sessions yet
assertFalse(systemTable.hasBackupSessions());

List<FileStatus> walFiles = getListOfWALFiles(TEST_UTIL.getConfiguration());
List<FileStatus> walFilesBeforeBackup = getListOfWALFiles(TEST_UTIL.getConfiguration());
BackupLogCleaner cleaner = new BackupLogCleaner();
cleaner.setConf(TEST_UTIL.getConfiguration());
Map<String, Object> params = new HashMap<>();
params.put(HMaster.MASTER, TEST_UTIL.getHBaseCluster().getMaster());
cleaner.init(params);
cleaner.setConf(TEST_UTIL.getConfiguration());

Iterable<FileStatus> deletable = cleaner.getDeletableFiles(walFiles);
int size = Iterables.size(deletable);

// We can delete all files because we do not have yet recorded backup sessions
assertTrue(size == walFiles.size());
Iterable<FileStatus> deletable = cleaner.getDeletableFiles(walFilesBeforeBackup);
int size = Iterables.size(deletable);
assertEquals(walFilesBeforeBackup.size(), size);

String backupIdFull = fullTableBackup(tableSetFullList);
// Create a FULL backup (backupRoot 1)
String backupIdFull = backupTables(BackupType.FULL, tableSetFullList, backupRoot1.toString());
assertTrue(checkSucceeded(backupIdFull));
// Check one more time
deletable = cleaner.getDeletableFiles(walFiles);
// We can delete wal files because they were saved into backup system table table
size = Iterables.size(deletable);
assertTrue(size == walFiles.size());

List<FileStatus> newWalFiles = getListOfWALFiles(TEST_UTIL.getConfiguration());
LOG.debug("WAL list after full backup");
// New list of WAL files is greater than the previous one,
// because new WAL per RS have been opened after full backup
Set<FileStatus> walFilesAfterFullBackup =
mergeAsSet(walFilesBeforeBackup, getListOfWALFiles(TEST_UTIL.getConfiguration()));
assertTrue(walFilesBeforeBackup.size() < walFilesAfterFullBackup.size());

// New list of wal files is greater than the previous one,
// because new wal per RS have been opened after full backup
assertTrue(walFiles.size() < newWalFiles.size());
// We can only delete the WALs preceding the FULL backup
deletable = cleaner.getDeletableFiles(walFilesAfterFullBackup);
size = Iterables.size(deletable);
assertEquals(walFilesBeforeBackup.size(), size);

// Insert some data
Connection conn = ConnectionFactory.createConnection(conf1);
DieterDP-ng marked this conversation as resolved.
Show resolved Hide resolved
// #2 - insert some data to table
Table t1 = conn.getTable(table1);
DieterDP-ng marked this conversation as resolved.
Show resolved Hide resolved
Put p1;
for (int i = 0; i < NB_ROWS_IN_BATCH; i++) {
p1 = new Put(Bytes.toBytes("row-t1" + i));
p1.addColumn(famName, qualName, Bytes.toBytes("val" + i));
t1.put(p1);
}

t1.close();

Table t2 = conn.getTable(table2);
DieterDP-ng marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -118,23 +123,49 @@ public void testBackupLogCleaner() throws Exception {
p2.addColumn(famName, qualName, Bytes.toBytes("val" + i));
t2.put(p2);
}

t2.close();

// #3 - incremental backup for multiple tables

// Create an INCREMENTAL backup (backupRoot 1)
List<TableName> tableSetIncList = Lists.newArrayList(table1, table2, table3);
String backupIdIncMultiple =
backupTables(BackupType.INCREMENTAL, tableSetIncList, BACKUP_ROOT_DIR);
backupTables(BackupType.INCREMENTAL, tableSetIncList, backupRoot1.toString());
assertTrue(checkSucceeded(backupIdIncMultiple));
deletable = cleaner.getDeletableFiles(newWalFiles);

assertTrue(Iterables.size(deletable) == newWalFiles.size());
// There should be more WALs due to the rolling of Region Servers
Set<FileStatus> walFilesAfterIncBackup =
mergeAsSet(walFilesAfterFullBackup, getListOfWALFiles(TEST_UTIL.getConfiguration()));
assertTrue(walFilesAfterFullBackup.size() < walFilesAfterIncBackup.size());

// We can only delete the WALs preceding the INCREMENTAL backup
deletable = cleaner.getDeletableFiles(walFilesAfterIncBackup);
size = Iterables.size(deletable);
assertEquals(walFilesAfterFullBackup.size(), size);

// Create a FULL backup (backupRoot 2)
String backupIdFull2 = backupTables(BackupType.FULL, tableSetIncList, backupRoot2.toString());
assertTrue(checkSucceeded(backupIdFull2));

// There should be more WALs due to the rolling of Region Servers
Set<FileStatus> walFilesAfterFullBackup2 =
mergeAsSet(walFilesAfterFullBackup, getListOfWALFiles(TEST_UTIL.getConfiguration()));
assertTrue(walFilesAfterIncBackup.size() < walFilesAfterFullBackup2.size());

// We created a backup in a different root, so the WAL dependencies of the first root did not
// change. I.e. the same files should be deletable as after the incremental backup.
DieterDP-ng marked this conversation as resolved.
Show resolved Hide resolved
deletable = cleaner.getDeletableFiles(walFilesAfterFullBackup2);
size = Iterables.size(deletable);
assertEquals(walFilesAfterFullBackup.size(), size);

conn.close();
}
}

private Set<FileStatus> mergeAsSet(Collection<FileStatus> toCopy, Collection<FileStatus> toAdd) {
Set<FileStatus> result = new HashSet<>(toCopy);
result.addAll(toAdd);
return result;
}

@Test
public void testCleansUpHMasterWal() {
Path path = new Path("/hbase/MasterData/WALs/hmaster,60000,1718808578163");
Expand Down