Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-22623 - Add RegionObserver coprocessor hook for preWALAppend #390

Merged
merged 2 commits into from
Aug 9, 2019
Merged

HBASE-22623 - Add RegionObserver coprocessor hook for preWALAppend #390

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6f148b4
HBASE-22623 - Add RegionObserver coprocessor hook for preWALAppend
gjacoby126 Jul 17, 2019
797b56a
HBASE-22623 - Add RegionObserver coprocessor hook for preWALAppend
gjacoby126 Aug 7, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1104,4 +1104,16 @@ default DeleteTracker postInstantiateDeleteTracker(
throws IOException {
return delTracker;
}

/**
* Called just before the WAL Entry is appended to the WAL. Implementing this hook allows
* coprocessors to add extended attributes to the WALKey that then get persisted to the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having trouble understanding 'add extended attributes to the WALKey'. WALKey is read-only. You mean WALEdit here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WALKey is no longer read only. You can get and set extended attributes after HBASE-22622 WALKey Extended Attributes (#352)

* WAL, and are available to replication endpoints to use in processing WAL Entries.
* @param ctx the environment provided by the region server
* @param key the WALKey associated with a particular append to a WAL
*/
default void preWALAppend(ObserverContext<RegionCoprocessorEnvironment> ctx, WALKey key,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please mark this method as deprecated so we keep consistent on that WALEdit should not be exposed directly.

And no postWALAppend seems a bit strange to users but I think it is fine here. We can add it when we actually want to use it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Apache9 Just to make sure I understand you right...you want me to write a method which is Deprecated, as of the moment of its creation?

Adding a method to a public interface tells people they can use it. Deprecating a method tells people they should stop using it or refrain from starting to. Adding a deprecated method...just adds dead code.

Someday soon, there's going to be a Phoenix code review for changes I'll make that use this method, and someone's going to say, "You can't use this method; it's deprecated. -1" And they'll be completely right to say so.

Huh?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phoenix already uses lots of deprecated methods in HBase, and I believe you can not find alternate solutions, for example, this one

  /**
   * Called before creation of Reader for a store file.
   * Calling {@link org.apache.hadoop.hbase.coprocessor.ObserverContext#bypass()} has no
   * effect in this hook.
   *
   * @param ctx the environment provided by the region server
   * @param fs fileystem to read from
   * @param p path to the file
   * @param in {@link FSDataInputStreamWrapper}
   * @param size Full size of the file
   * @param cacheConf
   * @param r original reference file. This will be not null only when reading a split file.
   * @param reader the base reader, if not {@code null}, from previous RegionObserver in the chain
   * @return a Reader instance to use instead of the base reader if overriding
   * default behavior, null otherwise
   * @deprecated For Phoenix only, StoreFileReader is not a stable interface.
   */
  @Deprecated
  // Passing InterfaceAudience.Private args FSDataInputStreamWrapper, CacheConfig and Reference.
  // This is fine as the hook is deprecated any way.
  default StoreFileReader preStoreFileReaderOpen(ObserverContext<RegionCoprocessorEnvironment> ctx,
      FileSystem fs, Path p, FSDataInputStreamWrapper in, long size, CacheConfig cacheConf,
      Reference r, StoreFileReader reader) throws IOException {
    return reader;
  }

You can add javadoc to say that now it is only supposed to be used in Phoenix, but I still think we should mark it as deprecated, otherwise it will be confusing that why in WALObserver, we say that WALEdit is private and should not be used, but then in RegionObserver, we allow users to use it.

Copy link
Contributor Author

@gjacoby126 gjacoby126 Aug 8, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal isn't to create a Phoenix-specific hack, but a general-purpose HBase API which Phoenix can use. You're at the same time complaining about Phoenix's previous and existing use of unapproved hooks while resisting efforts to create approved APIs the right way.

preStoreFileReaderOpen wasn't created deprecated -- it's still alive and well in branch-1 -- but was deprecated in HBASE-18793, which you know because you were the one who deprecated it.

Given that I'm backporting this new hook to branch-1, if I create it deprecated that presumably means that as soon as 1.5 releases, the new hook is eligible for cleanup in master via semver rules before it's even been released!

The comments are confusing, not because of this patch, but because the position that created those comments is already self-contradictory in existing code. WALEdit must never be exposed to coprocessors...except the 6 times it already is in RegionObserver and the 4 times it is in WALObserver.

When a policy leads to increasingly strange and nonsensical results -- and creating an already deprecated method is nonsensical -- it's time to rethink the policy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new method is not going to be deprecated out of the bat.

We are not monolithic in our approach (and hostility) to coprocessor interfaces as a community. Imposing that disagreement on contributors is not appropriate.

I am going to merge this as is and we can follow up on what should or should not be deprecated as a larger conversation on the future of coprocessors. The community has some big disagreements in approach.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also agree the "policy", such as it is, is contradictory and confusing. We need to attack the bigger picture on dev@ in a discussion about the future of coprocessors and our tolerance (or not) to the requests of the Phoenix project. The opinions are not monolithic. There are some supporters, there are some hostile positions, both are valid in my view, we need to sort out the disagreement. This issue isn't the right scope for that. The contradictory positions are evident in the tug and pull of the suggestions to the contributor.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copied these comments over onto the JIRA for visibility.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I sent an email with the subject "Coprocessors, clean ups, compatibility, deprecations, Phoenix... it's a bit of a mess" to dev@

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting 24 hours for time zone turn around.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API will be used by adding to the passed in WALEdit? Using WALEdit #setters?

WALEdit edit)
throws IOException {
}
}
3 changes: 3 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7951,6 +7951,9 @@ private WriteEntry doWALAppend(WALEdit walEdit, Durability durability, List<UUID
if (walEdit.isReplay()) {
walKey.setOrigLogSeqNum(origLogSeqNum);
}
if (this.coprocessorHost != null) {
this.coprocessorHost.preWALAppend(walKey, walEdit);
}
WriteEntry writeEntry = null;
try {
long txid = this.wal.append(this.getRegionInfo(), walKey, walEdit, true);
Expand Down
12 changes: 12 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1720,6 +1720,18 @@ public List<Pair<Cell, Cell>> call(RegionObserver observer) throws IOException {
});
}

public void preWALAppend(WALKey key, WALEdit edit) throws IOException {
if (this.coprocEnvironments.isEmpty()){
return;
}
execOperation(new RegionObserverOperationWithoutResult() {
@Override
public void call(RegionObserver observer) throws IOException {
observer.preWALAppend(this, key, edit);
}
});
}

public Message preEndpointInvocation(final Service service, final String methodName,
Message request) throws IOException {
if (coprocEnvironments.isEmpty()) {
Expand Down
5 changes: 0 additions & 5 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALEdit.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,12 @@
* Used in HBase's transaction log (WAL) to represent a collection of edits (Cell/KeyValue objects)
* that came in as a single transaction. All the edits for a given transaction are written out as a
* single record, in PB format, followed (optionally) by Cells written via the WALCellEncoder.
* <p>This class is LimitedPrivate for CPs to read-only. The {@link #add} methods are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to ask you to back this out, but discussion on the JIRA ratholed on whether or not WALEdit should be immutable or not. We can keep the hook signature as proposed, because WALEdit is useful to the coprocessor even if read only, and this would allow the discussion on WALEdit to be tabled until some future time without consequence to today's need for this change.

* classified as private methods, not for use by CPs.</p>
* <p>WALEdit will accumulate a Set of all column family names referenced by the Cells
* {@link #add(Cell)}'d. This is an optimization. Usually when loading a WALEdit, we have the
* column family name to-hand.. just shove it into the WALEdit if available. Doing this, we can
* save on a parse of each Cell to figure column family down the line when we go to add the
* WALEdit to the WAL file. See the hand-off in FSWALEntry Constructor.
*/
// TODO: Do not expose this class to Coprocessors. It has set methods. A CP might meddle.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

@InterfaceAudience.LimitedPrivate({ HBaseInterfaceAudience.REPLICATION,
HBaseInterfaceAudience.COPROC })
public class WALEdit implements HeapSize {
Expand Down Expand Up @@ -163,13 +160,11 @@ public boolean isReplay() {
return this.replay;
}

@InterfaceAudience.Private
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

public WALEdit add(Cell cell, byte [] family) {
getOrCreateFamilies().add(family);
return addCell(cell);
}

@InterfaceAudience.Private
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same

public WALEdit add(Cell cell) {
// We clone Family each time we add a Cell. Expensive but safe. For CPU savings, use
// add(Map) or add(Cell, family).
Expand Down
8 changes: 7 additions & 1 deletion hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@

/**
* Key for WAL Entry.
* Read-only. No Setters. For limited audience such as Coprocessors.
*/
@InterfaceAudience.LimitedPrivate({HBaseInterfaceAudience.REPLICATION,
HBaseInterfaceAudience.COPROC})
Expand Down Expand Up @@ -86,6 +85,13 @@ default long getNonce() {
*/
long getOrigLogSeqNum();

/**
* Add a named String value to this WALKey to be persisted into the WAL
* @param attributeKey Name of the attribute
* @param attributeValue Value of the attribute
*/
void addExtendedAttribute(String attributeKey, byte[] attributeValue);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so WALKey goes from being read-only to now carrying burden? WALEdit is for freight?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I've made this remark before... Looking for the response then, I see loads of commentary in issue... Let me review.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, don't find where its ok to add freight to WALKey marked read-only (but alternative of letting WALEdit be mutable is a 'rat-hole'.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WALKey is not read only.

WALedit is not where we want to add the state

The WALKey change is already committed.

Having a WALedit parameter on this hook is useful whether it is read only or not.

Still going to merge this now.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is the second half of work that began with HBASE-22622 WALKey Extended Attributes (#352)


/**
* Return a named String value injected into the WALKey during processing, such as by a
* coprocessor
Expand Down
39 changes: 39 additions & 0 deletions hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKeyImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,37 @@ public WALKeyImpl(final byte[] encodedRegionName,
mvcc, null, null);
}

/**
* Copy constructor that takes in an existing WALKeyImpl plus some extended attributes.
* Intended for coprocessors to add annotations to a system-generated WALKey
* for persistence to the WAL.
* @param key Key to be copied into this new key
* @param extendedAttributes Extra attributes to copy into the new key
*/
public WALKeyImpl(WALKeyImpl key,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of CP creating a new WALKey object, we can give API to add the ext attributes? It allows to add any custom key value pair so that later CPs or Replication EPs an make use. So we might not even allow the CP to return a brand new WALKey object. (?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also WALKeyImpl is marked LimitedPrivate({HBaseInterfaceAudience.REPLICATION}) I can see. So what this patch trying to do is to expose the class for CPs also. I can not see where WALKeyImpl to be used by Replication area. WALKey was enough. Anyways for this CP hook, IMO its enough to pass WALKey interface. We can add the API to add ext attributes into the interface. That is just adding ext attributes. It should be fine as it is not exposing any setters which takes other key attributes of the object.

Map<String, byte[]> extendedAttributes){
init(key.getEncodedRegionName(), key.getTableName(), key.getSequenceId(),
key.getWriteTime(), key.getClusterIds(), key.getNonceGroup(), key.getNonce(),
key.getMvcc(), key.getReplicationScopes(), extendedAttributes);

}

/**
* Copy constructor that takes in an existing WALKey, the extra WALKeyImpl fields that the
* parent interface is missing, plus some extended attributes. Intended
* for coprocessors to add annotations to a system-generated WALKey for
* persistence to the WAL.
*/
public WALKeyImpl(WALKey key,
List<UUID> clusterIds,
MultiVersionConcurrencyControl mvcc,
final NavigableMap<byte[], Integer> replicationScopes,
Map<String, byte[]> extendedAttributes){
init(key.getEncodedRegionName(), key.getTableName(), key.getSequenceId(),
key.getWriteTime(), clusterIds, key.getNonceGroup(), key.getNonce(),
mvcc, replicationScopes, extendedAttributes);

}
/**
* Create the log key for writing to somewhere.
* We maintain the tablename mainly for debugging purposes.
Expand Down Expand Up @@ -464,6 +495,14 @@ public UUID getOriginatingClusterId(){
return clusterIds.isEmpty()? HConstants.DEFAULT_CLUSTER_ID: clusterIds.get(0);
}

@Override
public void addExtendedAttribute(String attributeKey, byte[] attributeValue){
if (extendedAttributes == null){
extendedAttributes = new HashMap<String, byte[]>();
}
extendedAttributes.put(attributeKey, attributeValue);
}

@Override
public byte[] getExtendedAttribute(String attributeKey){
return extendedAttributes != null ? extendedAttributes.get(attributeKey) : null;
Expand Down
18 changes: 18 additions & 0 deletions hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import static org.junit.Assert.assertTrue;

import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
Expand Down Expand Up @@ -124,7 +125,11 @@ public class SimpleRegionObserver implements RegionCoprocessor, RegionObserver {
final AtomicInteger ctPostStartRegionOperation = new AtomicInteger(0);
final AtomicInteger ctPostCloseRegionOperation = new AtomicInteger(0);
final AtomicBoolean throwOnPostFlush = new AtomicBoolean(false);
final AtomicInteger ctPreWALAppend = new AtomicInteger(0);

static final String TABLE_SKIPPED = "SKIPPED_BY_PREWALRESTORE";
Map<String, byte[]> extendedAttributes = new HashMap<String,byte[]>();
static final byte[] WAL_EXTENDED_ATTRIBUTE_BYTES = Bytes.toBytes("foo");

public void setThrowOnPostFlush(Boolean val){
throwOnPostFlush.set(val);
Expand Down Expand Up @@ -631,6 +636,15 @@ public StoreFileReader postStoreFileReaderOpen(ObserverContext<RegionCoprocessor
return reader;
}

@Override
public void preWALAppend(ObserverContext<RegionCoprocessorEnvironment> ctx,
WALKey key, WALEdit edit) throws IOException {
ctPreWALAppend.incrementAndGet();

key.addExtendedAttribute(Integer.toString(ctPreWALAppend.get()),
Bytes.toBytes("foo"));
}

public boolean hadPreGet() {
return ctPreGet.get() > 0;
}
Expand Down Expand Up @@ -864,6 +878,10 @@ public int getCtPostWALRestore() {
return ctPostWALRestore.get();
}

public int getCtPreWALAppend() {
return ctPreWALAppend.get();
}

public boolean wasStoreFileReaderOpenCalled() {
return ctPreStoreFileReaderOpen.get() > 0 && ctPostStoreFileReaderOpen.get() > 0;
}
Expand Down
86 changes: 86 additions & 0 deletions ...server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import java.io.IOException;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import org.apache.hadoop.conf.Configuration;
Expand Down Expand Up @@ -70,20 +71,26 @@
import org.apache.hadoop.hbase.regionserver.StoreFile;
import org.apache.hadoop.hbase.regionserver.compactions.CompactionLifeCycleTracker;
import org.apache.hadoop.hbase.regionserver.compactions.CompactionRequest;
import org.apache.hadoop.hbase.regionserver.wal.WALActionsListener;
import org.apache.hadoop.hbase.testclassification.CoprocessorTests;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.tool.BulkLoadHFiles;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.hbase.util.EnvironmentEdgeManager;
import org.apache.hadoop.hbase.util.JVMClusterUtil;
import org.apache.hadoop.hbase.util.Threads;
import org.apache.hadoop.hbase.wal.WALEdit;
import org.apache.hadoop.hbase.wal.WALKey;
import org.apache.hadoop.hbase.wal.WALKeyImpl;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.rules.TestName;
import org.mockito.Mockito;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

Expand Down Expand Up @@ -663,6 +670,66 @@ public void testPreWALRestoreSkip() throws Exception {
table.close();
}

//called from testPreWALAppendIsWrittenToWAL
private void testPreWALAppendHook(Table table, TableName tableName) throws IOException {
int expectedCalls = 0;
String [] methodArray = new String[1];
methodArray[0] = "getCtPreWALAppend";
Object[] resultArray = new Object[1];

Put p = new Put(ROW);
p.addColumn(A, A, A);
table.put(p);
resultArray[0] = ++expectedCalls;
verifyMethodResult(SimpleRegionObserver.class, methodArray, tableName, resultArray);

Append a = new Append(ROW);
a.addColumn(B, B, B);
table.append(a);
resultArray[0] = ++expectedCalls;
verifyMethodResult(SimpleRegionObserver.class, methodArray, tableName, resultArray);

Increment i = new Increment(ROW);
i.addColumn(C, C, 1);
table.increment(i);
resultArray[0] = ++expectedCalls;
verifyMethodResult(SimpleRegionObserver.class, methodArray, tableName, resultArray);

Delete d = new Delete(ROW);
table.delete(d);
resultArray[0] = ++expectedCalls;
verifyMethodResult(SimpleRegionObserver.class, methodArray, tableName, resultArray);
}

@Test
public void testPreWALAppend() throws Exception {
SimpleRegionObserver sro = new SimpleRegionObserver();
ObserverContext ctx = Mockito.mock(ObserverContext.class);
WALKey key = new WALKeyImpl(Bytes.toBytes("region"), TEST_TABLE,
EnvironmentEdgeManager.currentTime());
WALEdit edit = new WALEdit();
sro.preWALAppend(ctx, key, edit);
Assert.assertEquals(1, key.getExtendedAttributes().size());
Assert.assertArrayEquals(SimpleRegionObserver.WAL_EXTENDED_ATTRIBUTE_BYTES,
key.getExtendedAttribute(Integer.toString(sro.getCtPreWALAppend())));
}

@Test
public void testPreWALAppendIsWrittenToWAL() throws Exception {
final TableName tableName = TableName.valueOf(TEST_TABLE.getNameAsString() +
"." + name.getMethodName());
Table table = util.createTable(tableName, new byte[][] { A, B, C });

PreWALAppendWALActionsListener listener = new PreWALAppendWALActionsListener();
List<HRegion> regions = util.getHBaseCluster().getRegions(tableName);
//should be only one region
HRegion region = regions.get(0);
region.getWAL().registerWALActionsListener(listener);
testPreWALAppendHook(table, tableName);
boolean[] expectedResults = {true, true, true, true};
Assert.assertArrayEquals(expectedResults, listener.getWalKeysCorrectArray());

}
// check each region whether the coprocessor upcalls are called or not.
private void verifyMethodResult(Class<?> coprocessor, String methodName[], TableName tableName,
Object value[]) throws IOException {
Expand Down Expand Up @@ -711,4 +778,23 @@ private static void createHFile(Configuration conf, FileSystem fs, Path path, by
writer.close();
}
}

private static class PreWALAppendWALActionsListener implements WALActionsListener {
boolean[] walKeysCorrect = {false, false, false, false};

@Override
public void postAppend(long entryLen, long elapsedTimeMillis,
WALKey logKey, WALEdit logEdit) throws IOException {
for (int k = 0; k < 4; k++) {
if (!walKeysCorrect[k]) {
walKeysCorrect[k] = Arrays.equals(SimpleRegionObserver.WAL_EXTENDED_ATTRIBUTE_BYTES,
logKey.getExtendedAttribute(Integer.toString(k + 1)));
}
}
}

boolean[] getWalKeysCorrectArray() {
return walKeysCorrect;
}
}
}
12 changes: 11 additions & 1 deletion hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,7 @@
import org.apache.hadoop.hbase.Waiter;
import org.apache.hadoop.hbase.client.Append;
import org.apache.hadoop.hbase.client.ColumnFamilyDescriptor;
import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
import org.apache.hadoop.hbase.client.Delete;
import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.Get;
Expand Down Expand Up @@ -165,7 +166,6 @@
import org.apache.hadoop.hbase.wal.WALProvider;
import org.apache.hadoop.hbase.wal.WALProvider.Writer;
import org.apache.hadoop.hbase.wal.WALSplitUtil;
import org.apache.hadoop.hbase.wal.WALSplitter;
import org.apache.hadoop.metrics2.MetricsExecutor;
import org.junit.After;
import org.junit.Assert;
Expand Down Expand Up @@ -401,6 +401,7 @@ public void testMemstoreSizeAccountingWithFailedPostBatchMutate() throws IOExcep
String testName = "testMemstoreSizeAccountingWithFailedPostBatchMutate";
FileSystem fs = FileSystem.get(CONF);
Path rootDir = new Path(dir + testName);
ChunkCreator.initialize(MemStoreLABImpl.CHUNK_SIZE_DEFAULT, false, 0, 0, 0, null);
FSHLog hLog = new FSHLog(fs, rootDir, testName, CONF);
hLog.init();
region = initHRegion(tableName, null, null, false, Durability.SYNC_WAL, hLog,
Expand Down Expand Up @@ -2427,7 +2428,16 @@ public Void answer(InvocationOnMock invocation) throws Throwable {
return null;
}
}).when(mockedCPHost).preBatchMutate(Mockito.isA(MiniBatchOperationInProgress.class));
ColumnFamilyDescriptorBuilder builder = ColumnFamilyDescriptorBuilder.
newBuilder(COLUMN_FAMILY_BYTES);
ScanInfo info = new ScanInfo(CONF, builder.build(), Long.MAX_VALUE,
Long.MAX_VALUE, region.getCellComparator());
Mockito.when(mockedCPHost.preFlushScannerOpen(Mockito.any(HStore.class),
Mockito.any())).thenReturn(info);
Mockito.when(mockedCPHost.preFlush(Mockito.any(), Mockito.any(StoreScanner.class),
Mockito.any())).thenAnswer(i -> i.getArgument(1));
region.setCoprocessorHost(mockedCPHost);

region.put(originalPut);
region.setCoprocessorHost(normalCPHost);
final long finalSize = region.getDataInMemoryWithoutWAL();
Expand Down