Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RheaKVStore#bContainsKey(byte[]) inconsistent results returned after Snapshot. #905

Closed
lfygh opened this issue Nov 18, 2022 · 3 comments · Fixed by #909
Closed

RheaKVStore#bContainsKey(byte[]) inconsistent results returned after Snapshot. #905

lfygh opened this issue Nov 18, 2022 · 3 comments · Fixed by #909
Assignees
Labels
bug Something isn't working

Comments

@lfygh
Copy link
Contributor

lfygh commented Nov 18, 2022

Describe the bug

RheaKVStore#bContainsKey(byte[]) inconsistent results returned after Snapshot.

1、put a key to RheaKVStore

2、wait background thread to do snapshot

3、call RheaKVStore#bContainsKey(byte[]) , it returns false (error, because I haven’t remove it);

4、but call RheaKVStore#bGet(byte[]), it returns the value.

5、call RheaKVStore#bContainsKey(byte[]) again, it returns true.

Expected behavior

call RheaKVStore#bContainsKey(byte[]) , it always returns true until explicitly remove it.
.

Actual behavior

call RheaKVStore#bContainsKey(byte[]) , it returns false after the snapshot.

Steps to reproduce

it did the test under jraft-example/src/main/java/com/alipay/sofa/jraft/example/rheakv package.

1、add options to server1,server2,server3

       NodeOptions nodeOptions = new NodeOptions(); // 1、new options
        nodeOptions.setSnapshotIntervalSecs(2);  // 2、smaller SnapshotInterval for fast do snapshot

        final RegionEngineOptions rOpts = new RegionEngineOptions();
        rOpts.setRegionId(Constants.DEFAULT_REGION_ID);
        rOpts.setNodeOptions(nodeOptions);
        List<RegionEngineOptions> rOptsList = Lists.newArrayList();
        rOptsList.add(rOpts);
        final PlacementDriverOptions pdOpts = PlacementDriverOptionsConfigured.newConfigured()
                .withFake(true) // use a fake pd
                .config();
        final StoreEngineOptions storeOpts1 = StoreEngineOptionsConfigured.newConfigured() //
                .withStorageType(StorageType.RocksDB)
                .withRocksDBOptions(RocksDBOptionsConfigured.newConfigured().withDbPath(Configs.DB_PATH).config())
                .withRaftDataPath(Configs.RAFT_DATA_PATH)
                .withServerAddress(new Endpoint("127.0.0.1", 8181))
                .withCommonNodeOptions(nodeOptions) //  3、add nodeOptions
                .withRegionEngineOptionsList(rOptsList)
                .config();

2、example class

package com.alipay.sofa.jraft.example.rheakv;

public class Example {
    private static final Logger LOG = LoggerFactory.getLogger(BContainsKeyFalseExample.class);
    
    public static void main(final String[] args) throws Exception {
        final Client client = new Client();
        client.init();
        get(client.getRheaKVStore());
        client.shutdown();
    }

    public static void get(final RheaKVStore rheaKVStore) throws InterruptedException {

        final byte[] key = writeUtf8("hello");
        final byte[] value = writeUtf8("world");
        rheaKVStore.bPut(key, value);

        for (int i = 0; i < 5; i++) {
            TimeUnit.SECONDS.sleep(1);
            final boolean b= rheaKVStore.bContainsKey(key);
            LOG.info("Sync bContainsKey result={}", b);

            final byte[] b3 = rheaKVStore.bGet(key);
            LOG.info("Sync bGet result={}", readUtf8(b3));
        }
    }
}
2022-11-18 17:01:10 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:10 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:11 [main] INFO  Example:31 - Sync bContainsKey result = false
2022-11-18 17:01:11 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:12 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:12 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:13 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:13 [main] INFO  Example:34 - Sync bGet result = world
2022-11-18 17:01:14 [main] INFO  Example:31 - Sync bContainsKey result = true
2022-11-18 17:01:14 [main] INFO  Example:34 - Sync bGet result = world

More

I copied the code from com.alipay.sofa.jraft.rhea.storage.RocksRawKVStore#backupDB, and use RocksDB API
to shrink the scope, yes, it is reproduced.

Are there some options I did not set? or is it a bug?

Minimal yet complete reproducer code (or GitHub URL to code)

Environment

  • SOFAJRaft version:
    fork from the GitHub master branch

  • JVM version (e.g. java -version):

    openjdk version "17.0.4" 2022-07-19 LTS
    OpenJDK Runtime Environment Zulu17.36+13-CA (build 17.0.4+8-LTS)
    OpenJDK 64-Bit Server VM Zulu17.36+13-CA (build 17.0.4+8-LTS, mixed mode, sharing)

  • OS version (e.g. uname -a):
    Darwin MacBook-Pro.local 20.5.0 Darwin Kernel Version 20.5.0: Sat May 8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 arm64

  • Maven version:

  • IDE version:

@killme2008
Copy link
Contributor

@fengjiachun PTAL

@fengjiachun
Copy link
Contributor

@lfygh

Thank you very much, I reproduced this bug, I will check the exact cause and reply later.

@fengjiachun
Copy link
Contributor

@lfygh

Thanks a lot, this is a bug, during one of the rocksdb upgrades, we misunderstood the new API of rocksdb, resulting in a wrong way to use it: The valueHolder only return the value when the key is found in memory.

See

I will fix it and release a new version ASAP.

@fengjiachun fengjiachun self-assigned this Nov 28, 2022
@fengjiachun fengjiachun added the bug Something isn't working label Nov 28, 2022
@fengjiachun fengjiachun mentioned this issue Dec 13, 2022
10 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants