BlockBasedTableConfig is not loaded from Options file

[caronzh03]

Expected behavior

Given an options file auto-generated by a previously built rocksdb instance, OptionsUtil.loadLatestOptions() should load that options file and deserialize the [TableOptions/BlockBasedTable "default"] section into a BlockBasedTableConfig object.

Actual behavior

OptionsUtil.loadLatestOptions() does not parse the [TableOptions/BlockBasedTable "default"] section of a given Options file. As a result, the TableFormatConfig field in each ColumnFamilyOption is null.

Steps to reproduce the behavior

I'm using rocksdb's Java library.

// Construct DBOptions by loading an Options file on disk
DBOptions loadedDBOptions = new DBOptions();
List<ColumnFamilyDescriptor> loadedCFDescriptors = new ArrayList<>();
OptionsUtil.loadLatestOptions(rocksDbPath, Env.getDefault(), loadedDBOptions, loadedCFDescriptors);

// check if `TableFormatConfig` object is correctly populated in each `ColumnFamilyOption`
TableFormatConfig tableFormatConfig = loadedCFDescriptors.get(0).getOptions().tableFormatConfig();

// tableFormatConfig == null !

Here's the auto-generated Options file: https://gist.github.com/caronzh03/317e2e703a0261f1710c50a79ba5d81e

According to wiki, the BlockBasedTableOptions should be deserialized, no?

[mrambacher]

@sagar0 @adamretter Is this still an issue?

[adamretter]

@mrambacher I could take a look at this in the next few days if you want confirmation?

[adamretter]

@caronzh03 I think there are two issues here:

1. The Java and C++ APIs have diverged here somewhat. Java talks about TableConfigs but C++ talks about TableFactory.

2. The Java API for TableConfigs underneath relies on the C++ API, so we are (currently) limited to what the C++ API allows us to read back in-terms of options.

The C++ API does not allow you to directly retrieve the options for the TableFactory, it does allow you to print out the options that have been loaded though. I wrote out some C++ which is equivalent to your Java and that shows that the options are loaded from your options file:

#include "rocksdb/db.h"
#include "rocksdb/env.h"
#include "rocksdb/options.h"
#include "rocksdb/utilities/options_util.h"
#include "table/block_based/block_based_table_factory.h"

#include <cstring>
#include <iostream>
#include <string>

using namespace ROCKSDB_NAMESPACE;
using namespace std;

int main(int /*argv*/, char*[] /*argc[]*/) {

    string options_file_name("/tmp/rocksdb_options_file.ini");
    Env* env = Env::Default();

    DBOptions* db_options = new DBOptions();
    vector<ColumnFamilyDescriptor>* cf_descs = new vector<ColumnFamilyDescriptor>();

    //LoadLatestOptions
    Status s = LoadOptionsFromFile(options_file_name, env, db_options, cf_descs);
    cout << "OK=" << s.ok() << endl;

    // check if `TableFormatConfig` object is correctly populated in each `ColumnFamilyOption`
    ColumnFamilyDescriptor default_cf = cf_descs->at(0);
    shared_ptr<TableFactory> table_factory = default_cf.options.table_factory;
    cout << "table_factory(is null)=" << (table_factory.get() == nullptr) << endl;

    cout << "## table_factory: " << table_factory->Name() << endl;

    if (strcmp(table_factory->Name(), TableFactory::kBlockBasedTableName()) == 0) {
        auto* block_base_table_factory = reinterpret_cast<BlockBasedTableFactory*>(table_factory.get());
        string printable_options = block_base_table_factory->GetPrintableOptions();
        
        cout << "### printable_options" << endl;
        cout << printable_options << endl;
    } else {
        cout << "## NOT IMPLEMENTED" << endl;
    }

    return 0;
}

I think it would be best here to align the Java API to that of the C++ API, and I have opened a new issue for that - #9101

If the C++ API needs to offer more in future than it does currently, then I think it would be better to discuss that as a separate issue.

[mrambacher]

@caronzh03 I think there are two issues here:

1. The Java and C++ APIs have diverged here somewhat. Java talks about TableConfigs but C++ talks about TableFactory.
2. The Java API for TableConfigs underneath relies on the C++ API, so we are (currently) limited to what the C++ API allows us to read back in-terms of options.

The C++ API does not allow you to directly retrieve the options for the TableFactory, it does allow you to print out the options that have been loaded though. I wrote out some C++ which is equivalent to your Java and that shows that the options are loaded from your options file:

#include "rocksdb/db.h"
#include "rocksdb/env.h"
#include "rocksdb/options.h"
#include "rocksdb/utilities/options_util.h"
#include "table/block_based/block_based_table_factory.h"

#include <cstring>
#include <iostream>
#include <string>

using namespace ROCKSDB_NAMESPACE;
using namespace std;

int main(int /*argv*/, char*[] /*argc[]*/) {

    string options_file_name("/tmp/rocksdb_options_file.ini");
    Env* env = Env::Default();

    DBOptions* db_options = new DBOptions();
    vector<ColumnFamilyDescriptor>* cf_descs = new vector<ColumnFamilyDescriptor>();

    //LoadLatestOptions
    Status s = LoadOptionsFromFile(options_file_name, env, db_options, cf_descs);
    cout << "OK=" << s.ok() << endl;

    // check if `TableFormatConfig` object is correctly populated in each `ColumnFamilyOption`
    ColumnFamilyDescriptor default_cf = cf_descs->at(0);
    shared_ptr<TableFactory> table_factory = default_cf.options.table_factory;
    cout << "table_factory(is null)=" << (table_factory.get() == nullptr) << endl;

    cout << "## table_factory: " << table_factory->Name() << endl;

    if (strcmp(table_factory->Name(), TableFactory::kBlockBasedTableName()) == 0) {
        auto* block_base_table_factory = reinterpret_cast<BlockBasedTableFactory*>(table_factory.get());
        string printable_options = block_base_table_factory->GetPrintableOptions();
        
        cout << "### printable_options" << endl;
        cout << printable_options << endl;
    } else {
        cout << "## NOT IMPLEMENTED" << endl;
    }

    return 0;
}

I think it would be best here to align the Java API to that of the C++ API, and I have opened a new issue for that - #9101

If the C++ API needs to offer more in future than it does currently, then I think it would be better to discuss that as a separate issue.

@adamretter You can do (instead of reinterpret_cast).
auto *block_based_table_factory = table_factory->CheckedCast();
If the table factory is not a BlockBasedTableFactory, it will return nullptr.

Additionally, you can get at the underlying options object (but are not allowed to modify it):
auto *block_based_table_options = table_factory->GetOptions();
Again, if the class is not a BlockBasedTableFactory or does not have BlockBasedTableOptions, this will return nullptr.

I do agree that the C++ and Java APIs have strayed and should be unified.