CompactionFilterFactory As SidePlugin

[中文版 (Chinese)]

Background

In slightly more complicated RocksDB applications, the use of CompactionFilter is indispensable. Although the CompactionFilter object can be directly set in Options, we still use CompactionFilterFactory more, because CompactionFilter is often stateful rather than a pure function, so it needs to be dynamic created and maintain its state in a single subcompact.

Because C++ has no language-level reflection, to manage and configure CompactionFilterFactory through ToplingDB's SidePlugin, we need to modify a little code, mainly to register our custom CompactionFilterFactory with the SidePlugin system.

In addition to CompactionFilterFactory, there are many other objects that need to be incorporated into the SidePlugin system in a similar way, such as Comparator, MergeOperator, EventHandler .... Because CompactionFilterFactory is more commonly used than other objects, we use CompactionFilterFactory as an example to expand here.

Migrate to ToplingDB, mainly to register plugins to SidePlugin

// file my_filter.cc
//
#include <topling/side_plugin_factory.h> // To register a plugin, this header file is required

// MySimpleCompactionFilterFactory is existing code and does not require any changes
class MySimpleCompactionFilterFactory : public CompactionFilterFactory {
public:
  // implementation details
};

// Add this line to register the plugin, MySimpleCompactionFilterFactory must have a default constructor
ROCKSDB_REG_DEFAULT_CONS(MySimpleCompactionFilterFactory, CompactionFilterFactory);

// MyComplexCompactionFilterFactory is existing code and does not require any changes
class MyComplexCompactionFilterFactory : public CompactionFilterFactory {
  std::string str_conf_option;
  int int_conf_option = 0; // 0 is default
public:
  // implementation details

  // Originally need to create and configure this complex MyComplexCompactionFilterFactory in user code,
  // but now str_conf_option and int_conf_option can be configured in json/yaml
  MyComplexCompactionFilterFactory(const json& js, const SidePluginRepo& repo) {
    ROCKSDB_JSON_OPT_PROP(js, str_conf_option);
    ROCKSDB_JSON_OPT_PROP(js, int_conf_option);
  }
};

// Add this line. Registering the plugin requires a constructor (const json&, const SidePluginRepo&)
ROCKSDB_REG_JSON_REPO_CONS(MyComplexCompactionFilterFactory, CompactionFilterFactory);

Note: After the plugin is registered, all symbols in this file (my_filter.cc) may not be referenced by codes in other files. In this case, if my_filter.o is compiled as a static link library, when the static link library is linked to the executable file, my_filter.o will be deleted by the linker. So the plugin must be linked in the form of .o or dynamic library (.so), and cannot be linked in the form of static library!

Then, in the yaml configuration file (you can try to convert this yaml to json yourself if you want):

CompactionFilterFactory:
  simple_filter1: MySimpleCompactionFilterFactory
  simple_filter2:
    class: MySimpleCompactionFilterFactory
    params: # defining params is not required here
  complex_filter:
    class: MyComplexCompactionFilterFactory
    params:
      str_conf_option: some string
      int_conf_option: 123
DBOptions:
  dbo:
    create_if_missing: true
CFOptions:
  simple_cf:
    compaction_filter_factory: "$simple_filter1"
  complex_cf:
    compaction_filter_factory: "$complex_filter"
databases:
  mydb1:
    method: DB::Open
    params:
      db_options: "$dbo"
      column_families:
        default: "$simple_cf"
        complex: "$complex_cf"
      path: /path/to/mydb1
  mydb2:
    method: DB::Open
    params:
      db_options: "$dbo"
      column_families:
        default: "$simple_cf"
        complex: "$complex_cf"
      path: /path/to/mydb2

Finally, according to document 101, import the configuration and open the db.

In addition, there is a case where you do not need to register, but define plugin objects through Ingest Predefine Object.