Kyoto Cabinet

Implementing MAF indexing with Kyoto Cabinet has gone surprisingly well. It’s now functional, and while I haven’t tested performance yet, it seems promising.

Storing index data in binary form means the keys are 12 bytes and the values are 16 bytes, for a total of 28 bytes per record. Database overhead is minimal; an index for a 315 MB MAF file with 95437 records is 3.2 MB, or about 36 bytes per record.

Installation

Mac OS X 10.7.4 with Homebrew

Tested with:

• Oracle Java SE 7 JDK for Mac OS X from Oracle (recommended)
  • Open /Applications/Utilities/Java Preferences.app and set JDK 1.7.0 as the default
• XCode 4.3.2
• Install XCode command line tools (Preferences > Downloads > Components)

Installation:

• brew install kyoto-cabinet

Ubuntu 12.04 LTS 64-bit (precise)

• apt-get install liblzo2-dev liblzma-dev zlib1g-dev build-essential
• wget http://fallabs.com/kyotocabinet/pkg/kyotocabinet-1.2.76.tar.gz
• tar xzf kyotocabinet-1.2.76.tar.gz
• cd kyotocabinet-1.2.76
• ./configure –enable-zlib –enable-lzo –enable-lzma && make
• make install

Debian testing (wheezy) 32-bit

• apt-get install liblzo2-dev liblzma-dev build-essential
• wget http://fallabs.com/kyotocabinet/pkg/kyotocabinet-1.2.76.tar.gz
• tar xzf kyotocabinet-1.2.76.tar.gz
• cd kyotocabinet-1.2.76
• ./configure –enable-zlib –enable-lzo –enable-lzma && make
• make install

Fedora 17 64-bit

• NOTE: this installs a version that is slightly out of date and won’t work with kyotocabinet-java.
• yum install kyotocabinet-devel