iibench

The Indexed insertion Benchmark (short iiBench) for document stores like MongoDB and OrientDB assesses the behavior of an index under permanent load (concurrent inserts) and queries on the same index at the same time. The original benchmark was developed by Percona (cf. https://github.com/tmcallaghan/iibench-mongodb), specifically targetting MongoDB and last updated in 2014.

This repository was transferred from dritter-sap/iibench-mongodb, originally forked from iibench-mongodb, but not MongoDB-specific any longer.

This repository denotes the latest version of the benchmark. All further contributions should go here.

Contributions (so far):

• fixed bugs (e.g., threading problems)
• ported benchmark to mongodb v3.11.1
• added a benchmark for orientdb v3.2.0 (latest minor version)
• added post-processing for visualization of results, plotting (using xchart, and csv-jdbc)
• improved CLI (using airline)

Supported indexes

• hash
• range / tree

(see open topics / issues below)

Requirements

• Java 1.8+

Running the benchmark

git clone https://github.com/dritter-sap/iibench.git
cd iibench
mvn clean package

OrientDB

start via docker, e.g. docker run -d --name orientdb -p 2424:2424 -p 2480:2480 -e ORIENTDB_ROOT_PASSWORD=root orientdb:latest

[cd target]

java -jar iibench.jar -host plocal:localhost -user root -password <password> -maxRows 1000000 -numDocsPerInsert 1000 -queryNumDocsBegin 1000000 -numWriterThreads 1 -numQueryThreads 1 -dbType orientdb

MongoDB

start via docker, e.g. docker run --net host --rm -d -p 27017:27017 mongo

[cd target]

java -jar iibench.jar -host localhost -port 27017 -user root -password <password> -maxRows 1000000 -numDocsPerInsert 1000 -queryNumDocsBegin 100000 -numWriterThreads 1 -numQueryThreads 1 -dbType mongodb

Open topics / issues

• initializer: more sophisticated data model + queries, JSON-specific
• fix multi-threading
• add an optional padding field to increase data set size quickly
  • user defined size
  • user defined compressible amount
• additional query workload features
  • randomly select query type (based on indexes)
• compare other features in Launchpad version
• Update driver versions:
  • OrientDB 3.1.x
  • OrientDB 3.2.x (current version)
• allow for interleaved insert, query
• beyond Java 1.8
• move plotting to https://github.com/manuelh9r/XTChart?

Further reading

• MongoDB 3.2 WiredTiger Benchmark: https://dzone.com/articles/mongodb-32-wiredtiger-in-iibench
• MongoDB benchmark: sysbench-mongodb IO-bound workload comparison: https://www.percona.com/blog/2015/07/14/mongodb-benchmark-sysbench-mongodb-io-bound-workload-comparison/
• Linkbench for MySQL & MongoDB with a cached database: http://smalldatum.blogspot.com/2015/07/linkbench-for-mysql-mongodb-with-cached.html
• PostgreSQL vs. MongoDB: https://www.percona.com/live/e17/sites/default/files/slides/High%20Performance%20JSON%20-%20PostgreSQL%20vs.%20MongoDB%20-%20FileId%20-%20115573.pdf
• MongoDB sysbench: https://lab-docs.percona.com/en/latest/mongodb-sysbench-hppro2.html