mpdict

A Multi-Process Dict for Python 3

• Based on Boost::interprocess library
• 20X faster than using Redis, with lower memory usage and no background daemon.
• 1 Million "set" op/s on i7@3.5GHz CPU

Designed to serve as and in-memory database for Python processes in a multi-worker service, allowing to scale at a lower CPU & memory cost.

Installation

$ sudo apt install g++ libboost-container-dev

<...>

$ python3 setup.py install --user

running install
running build
running build_ext
building 'mpdict' extension
creating build/temp.linux-x86_64-3.6
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c mpdict.cpp -o build/temp.linux-x86_64-3.6/mpdict.o -std=gnu++1z -Wall -Werror -mtune=native
creating build/lib.linux-x86_64-3.6
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/mpdict.o -lstdc++ -lboost_container -lrt -o build/lib.linux-x86_64-3.6/mpdict.cpython-36m-x86_64-linux-gnu.so
running install_lib
copying build/lib.linux-x86_64-3.6/mpdict.cpython-36m-x86_64-linux-gnu.so -> /home/user/.local/lib/python3.6/site-packages
running install_egg_info
Writing /home/user/.local/lib/python3.6/site-packages/mpdict-0.1.egg-info

Performance Check

(On AMD A8-4500M based laptop at ~2 GHz)

Adding 1 Million items to a "mpdict" takes 3 seconds and uses 80 MB

$ /usr/bin/time -v python3.7 mp_dict_test.py mpdict 3 1000000
Testing mpdict on 3 processes X 1e+06 items...
Elapsed 3.13951 seconds
d[999781]: ('Foreground0', 'Foreground') ('Foreground1', 'Foreground') ('Foreground10', 'Foreground') ('Foreground100', 'Foreground') ('Foreground1000', 'Foreground') ('Foreground10000', 'Foreground') ('Foreground100000', 'Foreground') ('Foreground100001', 'Foreground') ('Foreground100002', 'Foreground')
	Command being timed: "python3.7 mp_dict_test.py mpdict 3 1000000"
	User time (seconds): 4.40
	System time (seconds): 0.57
	Percent of CPU this job got: 153%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.25
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 80632
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 45307
	Voluntary context switches: 27576
	Involuntary context switches: 200
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Comparison with a Redis based dictionary

Starting redis-server with some optimization

$ sudo sysctl vm.overcommit_memory=1
vm.overcommit_memory = 1
$ echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled
never
$ redis-server --port 0 --unixsocket /tmp/testRedisDict
27237:C 07 Mar 18:31:39.779 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
27237:C 07 Mar 18:31:39.779 # Redis version=4.0.9, bits=64, commit=00000000, modified=0, pid=27237, just started
...
27237:M 07 Mar 18:31:39.782 * The server is now ready to accept connections at /tmp/testRedisDict

Add 100,000 items to Redis-based dict from 3 processes

$ /usr/bin/time -v python3.7 mp_dict_test.py redis
Testing redis on 3 processes X 100000 items...
Elapsed 7.81296 seconds
d[99999]: (b'Foreground12887', b'Foreground') (b'Foreground23199', b'Foreground') (b'Foreground9280', b'Foreground') (b'b25846', b'bbbbbbbbbb') (b'Foreground5959', b'Foreground') (b'a31296', b'aaaaaaaaaa') (b'a31122', b'aaaaaaaaaa') (b'a29595', b'aaaaaaaaaa') (b'a24278', b'aaaaaaaaaa')
	Command being timed: "python3.7 mp_dict_test.py redis"
	User time (seconds): 9.67
	System time (seconds): 1.84
	Percent of CPU this job got: 126%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.06
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 20496
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 6581
	Voluntary context switches: 131071
	Involuntary context switches: 363
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Note that this takes more than double the time to process 10X less items, and uses ~4X more memory per item