Avoid multi-threaded auto-commit

[mahendra]

Note: This is done on top of #33

This is done using shared memory objects provided by multiprocessing.

Design

• We start a commit process which waits on a multiprocessing.Queue with timeout = auto_commit_every_t
• In the main proc, when _auto_commit triggers a commit, we send a message on the queue
• The commit proc wakes up and does a commit
• If timeout is not None, then the commit proc will wake up after the specified timeout and to the commit.
• Data is shared between both procs using shared memory.
• We avoid the use of Locks completely

Also fix a bug in auto-commit for - auto_commit_every_n

[mumrah]

@mahendra now that we have CI and the test are working again, can you rebase your changes and add some tests?

[mahendra]

sure. working on it.

[mahendra]

This works now, but then commit will stop working in windows because of the issue mentioned in #46. do review this and let me know if it is ready for merge. Will keep it on hold till we sort out #46. Then we can commit both together.

[mahendra]

@mumrah I have re-arranged the code so that it will work in windows. The trick is in ensuring that the target parameter passed to multiprocessing.Process is a normal function.

Do have a look and let me know.

[rdiomar]

The problem we have with forking long-running python processes is that the child can potentially have the same memory footprint as the parent, which can be pretty large.

[dpkp]

Closing inactive PR.