Remapping

[benbjohnson]

Overview

This pull request includes the ability to remap data files when allocating new pages from the end. This differs from LMDB's approach where there is a single fixed limit on the mmap size.

Pros & Cons

Contrasting with LMDB's Approach

LMDB's approach works well for the most part in that a user can simply set the mmap size to the size of the hard disk. One issue I ran into with this previously was that I was running many LMDB databases in the same process and setting a high limit for all of them which exhausted the virtual memory space for my process.

Bolt's Approach

Bolt's approach is to start with a small memory mmap (4MB) and double it whenever the data file exceeds that size. The mmap uses a sync.RWLock so remapping can get exclusive access to the map. The downside of this is that it requires waiting for all read transactions to complete before remapping can occur. Bolt is aimed at short read transactions (because of its page reclamation design) so this shouldn't be an issue.

Summary

I don't think either LMDB or Bolt has a better approach. I think they're just trade offs. Bolt's API aims to be simple and limit the number of configuration parameters but does it at some performance cost.

Feedback welcome.

[thehydroimpulse]

@benbjohnson Just curious, but say you have an existing bolt database on disk. When bolt starts up, does it still allocate 4MB for the mmap, or does it start at the size of the existing database?

[benbjohnson]

@thehydroimpulse Good question. The mmap size will be:

1. 4MB if the bolt file is less than 4MB.
2. Double the file size if the bolt file is between 4MB and 1GB.
3. The file size + 1GB if the bolt file is greather than 1GB.

The 4MB & 1GB numbers are somewhat arbitrary. The idea is that the DB should have room to grow before hitting a remapping.

[thehydroimpulse]

@benbjohnson Awesome, thanks for the explanation. Also, does Bolt have the same mmap semantics as LMDB? Such that the mapped region, by default, is read-only to avoid accidental errors/mistakes (given that it seems like Bolt and LMDB maintain copy-on-write semantics, so there isn't any need to do any recovery)?

Bolt looks super awesome btw.

[benbjohnson]

@thehydroimpulse Yep, Bolt and LMDB are very similar in that respect. LMDB has an MDB_WRITEMAP option for direct write access but I'm not that brave. Bolt is always read-only.

It's all copy-on-write and there's no recovery required in the case of a crash. I need to add a checksum to the meta page to guarantee that it's not corrupted (#25) but otherwise it should be safe.

[thehydroimpulse]

@benbjohnson Awesome. Thanks.