What is YapDatabase

YapDatabase is a comprised of 2 main features:

• a collection/key/value store built atop sqlite for iOS & Mac (the foundation)
• a plugin architecture that provides for advanced functionality such as Views, Secondary Indexes, Full Text Search, etc.

 

It has the following features:

• Concurrency. You can read from the database while another thread is simultaneously making modifications to the database. So you never have to worry about blocking the main thread, and you can easily write to the database on a background thread. And, of course, you can read from the database on multiple threads simultaneously.
• Built-In Caching. A configurable object cache is built-in. Of course sqlite has caching too. But it's caching raw bytes, and we're dealing with objects. So having a built-in cache means you can skip the deserialization process, and get your objects even faster.
• Metadata. Ever wanted to store extra data along with your object? Like maybe a timestamp of when it was downloaded. Or a fully separate but related object? You're in luck. Metadata support comes standard. Along with its own separate configurable cache too!
• Performance. Fetch thousands of objects on the main thread without dropping a frame.
• Extensions. More than just a key/value store, YapDatabase comes with an extensions architecture built-in. You can even create your own extensions.
• Views. Need to filter, group & sort your data? No problem. YapDatabase comes with Views. And you don't even need to write esoteric SQL queries. Views work using closures and Swift code (or objective-c). Plus they automatically update themselves, and they make animating tables really easy.
• Secondary Indexing. Speed up your queries by indexing important properties. And then use SQL style queries to quickly find your items.
• Full Text Search. Built atop sqlite's FTS module (contributed by google), you can add extremely speedy searching to your app with minimal effort.
• More Extensions. CloudKit, Geospatial Queries, ActionManager, ...

 

Concurrency

It all starts with sqlite. The powerful embedded database builtin to iOS and macOS, and the database used by Apple within Core Data. Sqlite has been around since 2000, which means there's plenty of outdated information on the web about it. Like this often touted bit of mis-information:

WRONG: Writing to a sqlite database locks the file and blocks others threads from concurrently reading it.

TRUTH: Starting with v3.7 sqlite added "write ahead logging". Here's what the sqlite website says about WAL:

1. WAL is significantly faster in most scenarios.
2. WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and writing can proceed concurrently.

YapDB is built from the ground up to explicitly take advantage of WAL and the latest improvements in sqlite. You get concurrency out of the box. So reading from the database won't block. Ever. You can still only have a single write transaction at a time, but YapDB handles all those details. So if you've used sqlite directly before, you can say goodbye to worrying about "busy" errors.

 

Caching

In addition to concurrency, YapDB has built-in caching. This means your code will hit the disk less often, and skip the deserialization step too.

Caching is configurable too. You can configure the limits per-connection at any time.

Plus it's integrated into the concurrency model. So update an object on your background connection and the cache(s) on other connections are automatically updated at the proper time.

 

Collections & Metadata

Plain-old key/value is a little restrictive. Many modern "key/value" stores such as MongoDb or Riak have an extra level called a collection or a bucket, etc. YapDatabase follows suite. So instead of using key/value, you use collection/key/value.

YapDB also supports optional "metadata", which is stored alongside the value. The metadata can be anything you need. From a simple timestamp (e.g. when the object was downloaded, or when it expires), to something more complicated like pre-calculated computational results.

Naturally metadata has its own separate dedicated cache. Configure it however you need.

 

Performance

YapDB strives to strike the right balance between providing an easy to use API, and being as close to sqlite as possible. In sqlite there are 3 layers:

• the database file itself
• a "connection" to the database
• a transaction, executed on a connection, to read and/or write

YapDB mirrors these layers in its API, and then heavily optimizes on top of it. The end result is a sense of familiarity for those with sqlite experience, yet no sqlite knowledge is required. The only thing visible is a clean API, and only sqlite guru's will notice how the API secretly enforces performance enhancing techniques.

 

Extensions

A collection/key/value database is nice, but often doesn't provide all the functionality you need as an app developer. And that's where extensions come in.

The collection/key/value architecture of YapDB is just the foundation for an advanced plugin system. These plugins are called "extensions". And they integrate seamlessly.

Views allow you Sort, Group & Filter your data. Perfect for tableViews, collectionViews, and more. And they even provide notifications that allow you to animate changes to your tableViews/collectionViews.

Secondary Indexes help to optimize your queries so you can find your item(s) faster. They even support SQL style queries, just like you're used to.

Full Text Search provides blazing fast search using SQLite's FTS module (written by Google).

This is just the tip of the iceberg. There are many more extensions available.