[RFC] Async Storage v2

[krizzu]

Async Storage v2

Motivation

A few months back, due to the lean core effort, Async Storage was moved to a standalone repo, making it more open for bug fixes and feature implementations. By adding semantic-versioning, we've made sure that releases happen more often, keeping the library in a good shape.

One of the discussed topics among the community was adding support for out of the tree platforms, so in a case where you have multi-platform project (say, Web app and RN app, sharing components and/or business logic), you don't have to use two different libraries for persisting data.

In a new major release, I'd like to introduce a concept of replaceable storage backends, making Async Storage more open not only to other platforms but also to other storage implementations.

Main changes

Current Async Storage is strictly coupled with its storage implementation (SQLite for Android and serialized dictionary/files for iOS). The replaceable storage feature would give you the ability to use adequate storage for a given task - think of using Shared Preferences for simple key-values, while still being able to use SQLite database for larger data, using the same API.

Key benefits :

• Use one or more storages that matches your application needs, using the same API
• Support for out-of-the-tree platforms
• Storages are developed independently

Implementation

There are three main parts of this feature:

• Async Storage factory
• Async Storage Public API
• Storage implementation (using Common Storage API)

Async Storage Factory

This factory class creates an instance of AsyncStorage with selected storage set. You can pick storage used based on platform or other condition.

class AsyncStorageFactory {
  static create(storage: AsyncStorageBackend, opts?: FactoryOptions): AsyncStorage {}
}

Creation of Async Storage can be enhanced by passing FactoryOptions options. Features such as logging can be set up here (added later, over time).

type FactoryOptions {
	// use default or provide logger function
	enableLogging: (logger?: boolean | function): any
}

Public API

class AsyncStorage {

 /* 
  * expected to return a value stored under 'key'
  * `null` when value not found
  */ 
  get(key: string, options?: AsyncStorageOptions): Promise<mixed> {}


 /*
  * expected to save `item`, using `key`
  */
  set(key: string, item: mixed, options?: AsyncStorageOptions): Promise<void> {}


 /*
  * expected to get a value for each passed key
  * returns array with retrieved items
  */
  getMultiple(
    keys: Array<string>,
    options?: AsyncStorageOptions,
  ): Promise<Array<mixed>> {}


 /*
  * expected to save key-values in storage
  */
  setMultiple(keyValueArray: Array<Object>, options?: AsyncStorageOptions): Promise<void> {}


 /*
  * expected to removed item, stored under `key`
  * returned value denotes success of operation ( true = success )
  */
  remove(key: string, options?: AsyncStorageOptions): Promise<void> {}


 /*
  * expected to remove multiple values
  */
  removeMultiple(keys: Array<string>, options?: AsyncStorageOptions): Promise<void> {}
  
  
 /*
  * expected to return all keys used to store values
  */
  getKeys(options?: AsyncStorageOptions): Promise<Array<string>> {}
  
 /*
  * expected to clear all stored data in storage
  */
  removeWholeDataFromTheStorage(options?: AsyncStorageOptions): Promise<void>

 /*
  * returns the instance of the storage backend. Useful if it has additional functionality not covered by public API
  */
  instance(): AsyncStorageBackend
}

1. Supported actions are write/update, read and delete.

2. Promise based API (no callbacks). This is mainly to focus on one API model and leverage Promise API (callbacks could be implemented through AsyncStorageOptions).

3. AsyncStorageOptions is optional config passed down to storage implementation. Type is dependent on storage used.

type AsyncStorageOptions<T> = T;

Storage Backend API

interface AsyncStorageBackend<T> {

  getSingle<T>(key: string, opts?: T): Promise<T | null>

  setSingle<T>(key: string, value: T, opts?: T): Promise<void>

  getMany(keys: Array<string>, opts?: T): Promise<Array<any>>

  setMany(values: Array<{ [key: string]: any }>, opts?: T): Promise<void>

  removeSingle(key: string, opts?: T): Promise<void>

  removeMany(keys: Array<string>, opts?: T): Promise<void>

  getKeys(opts?: T): Promise<Array<string>>

  dropStorage(opts?: T): Promise<void>
}

AsyncStorageBackend interface (Common Storage API on the diagram) must be applied by any Storage Backend in order to be valid. Those methods are called by Public API, passing options down.

As an example, where's how Storage Backend would look for a Web (using localStorage):

class WebStorageBackend implements AsyncStorageBackend<OptionsType> {
  async getSingle(key: string, opts?: OptionsType) {
    return window.localStorage.getItem(key)
  }

  async setSingle(key: string, value: any, opts?: OptionsType) {
    return window.localStorage.setItem(key, value)
  }

  // ...
}

Other changes

• Migration to Typescript (keeping support for Flow)

• Monorepo (see questions below)

• TBD.

Questions

• API (both public and common storage)

• Do we plan to maintain Storage Backends in the main repo? If yes, should we move to monorepo?

[satya164]

One feedback I have is that since the public API and the storage backend API have 1:1 mapping, it'll be better not to have different naming for them and have the same API for less cognitive overhead.

Though it also begs the question, if the storage backend and the public API have the same methods, then what'll be the advantage of AsyncStorageFactory.create instead of using the storage backend directly? Will there be some additional functionality added by the factory?

Regarding the storage backend for web, instead of localStorage, I'd suggest using IndexedDB for the default implementation because localStorage is sync and doesn't work in web workers etc.

[3rd-Eden]

If we are going to completely break the public API, I would suggest that we look for API compatibility with the kv-storage proposal:

https://github.com/WICG/kv-storage

[matt-oakes]

This sounds good! I have one question and a couple of comments:

Maintaining backwards compatibility

Could the new public API be implemented without completely breaking older code?

I did this with the NetInfo module by checking the type of the parameters and falling back to old behaviour if needed. This should help smooth out the adoption of the new version as it will be less of a problem if two libraries depend on AsyncStorage and one hasn't been updated.

It's more work to do this, but is much nicer for users of library. You can see a chunk of the NetInfo compatibility code here:

https://github.com/react-native-community/react-native-netinfo/blob/5dab2534d49b32ffdc386c807f6f3e077c644c6c/src/index.ts#L33-L94

Using kv-storage API as a base

I like the idea, but it seems like their API does not support getting and setting multiple keys at once. This is a really good optimisation for React Native due to the bridge and not something we should loose.

[krizzu]

Thank you all for the comments and suggestions!

@satya164 I've left names different to explicitly distinguish public API from Backend API, for cases like calling Storage directly (not through AsyncStorage API).
Regarding mapping, I thought about adding options to the factory, to extends the AS instance (for example, adding fallback storage, in case your primary is not available or some kind of middleware extension, like a logger, to ease storage debugging).

@3rd-Eden As @matt-oakes said we have methods to do operations on more than one data (multi set/get/remove), for optimization reasons, which are not supported in kv-storage. I also think that more explicit names like getItem, are more self-explaining.

@matt-oakes I think that adding the factory is already a big breaking change, but we could add support for old methods name (like getMulti/setMulti), by providing a flag to the factory (while creating an instance).

thanks.

[pbitkowski]

Awesome idea! I would go with monorepo with current implementations for Android / iOS. We can let people create their own implementations, but let's provide bottom line for people who don't want to do this.

I agree with @satya164, making explicit distinguish sounds good, but why do we want to change public API? I would change the Backend API method names, not the Public API if they must be easy to differentiate.

Good direction for improvements! I will keep my fingers crossed.

[zamotany]

I like the idea, tough I'm against having backward compatibility as a prime idea. We are talking about a new major version of Async Storage, so it is fine to change API. IMO the public API that kv-storage has is better that the current Async Storage API - for instance setItem is no better than just set, like what else could I set except for item? It's unnecessarily verbose.

That being said backward compatibility is a important feature for some people, so IMO it is better to have it as a opt-in in factory - there could be create and createLegacy or just a flag in create.

[ferrannp]

I love the idea to use SharedPrefernces and NSUserDefaults. I really do not need SQLite to save key-value settings.

Also I think it is nice pointing out that with the Async Storage Factory, it should be easier to also access the data from native directly (useful for brownfield apps and others).

Do we plan to maintain Storage Backends in the main repo? If yes, should we move to monorepo?

In favour of having some storages available.

--

 /*
  * expected to return all keys used to store values
  */
  getStorageKeys(options?: AsyncStorageOptions): Promise<Array<string>> {}
  
 /*
  * expected to clear all stored data in storage
  */
  removeWholeDataFromTheStorage(options?: AsyncStorageOptions): Promise<void>

Is the Storage word necessary? Why not getKeys or getAllKeys and something like clear or clearAll ?

[krizzu]

@pbitkowski @zamotany I want public API to be as clear as it can get. I'm fine with set/get/remove/multi*/ API.

@ferrannp I'm fine with getKeys, it's just a proposal. Regarding removeWholeDataFromTheStorage - more explicit name for irrecoverable action.

Regarding backward compatibility, I was thinking about providing LegacyStorage, which will use old storage location/files, with new API (so you're not giving up on your old data).

AsyncStorageFactory could be enhanced with options object. For example, a flag to enable logging, so it'd be easier to debug what's going in and out of storages.

[tido64]

Will the multi* functions still be relevant with the upcoming Fabric rewrite? If they're only there to reduce the number times you have to cross the bridge, I wonder if we have enough usage to warrant carrying it over to v2. I know we don't use them in our apps. I'd vote to have the API closer to kv-storage if that's what the community is moving towards.

Re: AsyncStorageBackend, I think another way to think about it is that it only provides the smallest building blocks for a storage. AsyncStorageFactory then gives us the ability to build things upon it, like logging as @krizzu mentioned.

[grabbou]

Great work on the RFC, I really like that.

One question - what do you think about supporting JavaScript objects by default and letting the backend to decide whether they want to serialise it entirely or not? Doing JSON.parse/JSON.serialise is not a big deal, but I feel like moving this responsibility over to storage backend would allow for more performant storage (if backend supports objects) and more expressive interface for setting and accessing values.

[krizzu]

@grabbou The main assumption is to let Storage decide what to do with given data (serialize it or not, modify it somehow, etc.).

Thank you all for the feedback. I'll do adjustments to API and add necessary details so we could begin the work soon.

thanks.

[crrobinson14]

I'd love to see two things:

1. A way to determine storage usage (bytes) via the API. There's a (lightly documented) method through the debugger, but the idea here would be to a. make this act a first-class citizen in the API, and b. allow programs to monitor themselves better (report it to QOS systems, log it to the console at app start, etc.)

2. Clarify the API around clearing the store. Right now there is a very convenient .clear() operation with specific instructions NOT to use it because an app can delete more than its own data. It's not clear at all what "other" data would be cleared here, or why the function is provided in the first place if it's so dangerous. The docs suggest clearing per key, but this is a nuisance for app developers to do and you can see how most might reach for the more convenient option.

Clearing the store is an operation nearly every app should include if there's any kind of logout function. It would be great to clean up how this operation is done.