Merge pull request #4223 from ipfs/docs/datastore

Add documentation for datastore configs

docs/config.md

@@ -68,11 +68,6 @@ Default: The ipfs.io bootstrap nodes
 Contains information related to the construction and operation of the on-disk
 storage system.
 
-- `Type`
-Denotes overall datastore type. The only currently valid option is `leveldb`.
-
-Default: `leveldb`
-
 - `Path`
 Path to the leveldb datastore directory. Set during init to either `$IPFS_PATH/datastore`, or `$HOME/.ipfs/datastore` if `$IPFS_PATH` is unset.
 
@@ -91,11 +86,6 @@ A time duration specifying how frequently to run a garbage collection. Only used
 
 Default: `1h`
 
-- `NoSync` *!*
-A boolean value denoting whether or not to disable sanity syncing in the flatfs datastore code. Setting this to true may significantly improve performance, but be careful using it as if the daemon is killed before a write is synchronized to disk, there is a chance of data loss.
-
-Default: `false`
-
 - `HashOnRead`
 A boolean value. If set to true, all block reads from disk will be hashed and verified. This will cause increased CPU utilization.
 
@@ -104,8 +94,42 @@ A number representing the size in bytes of the blockstore's bloom filter. A valu
 
 Default: `0`
 
-- `Params`
-Extra parameters for datastore construction, not currently used.
+- `Spec`
+Spec defines the structure of the ipfs datastore. It is a composable structure, where each datastore is represented by a json object. Datastores can wrap other datastores to provide extra functionality (eg metrics, logging, or caching).
+
+This can be changed manually, however, if you make any changes that require a different on-disk structure, you will need to run the [ipfs-ds-convert tool](https://github.com/ipfs/ipfs-ds-convert) to migrate data into the new structures.
+
+For more information on possible values for this configuration option, see docs/datastores.md 
+
+Default:
+```
+{
+  "mounts": [
+	{
+	  "child": {
+		"path": "blocks",
+		"shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
+		"sync": true,
+		"type": "flatfs"
+	  },
+	  "mountpoint": "/blocks",
+	  "prefix": "flatfs.datastore",
+	  "type": "measure"
+	},
+	{
+	  "child": {
+		"compression": "none",
+		"path": "datastore",
+		"type": "levelds"
+	  },
+	  "mountpoint": "/",
+	  "prefix": "leveldb.datastore",
+	  "type": "measure"
+	}
+  ],
+  "type": "mount"
+}
+```
 
 ## `Discovery`
 Contains options for configuring ipfs node discovery mechanisms.

docs/datastores.md

@@ -0,0 +1,79 @@
+# Datastore Configuration Options
+
+This document describes the different possible values for the `Datastore.Spec`
+field in the ipfs configuration file.
+
+## flatfs
+Stores each key value pair as a file on the filesystem.
+
+The shardFunc is prefixed with `/repo/flatfs/shard/v1` then followed by a descriptor of the sharding strategy. Some example values are:
+- `/repo/flatfs/shard/v1/next-to-last/2`
+  - Shards on the two next to last characters of the key
+- `/repo/flatfs/shard/v1/prefix/2`
+  - Shards based on the two character prefix of the key
+
+```json
+{
+	"type": "flatfs",
+	"path": "<relative path within repo for flatfs root>",
+	"shardFunc": "<a descriptor of the sharding scheme>",
+	"sync": true|false
+}
+```
+
+NOTE: flatfs should only be used as a block store (mounted at `/blocks`) as the
+current implementation is not complete.
+
+## levelds
+Uses a leveldb database to store key value pairs.
+
+```json
+{
+	"type": "levelds",
+	"path": "<location of db inside repo>",
+	"compression": "none" | "snappy",
+}
+```
+
+## badgerds
+Uses [badger](https://github.com/dgraph-io/badger) as a key value store.
+
+```json
+{
+	"type": "badgerds",
+	"path": "<location of badger inside repo",
+	"syncWrites": true|false
+}
+```
+
+## mount
+Allows specified datastores to handle keys prefixed with a given path.
+The mountpoints are added as keys within the child datastore definitions.
+
+```json
+{
+	"type": "mount",
+	"mounts": [
+		{
+			// Insert other datastore definition here, but add the following key:
+			"mountpoint": "/path/to/handle"
+		},
+		{
+			// Insert other datastore definition here, but add the following key:
+			"mountpoint": "/path/to/handle"
+		},
+	]
+}
+```
+
+## measure
+This datastore is a wrapper that adds metrics tracking to any datastore.
+
+```json
+{
+	"type": "measure",
+	"prefix": "sometag.datastore",
+	"child": { datastore being wrapped }
+}
+```
+