From 5139f534f6b6aef5fc5686796d1cec8d3f5206b8 Mon Sep 17 00:00:00 2001 From: Volker Mische Date: Thu, 22 Nov 2018 17:48:25 +0100 Subject: [PATCH 1/5] docs: major API changes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The whole IPLD APIs get a review to make them more consistent and easier to use. Changes to compared to the current spec: - No more callbacks, everything is Promises. - `get()`: - Is renamed to `resolve()` to indicate that it also takes a mandatory path argument. - Doesn’t have a `cid` in the return value anymore. That use case is covered by returning all objects that were traversed. Their `value` will always be the CID of the next one to traverse. So if you want to know the CID of the current IPLD Node, just look at the `value` of the previously returned IPLD Node. - Doesn’t return a single value, but an iterator. - `localResolve` option is removed. If you want to resolve a single IPLD Node only, just stop the iterator after the first item. - `getStream()` is removed without replacement. Use `resolve()` which uses async iterators instead. - `getMany()` is renamed to `get()`: - takes an iterable of CIDs as parameter. - `put()`: - takes an iterable of CIDs as parameter. - Doesn’t take `cid` as an option anymore. The CID is always calculated (https://github.com/ipld/js-ipld/issues/175). - The options don’t take the `format` (which is a string), but the `codec` (which is a `multicodec`) (https://github.com/ipld/js-ipld/issues/175). - the `version` option always defaults to `1`. - `.treeStream()` is removed without replacement (the only current user seems to be the IPFS DAG API tree command and the ipld-explorer-cli). IPLD Nodes are just normal JavaScript objects, so you can call `Object.keys()` recursively on a IPLD Node to get all its resolvable paths. If you want to follow all the CIDs, write the tree traversal yourself. This could perhaps be an example bundled with js-ipld. - `remove()`: - takes an iterable of CIDs as parameter. - `.support.add()` is renamed to `.addFormat()`. - `.support.rm()` is renamed to `.removeFormat()`. Almost all open issues in regards to the IPLD API got adressed. One bigger thing is the Patch API, which also isn’t part of the current specification. Here’s an overview of the issues: - https://github.com/ipld/js-ipld/issues/66 - [ ] IPLD Resolver `patch` API: There is no patch API yet - https://github.com/ipld/js-ipld/issues/70 - [x] lazu value lookups: Can be done as IPLD Formats is pure JavaScript - [x] get external / local paths only: @vmx doesn’t know what this is, considers it a “won’t fix” - [x] toJSON + fromJSON - roundtrip: A roundtrip is not a goal anymore => won’t fix - [ ] put + patch api #66: Patch API is still missing - [x] text summary: @vmx doesn’t see a strong use case for this => “won’t fix” - [x] globbing: Out of scope for js-ipld - https://github.com/ipld/js-ipld/issues/175 - [x] Deprecate passing a CID to `ipld.put`?: Current spec doesn’t allow `put()` with a CID - https://github.com/ipld/js-ipld/issues/182 - [x] API Review: Links to many other issues, I list the individual issues below - https://github.com/ipld/js-ipld/issues/183 - [x] getting the merkle proof with resolver.resolve: `resolve()` returns all CIDs along the traversed path - https://github.com/ipld/js-ipld/issues/184 - [ ] Pass down the `options` object to resolver.resolve(): This needs a custom resolver. @vmx isn’t sure if the js-ipld API should make this possible, or of it should just be easy enough to create your own resolver. - https://github.com/ipfs/interface-ipfs-core/issues/81 - [x] The `dag` API - One API to manipulate all the IPLD Format objects: Interesting discussion, but not relevant anymore. - https://github.com/ipfs/interface-ipfs-core/issues/121 - [x] dag api: add {ls, tree}: `tree()` is not part of js-ipld anymore as the IPLD Nodes are just JavaScript objects which you can easily traverse if you like. Though `tree()`-like functionality might be added as an example or separate module. - https://github.com/ipfs/interface-ipfs-core/issues/125 - [x] `dag get --localResolve` vs `dag resolve`: This is solved by the new `resolve()` API - https://github.com/ipfs/interface-ipfs-core/issues/137 - [x] add `level` option to ipfs.dag.tree: `tree()` is not part of js-ipld anymore. It should be considered if `tree()` is implemented (probably as an example or separate module) Closes #182. --- README.md | 73 ++++++++++++++++++++++++++++++++----------------------- 1 file changed, 43 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index acea380..9980373 100644 --- a/README.md +++ b/README.md @@ -90,6 +90,8 @@ initIpld('/tmp/ifpsrepo', (err, ipld) => { ## API +The IPLD API works strictly with CIDs and deserialized IPLD Nodes. Interacting with the binary data happens on lower levels. To access the binary data directly, use the [Block API](https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/BLOCK.md). + ### IPLD constructor > Creates and returns an instance of IPLD. @@ -150,64 +152,75 @@ const ipld = new Ipld({ }) ``` -### `.put(node, options, callback)` +### `.put(nodes, options)` -> Store the given node of a recognized IPLD Format. +> Stores the given IPLD Nodes of a recognized IPLD Format. -`options` is an object that must contain one of the following combinations: -- `cid` - the CID of the node -- `[hashAlg]`, `[version]` and `format` - the hashAlg, version and the format that should be used to create the CID of the node. The -`hashAlg` and `version` defaults to the default values for the `format`. + - `cids` (`Iterable`): deserialized IPLD nodes that should be inserted. -It may contain any of the following: + - `options` is applied to any of the `nodes` and is an object with the following properties: + - `codec` (`multicodec`, required): the multicodec of the format that IPLD Node should be encoded in. + - `hashCode` (`multicodec`, default: hash algorithm of the given multicodec): the hashing algorithm that is used to calculate the CID. + - `version` (`boolean`, default: 1): the CID version to use. + - `onlyHash` (`boolean`, default: false): if true the serialized form of the IPLD Node will not be passed to the underlying block store. -- `onlyHash` - If true the serialized form of the node will not be passed to the underlying block store but the passed callback will be invoked as if it had been +Returns an async iterator with the CIDs of the serialized IPLD Nodes. -`callback` is a function that should have the signature as following: `function (err, cid) {}`, where `err` is an Error object in case of error and `cid` is the cid of the stored object. -### `.get(cid [, path] [, options], callback)` +### `.resolve(cid, path)` -> Retrieve a node by the given `cid` or `cid + path` +> Retrieves IPLD Nodes along the `path` that is rooted at `cid`. -`options` is an optional object containing: + - `cid` (`CID`, required): the CID the resolving starts. + - `path` (`IPLD Path`, required): the path that should be resolved. -- `localResolve: bool` - if true, get will only attempt to resolve the path locally +Returns an async iterator of all the IPLD Nodes that were traversed during the path resolving. Every element is an object with these fields: + - `remainderPath` (`string`): the part of the path that wasn’t resolved yet. + - `value` (`*`): the value where the resolved path points to. If further traversing is possible, then the value is a CID object linking to another IPLD Node. If it was possible to fully resolve the path, `value` is the value the `path` points to. So if you need the CID of the IPLD Node you’re currently at, just take the `value` of the previously returned IPLD Node. -`callback` should be a function with the signature `function (err, result)`, the result being an object with: -- `value` - the value that resulted from the get -- `remainderPath` - If it didn't manage to successfully resolve the whole path through or if simply the `localResolve` option was passed. -- `cid` - Where the graph traversal finished - if `remainderPath` has a value, this will be where it has its root +### `.get(cids)` -### `.getMany(cids, callback)` +> Retrieve several IPLD Nodes at once. -> Retrieve several nodes at once + - `cids` (`Iterable`): the CIDs of the IPLD Nodes that should be retrieved. -`callback` should be a function with the signature `function (err, result)`, the result is an array with the nodes corresponding to the CIDs. +Returns an async iterator with the IPLD Nodes that correspond to the given `cids`. +Throws an error if a IPLD Node can’t be retrieved. -### `.getStream(cid [, path] [, options])` +### `.remove(cids)` -> Same as get, but returns a source pull-stream that is used to pass the fetched node. +> Remove IPLD Nodes by the given `cids` -### `.treeStream(cid [, path] [, options])` + - `cids` (`Iterable`): the CIDs of the IPLD Nodes that should be retrieved. -> Returns all the paths under a cid + path through a pull-stream. Accepts the following options: +Throws an error if any of the Blocks can’t be removed. This operation is not atomic, some Blocks might have already been removed. -- `recursive` - bool - traverse through links to complete the graph. -### `.remove(cid, callback)` +### `.addFormat(ipldFormatImplementation)` -> Remove a node by the given `cid` +> Add support to another IPLD Format -### `.support.add(multicodec, formatResolver, formatUtil)` + - `ipldFormatImplementation` (`IPLD Format`, required): the implementation of an IPLD Format. -> Add support to another IPLD Format -### `.support.rm(multicodec)` +### `.removeFormat(codec)` > Removes support of an IPLD Format + - `codec` (`multicodec`, required): the IPLD Format the should be removed. + + +### `.remove(cids)` + +> Remove IPLD Nodes by the given `cids` + +`cids` is an iterable — most often an array — with CIDs of the Blocks that should be removed. + +Throws an error if any of the Blocks can’t be removed. This operation is not atomic, some Blocks might have already been removed. + + ### Properties #### `defaultOptions` From 8165bf973fd30f93fb8251a47b5a0444138c4a63 Mon Sep 17 00:00:00 2001 From: Volker Mische Date: Mon, 28 Jan 2019 13:03:02 +0100 Subject: [PATCH 2/5] fix: various fixes Most importantly the `tree()` method is added again. It's really needed for backwards compatibility. --- README.md | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index 9980373..60d9eb2 100644 --- a/README.md +++ b/README.md @@ -156,12 +156,12 @@ const ipld = new Ipld({ > Stores the given IPLD Nodes of a recognized IPLD Format. - - `cids` (`Iterable`): deserialized IPLD nodes that should be inserted. + - `nodes` (`Iterable`): deserialized IPLD nodes that should be inserted. - `options` is applied to any of the `nodes` and is an object with the following properties: - - `codec` (`multicodec`, required): the multicodec of the format that IPLD Node should be encoded in. - - `hashCode` (`multicodec`, default: hash algorithm of the given multicodec): the hashing algorithm that is used to calculate the CID. - - `version` (`boolean`, default: 1): the CID version to use. + - `format` (`multicodec`, required): the multicodec of the format that IPLD Node should be encoded in. + - `hashAlg` (`multicodec`, default: hash algorithm of the given multicodec): the hashing algorithm that is used to calculate the CID. + - `version` (`number`, default: 1): the CID version to use. - `onlyHash` (`boolean`, default: false): if true the serialized form of the IPLD Node will not be passed to the underlying block store. Returns an async iterator with the CIDs of the serialized IPLD Nodes. @@ -193,32 +193,35 @@ Throws an error if a IPLD Node can’t be retrieved. > Remove IPLD Nodes by the given `cids` - - `cids` (`Iterable`): the CIDs of the IPLD Nodes that should be retrieved. + - `cids` (`Iterable`): the CIDs of the IPLD Nodes that should be removed. Throws an error if any of the Blocks can’t be removed. This operation is not atomic, some Blocks might have already been removed. -### `.addFormat(ipldFormatImplementation)` +### `.tree(cid, [path], [options])` -> Add support to another IPLD Format +> Returns all the paths that can be resolved into. - - `ipldFormatImplementation` (`IPLD Format`, required): the implementation of an IPLD Format. + - `cid` (`CID`, required): the CID to get the paths from. + - `path` (`IPLD Path`, default: ''): the path to start to retrieve the other paths from. + - `options`: + - `recursive` (`bool`, default: false): whether to get the paths recursively or not. `false` resolves only the paths of the given CID. +Returns an async iterator of all the paths (as Strings) you could resolve into. -### `.removeFormat(codec)` -> Removes support of an IPLD Format +### `.addFormat(ipldFormatImplementation)` - - `codec` (`multicodec`, required): the IPLD Format the should be removed. +> Add support for an IPLD Format + - `ipldFormatImplementation` (`IPLD Format`, required): the implementation of an IPLD Format. -### `.remove(cids)` -> Remove IPLD Nodes by the given `cids` +### `.removeFormat(codec)` -`cids` is an iterable — most often an array — with CIDs of the Blocks that should be removed. +> Remove support for an IPLD Format -Throws an error if any of the Blocks can’t be removed. This operation is not atomic, some Blocks might have already been removed. + - `codec` (`multicodec`, required): the codec of the IPLD Format to remove. ### Properties From 221fab054bafd1f3d299142cd1b98bf76a0d3cd3 Mon Sep 17 00:00:00 2001 From: Volker Mische Date: Mon, 28 Jan 2019 13:34:05 +0100 Subject: [PATCH 3/5] docs: update `options.loadFormat()` documentation Loading a format is no longer callback based but is using an async function. --- README.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 60d9eb2..b2108c0 100644 --- a/README.md +++ b/README.md @@ -132,21 +132,22 @@ const ipld = new Ipld({ }) ``` -##### `options.loadFormat(codec, callback)` +##### `options.loadFormat(codec)` | Type | Default | |------|---------| -| `Function` | `null` | +| `async Function` | `null` | -Function to dynamically load an [IPLD Format](https://github.com/ipld/interface-ipld-format). It is passed a string `codec`, the multicodec of the IPLD format to load and a callback function to call when the format has been loaded. e.g. +Function to dynamically load an [IPLD Format](https://github.com/ipld/interface-ipld-format). It is passed a `codec`, the multicodec code of the IPLD format to load and returns an IPLD Format implementation. For example: ```js +const multicodec = require('multicodec') const ipld = new Ipld({ - loadFormat (codec, callback) { - if (codec === 'git-raw') { - callback(null, require('ipld-git')) + async loadFormat (codec) { + if (codec === multicodec.GIT_RAW) { + return require('ipld-git') } else { - callback(new Error('unable to load format ' + codec)) + throw new Error('unable to load format ' + multicodec.print[codec]) } } }) From 1a2c0ad13c32f85c7b0b6aa0fa686b75271df53b Mon Sep 17 00:00:00 2001 From: Volker Mische Date: Mon, 28 Jan 2019 13:36:07 +0100 Subject: [PATCH 4/5] docs: update TOC of README `doctoc` was used to update the table of contents. --- README.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index b2108c0..a69a0e1 100644 --- a/README.md +++ b/README.md @@ -38,17 +38,19 @@ Want to get started? Check our examples folder. You can check the development st - [Install](#install) - [Usage](#usage) - [API](#api) - - IPLD Resolver - - [Constructor](#ipld-constructor) - - [`.put(node, options, callback)`](#putnode-options-callback) - - [`.get(cid [, path] [, options], callback)`](#getcid--path--options-callback) - - [`.getStream(cid [, path] [, options])`](#getstreamcid--path--options) - - [`.treeStream(cid [, path] [, options])`](#treestreamcid--path--options) - - [`.remove(cid, callback)`](#removecid-callback) - - [`.support.add(multicodec, formatResolver, formatUtil)`](#supportaddmulticodec-formatresolver-formatutil) - - [`.support.rm(multicodec)`](#supportrmmulticodec) - - [Properties](#properties) - - [`defaultOptions`](#defaultoptions) + - [IPLD constructor](#ipld-constructor) + - [`options.blockService`](#optionsblockservice) + - [`options.formats`](#optionsformats) + - [`options.loadFormat(codec)`](#optionsloadformatcodec) + - [`.put(nodes, options)`](#putnodes-options) + - [`.resolve(cid, path)`](#resolvecid-path) + - [`.get(cids)`](#getcids) + - [`.remove(cids)`](#removecids) + - [`.tree(cid, [path], [options])`](#treecid-path-options) + - [`.addFormat(ipldFormatImplementation)`](#addformatipldformatimplementation) + - [`.removeFormat(codec)`](#removeformatcodec) + - [Properties](#properties) + - [`defaultOptions`](#defaultoptions) - [Packages](#packages) - [Contribute](#contribute) - [License](#license) From 2365ae318689bd3824da5e3e196fa3ed0a0f5a71 Mon Sep 17 00:00:00 2001 From: Volker Mische Date: Wed, 6 Feb 2019 11:33:24 +0100 Subject: [PATCH 5/5] docs: change `put()` signature The `format` is no longer part of the the options object, but a separate parameter. The `version` option was renamed to `cidVersion`. --- README.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index a69a0e1..3024331 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ Want to get started? Check our examples folder. You can check the development st - [`options.blockService`](#optionsblockservice) - [`options.formats`](#optionsformats) - [`options.loadFormat(codec)`](#optionsloadformatcodec) - - [`.put(nodes, options)`](#putnodes-options) + - [`.put(nodes, format, options)`](#putnodes-format-options) - [`.resolve(cid, path)`](#resolvecid-path) - [`.get(cids)`](#getcids) - [`.remove(cids)`](#removecids) @@ -155,16 +155,15 @@ const ipld = new Ipld({ }) ``` -### `.put(nodes, options)` +### `.put(nodes, format, options)` > Stores the given IPLD Nodes of a recognized IPLD Format. - `nodes` (`Iterable`): deserialized IPLD nodes that should be inserted. - + - `format` (`multicodec`, required): the multicodec of the format that IPLD Node should be encoded in. - `options` is applied to any of the `nodes` and is an object with the following properties: - - `format` (`multicodec`, required): the multicodec of the format that IPLD Node should be encoded in. - `hashAlg` (`multicodec`, default: hash algorithm of the given multicodec): the hashing algorithm that is used to calculate the CID. - - `version` (`number`, default: 1): the CID version to use. + - `cidVersion` (`number`, default: 1): the CID version to use. - `onlyHash` (`boolean`, default: false): if true the serialized form of the IPLD Node will not be passed to the underlying block store. Returns an async iterator with the CIDs of the serialized IPLD Nodes.