Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Commit

Permalink
refactor: async iterables
Browse files Browse the repository at this point in the history
A roundup of the following PRs:

* closes #2658
* closes #2660
* closes #2661
* closes #2668
* closes #2674
* closes #2676
* closes #2680
* test fixes and other fix ups

---

To allow us to pass the interface tests, the timeout option is now supported in the `object.get` and `refs` APIs. It doesn't actually cancel the operation all the way down the stack, but allows the method call to return when the timeout is reached.

https://github.com/ipfs/js-ipfs/pull/2683/files#diff-47300e7ecd8989b6246221de88fc9a3cR170

---

Supersedes #2724

---

resolves #1438
resolves #1061
resolves #2257
resolves #2509
resolves #1670
refs ipfs-inactive/interface-js-ipfs-core#394

BREAKING CHANGE: `IPFS.createNode` removed

BREAKING CHANGE: IPFS is not a class that can be instantiated - use `IPFS.create`. An IPFS node instance is not an event emitter.

BREAKING CHANGE: Instance `.ready` property removed. Please use `IPFS.create` instead.

BREAKING CHANGE: Callbacks are no longer supported on any API methods. Please use a utility such as [`callbackify`](https://www.npmjs.com/package/callbackify) on API methods that return Promises to emulate previous behaviour.

BREAKING CHANGE: `PeerId` and `PeerInfo` classes are no longer statically exported from `ipfs-http-client` since they are no longer used internally.

BREAKING CHANGE: `pin.add` results now contain a `cid` property (a [CID instance](https://github.com/multiformats/js-cid)) instead of a string `hash` property.

BREAKING CHANGE: `pin.ls` now returns an async iterable.

BREAKING CHANGE: `pin.ls` results now contain a `cid` property (a [CID instance](https://github.com/multiformats/js-cid)) instead of a string `hash` property.

BREAKING CHANGE: `pin.rm` results now contain a `cid` property (a [CID instance](https://github.com/multiformats/js-cid)) instead of a string `hash` property.

BREAKING CHANGE: `add` now returns an async iterable.

BREAKING CHANGE: `add` results now contain a `cid` property (a [CID instance](https://github.com/multiformats/js-cid)) instead of a string `hash` property.

BREAKING CHANGE: `addReadableStream`, `addPullStream` have been removed.

BREAKING CHANGE: `ls` now returns an async iterable.

BREAKING CHANGE: `ls` results now contain a `cid` property (whose value is a [CID instance](https://github.com/multiformats/js-cid)) instead of a `hash` property.

BREAKING CHANGE: `files.readPullStream` and `files.readReadableStream` have been removed.

BREAKING CHANGE: `files.read` now returns an async iterable.

BREAKING CHANGE: `files.lsPullStream` and `files.lsReadableStream` have been removed.

BREAKING CHANGE: `files.ls` now returns an async iterable.

BREAKING CHANGE: `files.ls` results now contain a `cid` property (whose value is a [CID instance](https://github.com/multiformats/js-cid)) instead of a `hash` property.

BREAKING CHANGE: `files.ls` no longer takes a `long` option (in core) - you will receive all data by default.

BREAKING CHANGE: `files.stat` result now contains a `cid` property (whose value is a [CID instance](https://github.com/multiformats/js-cid)) instead of a `hash` property.

BREAKING CHANGE: `get` now returns an async iterable. The `content` property value for objects yielded from the iterator is now an async iterable that yields [`BufferList`](https://github.com/rvagg/bl) objects.

BREAKING CHANGE: `stats.bw` now returns an async iterable.

BREAKING CHANGE: `addFromStream` has been removed. Use `add` instead.

BREAKING CHANGE: `addFromFs` has been removed. Please use the exported `globSource` utility and pass the result to `add`. See the [glob source documentation](https://github.com/ipfs/js-ipfs-http-client#glob-source) for more details and an example.

BREAKING CHANGE: `addFromURL` has been removed. Please use the exported `urlSource` utility and pass the result to `add`. See the [URL source documentation](https://github.com/ipfs/js-ipfs-http-client#url-source) for more details and an example.

BREAKING CHANGE: `name.resolve` now returns an async iterable. It yields increasingly more accurate resolved values as they are discovered until the best value is selected from the quorum of 16. The "best" resolved value is the last item yielded from the iterator. If you are interested only in this best value you could use `it-last` to extract it like so:

```js
const last = require('it-last')
await last(ipfs.name.resolve('/ipns/QmHash'))
```

BREAKING CHANGE: `block.rm` now returns an async iterable.

BREAKING CHANGE: `block.rm` now yields objects of `{ cid: CID, error: Error }`.

BREAKING CHANGE: `dht.findProvs`, `dht.provide`, `dht.put` and `dht.query` now all return an async iterable.

BREAKING CHANGE: `dht.findPeer`, `dht.findProvs`, `dht.provide`, `dht.put` and `dht.query` now yield/return an object `{ id: CID, addrs: Multiaddr[] }` instead of a `PeerInfo` instance(s).

BREAKING CHANGE: `refs` and `refs.local` now return an async iterable.

BREAKING CHANGE: `object.data` now returns an async iterable that yields `Buffer` objects.

BREAKING CHANGE: `ping` now returns an async iterable.

BREAKING CHANGE: `repo.gc` now returns an async iterable.

BREAKING CHANGE: `swarm.peers` now returns an array of objects with a `peer` property that is a `CID`, instead of a `PeerId` instance.

BREAKING CHANGE: `swarm.addrs` now returns an array of objects `{ id: CID, addrs: Multiaddr[] }` instead of `PeerInfo` instances.

BREAKING CHANGE: `block.stat` result now contains a `cid` property (whose value is a [CID instance](https://github.com/multiformats/js-cid)) instead of a `key` property.

BREAKING CHANGE: `bitswap.wantlist` now returns an array of [CID](https://github.com/multiformats/js-cid) instances.

BREAKING CHANGE: `bitswap.stat` result has changed - `wantlist` and `peers` values are now an array of [CID](https://github.com/multiformats/js-cid) instances.

BREAKING CHANGE: the `init` option passed to the IPFS constructor will now not take any initialization steps if it is set to `false`. Previously, the repo would be initialized if it already existed. This is no longer the case. If you wish to initialize a node but only if the repo exists, pass `init: { allowNew: false }` to the constructor.

BREAKING CHANGE: removed `file ls` command from the CLI and HTTP API.

BREAKING CHANGE: Delegated peer and content routing modules are no longer included as part of core (but are still available if starting a js-ipfs daemon from the command line). If you wish to use delegated routing and are creating your node _programmatically_ in Node.js or the browser you must `npm install libp2p-delegated-content-routing` and/or `npm install libp2p-delegated-peer-routing` and provide configured instances of them in [`options.libp2p`](https://github.com/ipfs/js-ipfs#optionslibp2p). See the module repos for further instructions:

- https://github.com/libp2p/js-libp2p-delegated-content-routing
- https://github.com/libp2p/js-libp2p-delegated-peer-routing
  • Loading branch information
Alan Shaw committed Jan 23, 2020
1 parent bfa1c99 commit de01e54
Show file tree
Hide file tree
Showing 201 changed files with 5,014 additions and 6,152 deletions.
122 changes: 46 additions & 76 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@
],
"main": "src/core/index.js",
"browser": {
"./src/core/components/init-assets.js": false,
"./src/core/runtime/add-from-fs-nodejs.js": "./src/core/runtime/add-from-fs-browser.js",
"./src/core/runtime/init-assets-nodejs.js": "./src/core/runtime/init-assets-browser.js",
"./src/core/runtime/config-nodejs.js": "./src/core/runtime/config-browser.js",
"./src/core/runtime/dns-nodejs.js": "./src/core/runtime/dns-browser.js",
"./src/core/runtime/libp2p-nodejs.js": "./src/core/runtime/libp2p-browser.js",
Expand All @@ -25,7 +24,8 @@
"./src/core/runtime/repo-nodejs.js": "./src/core/runtime/repo-browser.js",
"./src/core/runtime/ipld-nodejs.js": "./src/core/runtime/ipld-browser.js",
"./test/utils/create-repo-nodejs.js": "./test/utils/create-repo-browser.js",
"stream": "readable-stream"
"stream": "readable-stream",
"ipfs-utils/src/files/glob-source": false
},
"browser-all-ipld-formats": {
"./src/core/runtime/ipld-browser.js": "./src/core/runtime/ipld-browser-all.js"
Expand Down Expand Up @@ -65,48 +65,41 @@
"@hapi/hapi": "^18.3.2",
"@hapi/joi": "^15.0.0",
"abort-controller": "^3.0.0",
"any-signal": "^1.1.0",
"array-shuffle": "^1.0.1",
"async-iterator-to-pull-stream": "^1.3.0",
"async-iterator-to-stream": "^1.1.0",
"base32.js": "~0.1.0",
"bignumber.js": "^9.0.0",
"binary-querystring": "~0.1.2",
"bl": "^4.0.0",
"bs58": "^4.0.1",
"buffer-peek-stream": "^1.0.1",
"byteman": "^1.3.5",
"callbackify": "^1.1.0",
"cid-tool": "~0.4.0",
"cids": "~0.7.1",
"cids": "^0.7.2",
"class-is": "^1.1.0",
"dag-cbor-links": "^1.3.2",
"datastore-core": "~0.7.0",
"datastore-pubsub": "^0.2.3",
"datastore-level": "^0.14.1",
"datastore-pubsub": "^0.3.0",
"debug": "^4.1.0",
"dlv": "^1.1.3",
"err-code": "^2.0.0",
"explain-error": "^1.0.4",
"file-type": "^12.0.1",
"fnv1a": "^1.0.1",
"fsm-event": "^2.1.0",
"get-folder-size": "^2.0.0",
"glob": "^7.1.3",
"hapi-pino": "^6.1.0",
"hashlru": "^2.3.0",
"human-to-milliseconds": "^2.0.0",
"interface-datastore": "~0.8.0",
"ipfs-bitswap": "^0.26.2",
"ipfs-bitswap": "github:ipfs/js-ipfs-bitswap#refactor/libp2p-async",
"ipfs-block": "~0.8.1",
"ipfs-block-service": "~0.16.0",
"ipfs-http-client": "^41.0.1",
"ipfs-http-response": "~0.4.0",
"ipfs-mfs": "^0.16.0",
"ipfs-http-client": "github:ipfs/js-ipfs-http-client#refactor/async-iterables2",
"ipfs-http-response": "^0.5.0",
"ipfs-mfs": "^1.0.0",
"ipfs-multipart": "^0.3.0",
"ipfs-repo": "^0.30.0",
"ipfs-unixfs": "^0.3.0",
"ipfs-unixfs-exporter": "^0.41.0",
"ipfs-unixfs-importer": "^0.44.0",
"ipfs-utils": "^0.4.2",
"ipfs-utils": "^0.7.1",
"ipld": "~0.25.0",
"ipld-bitcoin": "~0.3.0",
"ipld-dag-cbor": "~0.15.0",
Expand All @@ -115,100 +108,77 @@
"ipld-git": "~0.5.0",
"ipld-raw": "^4.0.0",
"ipld-zcash": "~0.4.0",
"ipns": "^0.6.1",
"ipns": "^0.7.0",
"is-domain-name": "^1.0.1",
"is-ipfs": "~0.6.1",
"is-pull-stream": "~0.0.0",
"is-stream": "^2.0.0",
"iso-url": "~0.4.6",
"it-all": "^1.0.1",
"it-pipe": "^1.0.1",
"it-concat": "^1.0.0",
"it-glob": "0.0.7",
"it-last": "^1.0.1",
"it-pipe": "^1.1.0",
"it-to-stream": "^0.1.1",
"iterable-ndjson": "^1.1.0",
"jsondiffpatch": "~0.3.11",
"just-safe-set": "^2.1.0",
"kind-of": "^6.0.2",
"ky": "^0.15.0",
"ky-universal": "~0.3.0",
"libp2p": "^0.26.2",
"libp2p-bootstrap": "~0.9.3",
"libp2p-crypto": "^0.16.2",
"libp2p": "github:libp2p/js-libp2p#refactor/async-await",
"libp2p-bootstrap": "^0.10.2",
"libp2p-crypto": "^0.17.1",
"libp2p-delegated-content-routing": "^0.4.1",
"libp2p-delegated-peer-routing": "^0.3.1",
"libp2p-floodsub": "^0.18.0",
"libp2p-gossipsub": "~0.0.5",
"libp2p-kad-dht": "~0.16.0",
"libp2p-keychain": "^0.5.4",
"libp2p-mdns": "~0.12.0",
"libp2p-delegated-peer-routing": "^0.4.0",
"libp2p-floodsub": "^0.20.0",
"libp2p-gossipsub": "github:ChainSafe/gossipsub-js#71cb905983b125b50c64a9b75d745dfd7fb8f094",
"libp2p-kad-dht": "^0.18.3",
"libp2p-keychain": "^0.6.0",
"libp2p-mdns": "^0.13.0",
"libp2p-mplex": "^0.9.3",
"libp2p-record": "~0.7.0",
"libp2p-secio": "~0.11.0",
"libp2p-tcp": "^0.13.0",
"libp2p-webrtc-star": "~0.16.0",
"libp2p-websocket-star-multi": "~0.4.3",
"libp2p-websockets": "~0.12.3",
"lodash.flatten": "^4.4.0",
"mafmt": "^6.0.10",
"libp2p-secio": "^0.12.1",
"libp2p-tcp": "^0.14.2",
"libp2p-webrtc-star": "^0.17.0",
"libp2p-websockets": "^0.13.0",
"mafmt": "^7.0.0",
"merge-options": "^2.0.0",
"mime-types": "^2.1.21",
"mkdirp": "~0.5.1",
"mortice": "^2.0.0",
"multiaddr": "^6.1.1",
"multiaddr-to-uri": "^5.0.0",
"multiaddr": "^7.2.1",
"multiaddr-to-uri": "^5.1.0",
"multibase": "~0.6.0",
"multicodec": "^1.0.0",
"multihashes": "~0.4.14",
"multihashing-async": "^0.8.0",
"node-fetch": "^2.3.0",
"p-iteration": "^1.1.8",
"p-defer": "^3.0.0",
"p-queue": "^6.1.0",
"peer-book": "^0.9.1",
"peer-id": "~0.12.2",
"peer-info": "~0.15.1",
"parse-duration": "^0.1.2",
"peer-id": "^0.13.5",
"peer-info": "^0.17.0",
"pretty-bytes": "^5.3.0",
"progress": "^2.0.1",
"promise-nodeify": "^3.0.1",
"promisify-es6": "^1.0.3",
"protons": "^1.0.1",
"pull-abortable": "^4.1.1",
"pull-cat": "^1.1.11",
"pull-defer": "~0.2.3",
"pull-file": "^1.1.0",
"pull-mplex": "~0.1.1",
"pull-ndjson": "^0.2.0",
"pull-pushable": "^2.2.0",
"pull-sort": "^1.0.1",
"pull-stream": "^3.6.14",
"pull-stream-to-async-iterator": "^1.0.2",
"pull-stream-to-stream": "^2.0.0",
"pull-traverse": "^1.0.3",
"readable-stream": "^3.4.0",
"receptacle": "^1.3.2",
"semver": "^6.3.0",
"stream-to-pull-stream": "^1.7.3",
"superstruct": "~0.6.2",
"stream-to-it": "^0.2.0",
"streaming-iterables": "^4.1.1",
"tar-stream": "^2.0.0",
"temp": "~0.9.0",
"timeout-abort-controller": "^1.1.0",
"update-notifier": "^4.0.0",
"uri-to-multiaddr": "^3.0.1",
"uri-to-multiaddr": "^3.0.2",
"varint": "^5.0.0",
"yargs": "^15.0.1",
"yargs-promise": "^1.1.0"
},
"devDependencies": {
"aegir": "^20.4.1",
"async": "^2.6.3",
"base64url": "^3.0.1",
"clear-module": "^4.0.0",
"delay": "^4.1.0",
"detect-node": "^2.0.4",
"dir-compare": "^1.7.3",
"execa": "^3.0.0",
"form-data": "^3.0.0",
"hat": "0.0.3",
"interface-ipfs-core": "^0.128.0",
"ipfs-interop": "^0.2.0",
"ipfsd-ctl": "^1.0.2",
"libp2p-websocket-star": "~0.10.2",
"lodash": "^4.17.15",
"interface-ipfs-core": "github:ipfs/interface-js-ipfs-core#refactor/async-iterables",
"ipfs-interop": "github:ipfs/interop#refactor/async-await",
"ipfsd-ctl": "github:ipfs/js-ipfsd-ctl#feat/force-kill",
"ncp": "^2.0.0",
"p-event": "^4.1.0",
"p-map": "^3.0.0",
Expand Down
20 changes: 2 additions & 18 deletions src/cli/bin.js
Original file line number Diff line number Diff line change
Expand Up @@ -37,38 +37,22 @@ updateNotifier({ pkg, updateCheckInterval: oneWeek }).notify()

const cli = new YargsPromise(parser)

let getIpfs = null

// Apply command aliasing (eg `refs local` -> `refs-local`)
const args = commandAlias(process.argv.slice(2))
cli
.parse(args)
.then(({ data, argv }) => {
getIpfs = argv.getIpfs
if (data) {
print(data)
}
})
.catch(({ error, argv }) => {
getIpfs = argv && argv.getIpfs

if (error.code === InvalidRepoVersionError.code) {
error.message = 'Incompatible repo version. Migration needed. Pass --migrate for automatic migration'
}

if (error.message) {
print(error.message)
debug(error)
} else {
print('Unknown error, please re-run the command with DEBUG=ipfs:cli to see debug output')
debug(error)
}
print(error.message || 'Unknown error, please re-run the command with DEBUG=ipfs:cli to see debug output')
debug(error)

process.exit(1)
})
.finally(() => {
if (getIpfs && getIpfs.instance) {
const cleanup = getIpfs.rest[0]
return cleanup()
}
})
14 changes: 7 additions & 7 deletions src/cli/commands/add.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
'use strict'

const promisify = require('promisify-es6')
const { promisify } = require('util')
const getFolderSize = promisify(require('get-folder-size'))
const byteman = require('byteman')
const mh = require('multihashes')
Expand Down Expand Up @@ -238,20 +238,20 @@ module.exports = {
})
: argv.getStdin() // Pipe directly to ipfs.add

let finalHash
let finalCid

try {
for await (const file of ipfs._addAsyncIterator(source, options)) {
for await (const file of ipfs.add(source, options)) {
if (argv.silent) {
continue
}

if (argv.quieter) {
finalHash = file.hash
finalCid = file.cid
continue
}

const cid = cidToString(file.hash, { base: argv.cidBase })
const cid = cidToString(file.cid, { base: argv.cidBase })
let message = cid

if (!argv.quiet) {
Expand All @@ -266,7 +266,7 @@ module.exports = {
bar.terminate()
}

// Tweak the error message and add more relevant infor for the CLI
// Tweak the error message and add more relevant info for the CLI
if (err.code === 'ERR_DIR_NON_RECURSIVE') {
err.message = `'${err.path}' is a directory, use the '-r' flag to specify directories`
}
Expand All @@ -279,7 +279,7 @@ module.exports = {
}

if (argv.quieter) {
log(cidToString(finalHash, { base: argv.cidBase }))
log(cidToString(finalCid, { base: argv.cidBase }))
}
})())
}
Expand Down
2 changes: 1 addition & 1 deletion src/cli/commands/bitswap/stat.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ module.exports = {
stats.dupDataReceived = prettyBytes(stats.dupDataReceived.toNumber()).toUpperCase()
stats.wantlist = `[${stats.wantlist.length} keys]`
} else {
const wantlist = stats.wantlist.map((elem) => cidToString(elem['/'], { base: cidBase, upgrade: false }))
const wantlist = stats.wantlist.map(cid => cidToString(cid, { base: cidBase, upgrade: false }))
stats.wantlist = `[${wantlist.length} keys]
${wantlist.join('\n ')}`
}
Expand Down
2 changes: 1 addition & 1 deletion src/cli/commands/bitswap/wantlist.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ module.exports = {
resolve((async () => {
const ipfs = await getIpfs()
const list = await ipfs.bitswap.wantlist(peer)
list.Keys.forEach(k => print(cidToString(k['/'], { base: cidBase, upgrade: false })))
list.forEach(cid => print(cidToString(cid, { base: cidBase, upgrade: false })))
})())
}
}
12 changes: 3 additions & 9 deletions src/cli/commands/block/put.js
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
'use strict'

const bl = require('bl')
const fs = require('fs')
const multibase = require('multibase')
const promisify = require('promisify-es6')
const concat = require('it-concat')
const { cidToString } = require('../../../utils/cid')

module.exports = {
Expand Down Expand Up @@ -41,14 +40,9 @@ module.exports = {
let data

if (argv.block) {
data = await promisify(fs.readFile)(argv.block)
data = await fs.readFileSync(argv.block)
} else {
data = await new Promise((resolve, reject) => {
argv.getStdin().pipe(bl((err, input) => {
if (err) return reject(err)
resolve(input)
}))
})
data = (await concat(argv.getStdin())).slice()
}

const ipfs = await argv.getIpfs()
Expand Down
4 changes: 2 additions & 2 deletions src/cli/commands/block/rm.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ module.exports = {
const ipfs = await getIpfs()
let errored = false

for await (const result of ipfs.block._rmAsyncIterator(hash, {
for await (const result of ipfs.block.rm(hash, {
force,
quiet
})) {
Expand All @@ -34,7 +34,7 @@ module.exports = {
}

if (!quiet) {
print(result.error || 'removed ' + result.hash)
print(result.error ? result.error.message : `removed ${result.cid}`)
}
}

Expand Down
2 changes: 1 addition & 1 deletion src/cli/commands/block/stat.js
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ module.exports = {
resolve((async () => {
const ipfs = await getIpfs()
const stats = await ipfs.block.stat(key)
print('Key: ' + cidToString(stats.key, { base: cidBase }))
print('Key: ' + cidToString(stats.cid, { base: cidBase }))
print('Size: ' + stats.size)
})())
}
Expand Down
2 changes: 1 addition & 1 deletion src/cli/commands/cat.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ module.exports = {
resolve((async () => {
const ipfs = await getIpfs()

for await (const buf of ipfs._catAsyncIterator(ipfsPath, { offset, length })) {
for await (const buf of ipfs.cat(ipfsPath, { offset, length })) {
process.stdout.write(buf)
}
})())
Expand Down
Loading

0 comments on commit de01e54

Please sign in to comment.