Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Commit

Permalink
refactor: make add only work on single items (#3167)
Browse files Browse the repository at this point in the history
`ipfs.add` now returns single items only.  Where a source would result in multiple items returned, only the last item is returned.  As a first pass `ipfs.add` still takes multiple items but only accepting single items is documented and considered supported.  In the future it may throw if multiple items are passed.

`ipfs.addAll` retains the previous behaviour of `ipfs.add`.

Examples:

```javascript
const { cid, path, mode, mtime } = await ipfs.add('Hello world')
```

```javascript
const { cid } = await ipfs.add(Uint8Array.from([0, 1, 2]))
```

```javascript
const { cid } = await ipfs.add(fs.createReadStream('/path/to/file.txt'))
```

```javascript
const { cid } = await ipfs.add({
  path: '/foo/bar/baz.txt',
  content: 'File content'
})

// Creates a DAG with multiple levels of directories.
// The returned cid is the CID for the root directory /foo

// You can retrieve the file content with an IPFS path
for await (const buf of ipfs.cat(`/ipfs/${cid}/bar/baz.txt`)) {
  ...
}

// Or get the CID of the nested file with ipfs.files.stat
const { cid: fileCid } = await ipfs.files.stat(`/ipfs/${cid}/bar/baz.txt`)

// or ipfs.dag.resolve
const { cid: fileCid } = await ipfs.dag.resolve(`/ipfs/${cid}/bar/baz.txt`)
```

```javascript
// To have `/foo` included in the ipfs path, wrap it in a directory:
const { cid } = await ipfs.add({
  path: '/foo/bar/baz.txt',
  content: 'File content'
}, {
  wrapWithDirectory: true
})

for await (const buf of ipfs.cat(`/ipfs/${cid}/foo/bar/baz.txt`)) {
  ...
}
```

BREAKING CHANGES:

- `ipfs.add` only works on single items - a Uint8Array, a String, an AsyncIterable<Uint8Array> etc
- `ipfs.addAll` works on multiple items
  • Loading branch information
achingbrain authored Jul 16, 2020
1 parent 8c122e8 commit 1760b89
Show file tree
Hide file tree
Showing 76 changed files with 979 additions and 704 deletions.
222 changes: 169 additions & 53 deletions docs/core-api/FILES.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,92 +9,96 @@ _Explore the Mutable File System through interactive coding challenges in our [P
- [Parameters](#parameters)
- [Options](#options)
- [Returns](#returns)
- [`ipfs.addAll(source, [options])`](#ipfsaddallsource-options)
- [Parameters](#parameters-1)
- [Options](#options-1)
- [Returns](#returns-1)
- [Example](#example)
- [Notes](#notes)
- [Chunking options](#chunking-options)
- [Hash algorithms](#hash-algorithms)
- [Importing files from the file system](#importing-files-from-the-file-system)
- [Importing a file from a URL](#importing-a-file-from-a-url)
- [`ipfs.cat(ipfsPath, [options])`](#ipfscatipfspath-options)
- [Parameters](#parameters-1)
- [Options](#options-1)
- [Returns](#returns-1)
- [Example](#example-1)
- [`ipfs.get(ipfsPath, [options])`](#ipfsgetipfspath-options)
- [Parameters](#parameters-2)
- [Options](#options-2)
- [Returns](#returns-2)
- [Example](#example-2)
- [`ipfs.ls(ipfsPath)`](#ipfslsipfspath)
- [Example](#example-1)
- [`ipfs.get(ipfsPath, [options])`](#ipfsgetipfspath-options)
- [Parameters](#parameters-3)
- [Options](#options-3)
- [Returns](#returns-3)
- [Example](#example-3)
- [The Mutable Files API](#the-mutable-files-api)
- [`ipfs.files.chmod(path, mode, [options])`](#ipfsfileschmodpath-mode-options)
- [Example](#example-2)
- [`ipfs.ls(ipfsPath)`](#ipfslsipfspath)
- [Parameters](#parameters-4)
- [Options](#options-4)
- [Returns](#returns-4)
- [Example](#example-4)
- [`ipfs.files.cp(...from, to, [options])`](#ipfsfilescpfrom-to-options)
- [Example](#example-3)
- [The Mutable Files API](#the-mutable-files-api)
- [`ipfs.files.chmod(path, mode, [options])`](#ipfsfileschmodpath-mode-options)
- [Parameters](#parameters-5)
- [Options](#options-5)
- [Returns](#returns-5)
- [Example](#example-5)
- [Notes](#notes-1)
- [`ipfs.files.mkdir(path, [options])`](#ipfsfilesmkdirpath-options)
- [Example](#example-4)
- [`ipfs.files.cp(...from, to, [options])`](#ipfsfilescpfrom-to-options)
- [Parameters](#parameters-6)
- [Options](#options-6)
- [Returns](#returns-6)
- [Example](#example-6)
- [`ipfs.files.stat(path, [options])`](#ipfsfilesstatpath-options)
- [Example](#example-5)
- [Notes](#notes-1)
- [`ipfs.files.mkdir(path, [options])`](#ipfsfilesmkdirpath-options)
- [Parameters](#parameters-7)
- [Options](#options-7)
- [Returns](#returns-7)
- [Example](#example-7)
- [`ipfs.files.touch(path, [options])`](#ipfsfilestouchpath-options)
- [Example](#example-6)
- [`ipfs.files.stat(path, [options])`](#ipfsfilesstatpath-options)
- [Parameters](#parameters-8)
- [Options](#options-8)
- [Returns](#returns-8)
- [Example](#example-8)
- [`ipfs.files.rm(...paths, [options])`](#ipfsfilesrmpaths-options)
- [Example](#example-7)
- [`ipfs.files.touch(path, [options])`](#ipfsfilestouchpath-options)
- [Parameters](#parameters-9)
- [Options](#options-9)
- [Returns](#returns-9)
- [Example](#example-9)
- [`ipfs.files.read(path, [options])`](#ipfsfilesreadpath-options)
- [Example](#example-8)
- [`ipfs.files.rm(...paths, [options])`](#ipfsfilesrmpaths-options)
- [Parameters](#parameters-10)
- [Options](#options-10)
- [Returns](#returns-10)
- [Example](#example-10)
- [`ipfs.files.write(path, content, [options])`](#ipfsfileswritepath-content-options)
- [Example](#example-9)
- [`ipfs.files.read(path, [options])`](#ipfsfilesreadpath-options)
- [Parameters](#parameters-11)
- [Options](#options-11)
- [Returns](#returns-11)
- [Example](#example-11)
- [`ipfs.files.mv(...from, to, [options])`](#ipfsfilesmvfrom-to-options)
- [Example](#example-10)
- [`ipfs.files.write(path, content, [options])`](#ipfsfileswritepath-content-options)
- [Parameters](#parameters-12)
- [Options](#options-12)
- [Returns](#returns-12)
- [Example](#example-12)
- [Notes](#notes-2)
- [`ipfs.files.flush(path, [options])`](#ipfsfilesflushpath-options)
- [Example](#example-11)
- [`ipfs.files.mv(...from, to, [options])`](#ipfsfilesmvfrom-to-options)
- [Parameters](#parameters-13)
- [Options](#options-13)
- [Returns](#returns-13)
- [Example](#example-13)
- [`ipfs.files.ls(path, [options])`](#ipfsfileslspath-options)
- [Example](#example-12)
- [Notes](#notes-2)
- [`ipfs.files.flush(path, [options])`](#ipfsfilesflushpath-options)
- [Parameters](#parameters-14)
- [Options](#options-14)
- [Returns](#returns-14)
- [Example](#example-13)
- [`ipfs.files.ls(path, [options])`](#ipfsfileslspath-options)
- [Parameters](#parameters-15)
- [Options](#options-15)
- [Returns](#returns-15)
- [Example](#example-14)

## The Regular API
The regular, top-level API for add, cat, get and ls Files on IPFS

### `ipfs.add(data, [options])`

> Import files and data into IPFS.
> Import a file or data into IPFS.
#### Parameters

Expand All @@ -104,19 +108,132 @@ The regular, top-level API for add, cat, get and ls Files on IPFS

`data` may be:

* `Bytes` (alias for `Buffer`|`ArrayBuffer`|`TypedArray`) [single file]
* `Bloby` (alias for: `Blob`|`File`) [single file]
* `string` [single file]
* `FileObject` (see below for definition) [single file]
* `Iterable<number>` [single file]
* `Iterable<Bytes>` [single file]
* `Iterable<Bloby>` [multiple files]
* `Iterable<string>` [multiple files]
* `Iterable<FileObject>` [multiple files]
* `AsyncIterable<Bytes>` [single file]
* `AsyncIterable<Bloby>` [multiple files]
* `AsyncIterable<String>` [multiple files]
* `AsyncIterable<FileObject>` [multiple files]
* `Blob`
* `String`
* `Uint8Array`
* `FileObject` (see below for definition)
* `Iterable<Uint8Array>`
* `AsyncIterable<Uint8Array>`

`FileObject` is a plain JS object of the following form:

```js
{
// The path you want to the file to be accessible at from the root CID _after_ it has been added
path?: string
// The contents of the file (see below for definition)
content?: FileContent
// File mode to store the entry with (see https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation)
mode?: number | string
// The modification time of the entry (see below for definition)
mtime?: UnixTime
}
```

If no `path` is specified, then the item will be added to the root level and will be given a name according to it's CID.

If no `content` is passed, then the item is treated as an empty directory.

One of `path` or `content` _must_ be passed.

`FileContent` is one of the following types:

```js
Uint8Array | Blob | String | Iterable<Uint8Array> | AsyncIterable<Uint8Array>
```

`UnixTime` is one of the following types:

```js
Date | { secs: number, nsecs?: number } | number[]
```

As an object, `secs` is the number of seconds since (positive) or before (negative) the Unix Epoch began and `nsecs` is the number of nanoseconds since the last full second.

As an array of numbers, it must have two elements, as per the output of [`process.hrtime()`](https://nodejs.org/dist/latest/docs/api/process.html#process_process_hrtime_time).

#### Options

An optional object which may have the following keys:

| Name | Type | Default | Description |
| ---- | ---- | ------- | ----------- |
| chunker | `String` | `'size-262144` | chunking algorithm used to build ipfs DAGs |
| cidVersion | `Number` | `0` | the CID version to use when storing the data |
| hashAlg | `String` | `'sha2-256'` | multihash hashing algorithm to use |
| onlyHash | `boolean` | `false` | If true, will not add blocks to the blockstore |
| pin | `boolean` | `true` | pin this object when adding |
| progress | function | `undefined` | a function that will be called with the byte length of chunks as a file is added to ipfs |
| rawLeaves | `boolean` | `false` | if true, DAG leaves will contain raw file data and not be wrapped in a protobuf |
| trickle | `boolean` | `false` | if true will use the [trickle DAG](https://godoc.org/github.com/ipsn/go-ipfs/gxlibs/github.com/ipfs/go-unixfs/importer/trickle) format for DAG generation |
| wrapWithDirectory | `boolean` | `false` | Adds a wrapping node around the content |
| timeout | `Number` | `undefined` | A timeout in ms |
| signal | [AbortSignal][] | `undefined` | Can be used to cancel any long running requests started as a result of this call |

#### Returns

| Type | Description |
| -------- | -------- |
| `UnixFSEntry` | A object describing the added data |

Each yielded object is of the form:

```JavaScript
{
path: '/tmp/myfile.txt',
cid: CID('QmHash'),
mode: Number,
mtime: { secs: Number, nsecs: Number },
size: 123
}
```

#### Example

```js
const file = {
path: '/tmp/myfile.txt',
content: 'ABC'
}

const result of await ipfs.add(content)

console.info(result)

/*
Prints:
{
"path": "tmp",
"cid": CID("QmWXdjNC362aPDtwHPUE9o2VMqPeNeCQuTBTv1NsKtwypg"),
"mode": 493,
"mtime": { secs: Number, nsecs: Number },
"size": 67
}
*/
```

Now [ipfs.io/ipfs/Qm..pg/myfile.txt](https://ipfs.io/ipfs/QmWXdjNC362aPDtwHPUE9o2VMqPeNeCQuTBTv1NsKtwypg/myfile.txt) returns the "ABC" string.

### `ipfs.addAll(source, [options])`

> Import multiple files and data into IPFS.
#### Parameters

| Name | Type | Description |
| ---- | ---- | ----------- |
| source | Object | Data to import (see below) |

`source` may be:

* `Iterable<Blob>`
* `Iterable<String>`
* `Iterable<Uint8Array>`
* `Iterable<FileObject>`
* `AsyncIterable<Blob>`
* `AsyncIterable<String>`
* `AsyncIterable<Uint8Array>`
* `AsyncIterable<FileObject>`

`FileObject` is a plain JS object of the following form:

Expand All @@ -142,7 +259,7 @@ One of `path` or `content` _must_ be passed.
`FileContent` is one of the following types:

```js
Bytes | Bloby | string | Iterable<number> | Iterable<Bytes> | AsyncIterable<Bytes>
Uint8Array | Blob | String | Iterable<Uint8Array> | AsyncIterable<Uint8Array>
```

`UnixTime` is one of the following types:
Expand Down Expand Up @@ -179,7 +296,7 @@ An optional object which may have the following keys:

| Type | Description |
| -------- | -------- |
| `AsyncIterable<Object>` | An async iterable that yields objects describing the added data |
| `AsyncIterable<UnixFSEntry>` | An async iterable that yields objects describing the added data |

Each yielded object is of the form:

Expand All @@ -201,7 +318,7 @@ const files = [{
content: 'ABC'
}]

for await (const result of ipfs.add(content)) {
for await (const result of ipfs.addAll(content)) {
console.log(result)
}

Expand Down Expand Up @@ -266,7 +383,7 @@ const addOptions = {
timeout: 10000
};

for await (const file of ipfs.add(globSource('./docs', globSourceOptions), addOptions)) {
for await (const file of ipfs.addAll(globSource('./docs', globSourceOptions), addOptions)) {
console.log(file)
}

Expand Down Expand Up @@ -295,9 +412,8 @@ const { urlSource } = IPFS

const ipfs = await IPFS.create()

for await (const file of ipfs.add(urlSource('https://ipfs.io/images/ipfs-logo.svg'))) {
console.log(file)
}
const file = await ipfs.add(urlSource('https://ipfs.io/images/ipfs-logo.svg'))
console.log(file)

/*
{
Expand Down
12 changes: 6 additions & 6 deletions examples/browser-add-readable-stream/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ const streamFiles = async (ipfs, directory, files) => {
}
})

for await (const data of ipfs.add(stream)) {
log(`Added ${data.path} hash: ${data.hash}`)
const data = await ipfs.add(stream)

// The last data event will contain the directory hash
if (data.path === directory) {
return data.cid
}
log(`Added ${data.path} hash: ${data.hash}`)

// The last data event will contain the directory hash
if (data.path === directory) {
return data.cid
}
}

Expand Down
10 changes: 4 additions & 6 deletions examples/browser-browserify/src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,11 @@ document.addEventListener('DOMContentLoaded', async () => {
async function store () {
const toStore = document.getElementById('source').value

for await (const file of node.add(toStore)) {
if (file && file.cid) {
console.log('successfully stored', file.cid)
const file = await node.add(toStore)

await display(file.cid)
}
}
console.log('successfully stored', file.cid)

await display(file.cid)
}

async function display (cid) {
Expand Down
17 changes: 8 additions & 9 deletions examples/browser-parceljs/public/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,17 @@ document.addEventListener('DOMContentLoaded', async () => {

log(`The IPFS node version is ${version.version}`)

for await (const entry of node.add({
const entry = await node.add({
path: 'hello-parcel.txt',
content: 'Hello from parcel.js bundled ipfs example'
})) {
log(`This page deployed ${entry.path} to IPFS and its CID is ${entry.cid}`)
})
log(`This page deployed ${entry.path} to IPFS and its CID is ${entry.cid}`)

const buffers = []
const buffers = []

for await (const buf of node.cat(entry.cid)) {
buffers.push(buf)
}

log(`The contents of the file was: ${Buffer.concat(buffers).toString()}`)
for await (const buf of node.cat(entry.cid)) {
buffers.push(buf)
}

log(`The contents of the file was: ${Buffer.concat(buffers).toString()}`)
})
2 changes: 1 addition & 1 deletion examples/browser-readablestream/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ const dragDrop = (ipfs) => {

const progress = log(`IPFS: Adding...`)

for await (const added of ipfs.add(files, {
for await (const added of ipfs.addAll(files, {
progress: (addedBytes) => {
progress.textContent = `IPFS: Adding ${addedBytes} bytes\r\n`
}
Expand Down
Loading

0 comments on commit 1760b89

Please sign in to comment.