Skip to content
This repository was archived by the owner on Feb 12, 2024. It is now read-only.

refactor: make add only work on single items #3167

Merged
merged 10 commits into from
Jul 16, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
222 changes: 169 additions & 53 deletions docs/core-api/FILES.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,92 +9,96 @@ _Explore the Mutable File System through interactive coding challenges in our [P
- [Parameters](#parameters)
- [Options](#options)
- [Returns](#returns)
- [`ipfs.addAll(source, [options])`](#ipfsaddallsource-options)
- [Parameters](#parameters-1)
- [Options](#options-1)
- [Returns](#returns-1)
- [Example](#example)
- [Notes](#notes)
- [Chunking options](#chunking-options)
- [Hash algorithms](#hash-algorithms)
- [Importing files from the file system](#importing-files-from-the-file-system)
- [Importing a file from a URL](#importing-a-file-from-a-url)
- [`ipfs.cat(ipfsPath, [options])`](#ipfscatipfspath-options)
- [Parameters](#parameters-1)
- [Options](#options-1)
- [Returns](#returns-1)
- [Example](#example-1)
- [`ipfs.get(ipfsPath, [options])`](#ipfsgetipfspath-options)
- [Parameters](#parameters-2)
- [Options](#options-2)
- [Returns](#returns-2)
- [Example](#example-2)
- [`ipfs.ls(ipfsPath)`](#ipfslsipfspath)
- [Example](#example-1)
- [`ipfs.get(ipfsPath, [options])`](#ipfsgetipfspath-options)
- [Parameters](#parameters-3)
- [Options](#options-3)
- [Returns](#returns-3)
- [Example](#example-3)
- [The Mutable Files API](#the-mutable-files-api)
- [`ipfs.files.chmod(path, mode, [options])`](#ipfsfileschmodpath-mode-options)
- [Example](#example-2)
- [`ipfs.ls(ipfsPath)`](#ipfslsipfspath)
- [Parameters](#parameters-4)
- [Options](#options-4)
- [Returns](#returns-4)
- [Example](#example-4)
- [`ipfs.files.cp(...from, to, [options])`](#ipfsfilescpfrom-to-options)
- [Example](#example-3)
- [The Mutable Files API](#the-mutable-files-api)
- [`ipfs.files.chmod(path, mode, [options])`](#ipfsfileschmodpath-mode-options)
- [Parameters](#parameters-5)
- [Options](#options-5)
- [Returns](#returns-5)
- [Example](#example-5)
- [Notes](#notes-1)
- [`ipfs.files.mkdir(path, [options])`](#ipfsfilesmkdirpath-options)
- [Example](#example-4)
- [`ipfs.files.cp(...from, to, [options])`](#ipfsfilescpfrom-to-options)
- [Parameters](#parameters-6)
- [Options](#options-6)
- [Returns](#returns-6)
- [Example](#example-6)
- [`ipfs.files.stat(path, [options])`](#ipfsfilesstatpath-options)
- [Example](#example-5)
- [Notes](#notes-1)
- [`ipfs.files.mkdir(path, [options])`](#ipfsfilesmkdirpath-options)
- [Parameters](#parameters-7)
- [Options](#options-7)
- [Returns](#returns-7)
- [Example](#example-7)
- [`ipfs.files.touch(path, [options])`](#ipfsfilestouchpath-options)
- [Example](#example-6)
- [`ipfs.files.stat(path, [options])`](#ipfsfilesstatpath-options)
- [Parameters](#parameters-8)
- [Options](#options-8)
- [Returns](#returns-8)
- [Example](#example-8)
- [`ipfs.files.rm(...paths, [options])`](#ipfsfilesrmpaths-options)
- [Example](#example-7)
- [`ipfs.files.touch(path, [options])`](#ipfsfilestouchpath-options)
- [Parameters](#parameters-9)
- [Options](#options-9)
- [Returns](#returns-9)
- [Example](#example-9)
- [`ipfs.files.read(path, [options])`](#ipfsfilesreadpath-options)
- [Example](#example-8)
- [`ipfs.files.rm(...paths, [options])`](#ipfsfilesrmpaths-options)
- [Parameters](#parameters-10)
- [Options](#options-10)
- [Returns](#returns-10)
- [Example](#example-10)
- [`ipfs.files.write(path, content, [options])`](#ipfsfileswritepath-content-options)
- [Example](#example-9)
- [`ipfs.files.read(path, [options])`](#ipfsfilesreadpath-options)
- [Parameters](#parameters-11)
- [Options](#options-11)
- [Returns](#returns-11)
- [Example](#example-11)
- [`ipfs.files.mv(...from, to, [options])`](#ipfsfilesmvfrom-to-options)
- [Example](#example-10)
- [`ipfs.files.write(path, content, [options])`](#ipfsfileswritepath-content-options)
- [Parameters](#parameters-12)
- [Options](#options-12)
- [Returns](#returns-12)
- [Example](#example-12)
- [Notes](#notes-2)
- [`ipfs.files.flush(path, [options])`](#ipfsfilesflushpath-options)
- [Example](#example-11)
- [`ipfs.files.mv(...from, to, [options])`](#ipfsfilesmvfrom-to-options)
- [Parameters](#parameters-13)
- [Options](#options-13)
- [Returns](#returns-13)
- [Example](#example-13)
- [`ipfs.files.ls(path, [options])`](#ipfsfileslspath-options)
- [Example](#example-12)
- [Notes](#notes-2)
- [`ipfs.files.flush(path, [options])`](#ipfsfilesflushpath-options)
- [Parameters](#parameters-14)
- [Options](#options-14)
- [Returns](#returns-14)
- [Example](#example-13)
- [`ipfs.files.ls(path, [options])`](#ipfsfileslspath-options)
- [Parameters](#parameters-15)
- [Options](#options-15)
- [Returns](#returns-15)
- [Example](#example-14)

## The Regular API
The regular, top-level API for add, cat, get and ls Files on IPFS

### `ipfs.add(data, [options])`

> Import files and data into IPFS.
> Import a file or data into IPFS.

#### Parameters

Expand All @@ -104,19 +108,132 @@ The regular, top-level API for add, cat, get and ls Files on IPFS

`data` may be:

* `Bytes` (alias for `Buffer`|`ArrayBuffer`|`TypedArray`) [single file]
* `Bloby` (alias for: `Blob`|`File`) [single file]
* `string` [single file]
* `FileObject` (see below for definition) [single file]
* `Iterable<number>` [single file]
* `Iterable<Bytes>` [single file]
* `Iterable<Bloby>` [multiple files]
* `Iterable<string>` [multiple files]
* `Iterable<FileObject>` [multiple files]
* `AsyncIterable<Bytes>` [single file]
* `AsyncIterable<Bloby>` [multiple files]
* `AsyncIterable<String>` [multiple files]
* `AsyncIterable<FileObject>` [multiple files]
* `Blob`
* `String`
* `Uint8Array`
* `FileObject` (see below for definition)
* `Iterable<Uint8Array>`
* `AsyncIterable<Uint8Array>`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest to go as far as drop Iterable<Uint8Array> and AsyncIterable<Uint8Array> from the list. I think convenience added here is no where near to the complexity it adds.

Iterables can trivially be turned into blobs:

ipfs.add(new Blob(chunks))

And both async and sync iterables could also trivially be turned into FileObjects

ipfs.add({ content: chunks })

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We support AsyncIterable<Uint8Array> in order to support adding node streams:

const fs = require('rs')

ipfs.add(fs.createReadStream('/path/to/file'))


`FileObject` is a plain JS object of the following form:

```js
{
// The path you want to the file to be accessible at from the root CID _after_ it has been added
path?: string
// The contents of the file (see below for definition)
content?: FileContent
// File mode to store the entry with (see https://en.wikipedia.org/wiki/File_system_permissions#Numeric_notation)
mode?: number | string
// The modification time of the entry (see below for definition)
mtime?: UnixTime
}
```

If no `path` is specified, then the item will be added to the root level and will be given a name according to it's CID.

If no `content` is passed, then the item is treated as an empty directory.

One of `path` or `content` _must_ be passed.

`FileContent` is one of the following types:

```js
Uint8Array | Blob | String | Iterable<Uint8Array> | AsyncIterable<Uint8Array>
```

`UnixTime` is one of the following types:

```js
Date | { secs: number, nsecs?: number } | number[]
```

As an object, `secs` is the number of seconds since (positive) or before (negative) the Unix Epoch began and `nsecs` is the number of nanoseconds since the last full second.

As an array of numbers, it must have two elements, as per the output of [`process.hrtime()`](https://nodejs.org/dist/latest/docs/api/process.html#process_process_hrtime_time).

#### Options

An optional object which may have the following keys:

| Name | Type | Default | Description |
| ---- | ---- | ------- | ----------- |
| chunker | `String` | `'size-262144` | chunking algorithm used to build ipfs DAGs |
| cidVersion | `Number` | `0` | the CID version to use when storing the data |
| hashAlg | `String` | `'sha2-256'` | multihash hashing algorithm to use |
| onlyHash | `boolean` | `false` | If true, will not add blocks to the blockstore |
| pin | `boolean` | `true` | pin this object when adding |
| progress | function | `undefined` | a function that will be called with the byte length of chunks as a file is added to ipfs |
| rawLeaves | `boolean` | `false` | if true, DAG leaves will contain raw file data and not be wrapped in a protobuf |
| trickle | `boolean` | `false` | if true will use the [trickle DAG](https://godoc.org/github.com/ipsn/go-ipfs/gxlibs/github.com/ipfs/go-unixfs/importer/trickle) format for DAG generation |
| wrapWithDirectory | `boolean` | `false` | Adds a wrapping node around the content |
| timeout | `Number` | `undefined` | A timeout in ms |
| signal | [AbortSignal][] | `undefined` | Can be used to cancel any long running requests started as a result of this call |

#### Returns

| Type | Description |
| -------- | -------- |
| `UnixFSEntry` | A object describing the added data |

Each yielded object is of the form:

```JavaScript
{
path: '/tmp/myfile.txt',
cid: CID('QmHash'),
mode: Number,
mtime: { secs: Number, nsecs: Number },
size: 123
}
```

#### Example

```js
const file = {
path: '/tmp/myfile.txt',
content: 'ABC'
}

const result of await ipfs.add(content)

console.info(result)

/*
Prints:
{
"path": "tmp",
"cid": CID("QmWXdjNC362aPDtwHPUE9o2VMqPeNeCQuTBTv1NsKtwypg"),
"mode": 493,
"mtime": { secs: Number, nsecs: Number },
"size": 67
}
*/
```

Now [ipfs.io/ipfs/Qm..pg/myfile.txt](https://ipfs.io/ipfs/QmWXdjNC362aPDtwHPUE9o2VMqPeNeCQuTBTv1NsKtwypg/myfile.txt) returns the "ABC" string.

### `ipfs.addAll(source, [options])`

> Import multiple files and data into IPFS.

#### Parameters

| Name | Type | Description |
| ---- | ---- | ----------- |
| source | Object | Data to import (see below) |

`source` may be:

* `Iterable<Blob>`
* `Iterable<String>`
* `Iterable<Uint8Array>`
* `Iterable<FileObject>`
* `AsyncIterable<Blob>`
* `AsyncIterable<String>`
* `AsyncIterable<Uint8Array>`
* `AsyncIterable<FileObject>`

`FileObject` is a plain JS object of the following form:

Expand All @@ -142,7 +259,7 @@ One of `path` or `content` _must_ be passed.
`FileContent` is one of the following types:

```js
Bytes | Bloby | string | Iterable<number> | Iterable<Bytes> | AsyncIterable<Bytes>
Uint8Array | Blob | String | Iterable<Uint8Array> | AsyncIterable<Uint8Array>
```

`UnixTime` is one of the following types:
Expand Down Expand Up @@ -179,7 +296,7 @@ An optional object which may have the following keys:

| Type | Description |
| -------- | -------- |
| `AsyncIterable<Object>` | An async iterable that yields objects describing the added data |
| `AsyncIterable<UnixFSEntry>` | An async iterable that yields objects describing the added data |

Each yielded object is of the form:

Expand All @@ -201,7 +318,7 @@ const files = [{
content: 'ABC'
}]

for await (const result of ipfs.add(content)) {
for await (const result of ipfs.addAll(content)) {
console.log(result)
}

Expand Down Expand Up @@ -266,7 +383,7 @@ const addOptions = {
timeout: 10000
};

for await (const file of ipfs.add(globSource('./docs', globSourceOptions), addOptions)) {
for await (const file of ipfs.addAll(globSource('./docs', globSourceOptions), addOptions)) {
console.log(file)
}

Expand Down Expand Up @@ -295,9 +412,8 @@ const { urlSource } = IPFS

const ipfs = await IPFS.create()

for await (const file of ipfs.add(urlSource('https://ipfs.io/images/ipfs-logo.svg'))) {
console.log(file)
}
const file = await ipfs.add(urlSource('https://ipfs.io/images/ipfs-logo.svg'))
console.log(file)

/*
{
Expand Down
12 changes: 6 additions & 6 deletions examples/browser-add-readable-stream/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -52,13 +52,13 @@ const streamFiles = async (ipfs, directory, files) => {
}
})

for await (const data of ipfs.add(stream)) {
log(`Added ${data.path} hash: ${data.hash}`)
const data = await ipfs.add(stream)

// The last data event will contain the directory hash
if (data.path === directory) {
return data.cid
}
log(`Added ${data.path} hash: ${data.hash}`)

// The last data event will contain the directory hash
if (data.path === directory) {
return data.cid
}
}

Expand Down
10 changes: 4 additions & 6 deletions examples/browser-browserify/src/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,11 @@ document.addEventListener('DOMContentLoaded', async () => {
async function store () {
const toStore = document.getElementById('source').value

for await (const file of node.add(toStore)) {
if (file && file.cid) {
console.log('successfully stored', file.cid)
const file = await node.add(toStore)

await display(file.cid)
}
}
console.log('successfully stored', file.cid)

await display(file.cid)
}

async function display (cid) {
Expand Down
17 changes: 8 additions & 9 deletions examples/browser-parceljs/public/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,18 +22,17 @@ document.addEventListener('DOMContentLoaded', async () => {

log(`The IPFS node version is ${version.version}`)

for await (const entry of node.add({
const entry = await node.add({
path: 'hello-parcel.txt',
content: 'Hello from parcel.js bundled ipfs example'
})) {
log(`This page deployed ${entry.path} to IPFS and its CID is ${entry.cid}`)
})
log(`This page deployed ${entry.path} to IPFS and its CID is ${entry.cid}`)

const buffers = []
const buffers = []

for await (const buf of node.cat(entry.cid)) {
buffers.push(buf)
}

log(`The contents of the file was: ${Buffer.concat(buffers).toString()}`)
for await (const buf of node.cat(entry.cid)) {
buffers.push(buf)
}

log(`The contents of the file was: ${Buffer.concat(buffers).toString()}`)
})
2 changes: 1 addition & 1 deletion examples/browser-readablestream/utils.js
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ const dragDrop = (ipfs) => {

const progress = log(`IPFS: Adding...`)

for await (const added of ipfs.add(files, {
for await (const added of ipfs.addAll(files, {
progress: (addedBytes) => {
progress.textContent = `IPFS: Adding ${addedBytes} bytes\r\n`
}
Expand Down
Loading