Skip to content

Commit d11d0ec

Browse files
jbenetjbenet
committed
Merge pull request #38 from ipfs/overviews
Added some useful overviews
2 parents 6d8f205 + 45b5d5f commit d11d0ec

File tree

3 files changed

+387
-0
lines changed

3 files changed

+387
-0
lines changed

‎overviews/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# quick-overview docs
2+
3+
These are simple documents meant to be provide quick overviews to various parts of the protocol.
4+
5+
- [Implementing HTTP API bindings](implement-api-bindings.md)
6+
- [Implementing IPFS itself](implement-ipfs.md)

‎overviews/implement-api-bindings.md

Lines changed: 175 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,175 @@
1+
# IPFS API Implementation Doc
2+
3+
This short document aims to give a quick guide to anyone implementing API bindings for IPFS implementations-- in particular go-ipfs.
4+
5+
Sections:
6+
- IPFS Types
7+
- API Transports
8+
- API Commands
9+
- Implementing bindings for the HTTP API
10+
11+
## IPFS Types
12+
13+
IPFS uses a set of type value that is useful to enumerate up front:
14+
15+
- `<ipfs-path>` is unix-style path, beginning with `/ipfs/<hash>/...` or `/ipns/<hash>/...` or `/ipns/<domain>/...`.
16+
- `<hash>` is a base58 encoded [multihash](https://github.com/jbenet/multihash) (there are [many implementations](https://github.com/jbenet/multihash#implementations)). Usually the hash of an ipfs object (or merkle dag node).
17+
18+
A note on streams: IPFS is a streaming protocol. Everything about it can be streamed. When importing files, API requests should aim to stream the data in, and handle back-pressure correctly, so that the IPFS node can handle it sequentially without too much memory pressure. (If using HTTP, this is typically handled for you by writes to the request body blocking.)
19+
20+
## API Transports
21+
22+
Like with everything else, IPFS aims to be flexible regarding the API transports. Currently, the [go-ipfs](https://github.com/ipfs/go-ipfs) implementation supports both an in-process API and an HTTP api. More can be added easily, by mapping the API functions over a transport. (This is similar to how gRPC is also _mapped on top of transports_, like HTTP).
23+
24+
Mapping to a transport involves leveraging the transport's features to express function calls. For example:
25+
26+
### CLI API Transport
27+
28+
In the commandline, IPFS uses a traditional flag and arg-based mapping, where:
29+
- the first arguments selects the command, as in git - e.g. `ipfs object get`
30+
- the flags specify options - e.g. `--enc=protobuf -q`
31+
- the rest are positional arguments - e.g. `ipfs object patch <hash1> add-link foo <hash2>`
32+
- files are specified by filename, or through stdin
33+
34+
(NOTE: when go-ipfs runs the daemon, the CLI API is actually converted to HTTP calls. otherwise, they execute in the same process)
35+
36+
### HTTP API Transport
37+
38+
In HTTP, our API layering uses a REST-like mapping, where:
39+
- the URL path selects the command - e.g `/object/get`
40+
- the URL query string implements option arguments - e.g. `&enc=protobuf&q=true`
41+
- the URL query also implements positional arguments - e.g. `&arg=<hash1>&arg=add-link&arg=foo&arg=<hash2>`
42+
- the request body streams file data - reads files or stdin
43+
- multiple streams are muxed with multipart (todo: add tar stream support)
44+
45+
46+
## API Commands
47+
48+
There is a "standard IPFS API" with a set of commands, which we are documenting clearly soon. But this is not yet extracted into its own document. Perhaps -- as part of this API Bindings effort -- we can document it all. It is currently defined as "all the commands exposed by the go-ipfs implementation". You can see [a listing here](https://github.com/ipfs/go-ipfs/blob/916f987de2c35db71815b54bbb9a0a71df829838/core/commands/root.go#L82-L111), or by running `ipfs commands` locally. **The good news is: we should be able to easily write a program that outputs a markdown API specification!**
49+
50+
(Note: the go-ipfs [commands library](https://github.com/ipfs/go-ipfs/tree/916f987de2c35db71815b54bbb9a0a71df829838/commands) also makes sure to keep the CLI and the HTTP API exactly in sync.)
51+
52+
## Implementing bindings for the HTTP API
53+
54+
As mentioned above, the API commands map to HTTP with:
55+
- the URL path selects the command - e.g `/object/get`
56+
- the URL query string implements option arguments - e.g. `&enc=protobuf&q=true`
57+
- the URL query also implements positional arguments - e.g. `&arg=<hash1>&arg=add-link&arg=foo&arg=<hash2>`
58+
- the request body streams file data - reads files or stdin
59+
- multiple streams are muxed with multipart (todo: add tar stream support)
60+
61+
To date, we have two different HTTP API clients:
62+
63+
- [js-ipfs-api](https://github.com/ipfs/js-ipfs-api) - simple javascript wrapper -- best to look at
64+
- [go-ipfs/commands/http](https://github.com/ipfs/go-ipfs/tree/916f987de2c35db71815b54bbb9a0a71df829838/commands/http) - generalized transport based on the [command definitions](https://github.com/ipfs/go-ipfs/tree/916f987de2c35db71815b54bbb9a0a71df829838/core/commands)
65+
66+
The Go implementation is good to answer harder questions, like how is multipart handled, or what headers should be set in edge conditions. But the javascript implementation is very concise, and easy to follow.
67+
68+
### Anatomy of js-ipfs-api
69+
70+
Currently, js-ipfs-api has three main files
71+
- [src/index.js](https://github.com/ipfs/js-ipfs-api/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/index.js) defines the functions clients of the API module will use. uses `RequestAPI`, and translates function call parameters to the API almost directly.
72+
- [src/get-files-stream.js](https://github.com/ipfs/js-ipfs-api/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/get-files-stream.js) implements the hardest part: file streaming. This one uses multipart.
73+
- [src/request-api.js](https://github.com/ipfs/js-ipfs-api/blob/66d1462bd02181d46e8baf4cd9d476b213426ad8/src/request-api.js) generic function call to perform the actual HTTP requests
74+
75+
### Note on multipart + inspecting requests
76+
77+
Despite all the generalization spoken about above, the IPFS API is actually very simple. You can inspect all the requests made with `nc` and the `--api` option (as of [this PR](https://github.com/ipfs/go-ipfs/pull/1598), or `0.3.8`):
78+
79+
```
80+
> nc -l 5002 &
81+
> ipfs --api /ip4/127.0.0.1/tcp/5002 swarm addrs local --enc=json
82+
POST /api/v0/version?enc=json&stream-channels=true HTTP/1.1
83+
Host: 127.0.0.1:5002
84+
User-Agent: /go-ipfs/0.3.8/
85+
Content-Length: 0
86+
Content-Type: application/octet-stream
87+
Accept-Encoding: gzip
88+
89+
90+
```
91+
92+
The only hard part is getting the file streaming right. It is (now) fairly easy to stream files to go-ipfs using multipart. Basically, we end up with HTTP requests like this:
93+
94+
```
95+
> nc -l 5002 &
96+
> ipfs --api /ip4/127.0.0.1/tcp/5002 add -r ~/demo/basic/test
97+
POST /api/v0/add?encoding=json&progress=true&r=true&stream-channels=true HTTP/1.1
98+
Host: 127.0.0.1:5002
99+
User-Agent: /go-ipfs/0.3.8/
100+
Transfer-Encoding: chunked
101+
Content-Disposition: form-data: name="files"
102+
Content-Type: multipart/form-data; boundary=2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f
103+
Accept-Encoding: gzip
104+
105+
1
106+
-
107+
e5
108+
-2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f
109+
Content-Disposition: form-data; name="file"; filename="test"
110+
Content-Type: multipart/mixed; boundary=acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be
111+
112+
113+
9c
114+
--acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be
115+
Content-Disposition: file; filename="test%2Fbar"
116+
Content-Type: application/octet-stream
117+
118+
119+
4
120+
bar
121+
122+
dc
123+
124+
--acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be
125+
Content-Disposition: file; filename="test%2Fbaz"
126+
Content-Type: multipart/mixed; boundary=2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd
127+
128+
129+
a0
130+
--2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd
131+
Content-Type: application/octet-stream
132+
Content-Disposition: file; filename="test%2Fbaz%2Fb"
133+
134+
135+
4
136+
bar
137+
138+
a2
139+
140+
--2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd
141+
Content-Disposition: file; filename="test%2Fbaz%2Ff"
142+
Content-Type: application/octet-stream
143+
144+
145+
4
146+
foo
147+
148+
44
149+
150+
--2799ac77a72ef7b8a0281945806b9f9a28f7681145aa8e91b052d599b2dd--
151+
152+
9e
153+
154+
--acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be
155+
Content-Disposition: file; filename="test%2Ffoo"
156+
Content-Type: application/octet-stream
157+
158+
159+
4
160+
foo
161+
162+
44
163+
164+
--acdb172fe12f25e8ffae9981ce6f4580abdefb0cae3ceebe464d802866be--
165+
166+
44
167+
168+
--2186ef15d8f2c4f100af72d6d345afe36a4d17ef11264ec5b8ec4436447f--
169+
170+
0
171+
172+
```
173+
174+
Which produces: http://gateway.ipfs.io/ipfs/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv
175+

‎overviews/implement-ipfs.md

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# IPFS Implementation Doc
2+
3+
This short document aims to give a quick guide to anyone implementing IPFS -- it is modelled after go-ipfs, and serving as a template for js-ipfs and py-ipfs.
4+
5+
Sections:
6+
- IPFS Types
7+
- API Transports
8+
- API Commands
9+
- Implementing bindings for the HTTP API
10+
11+
## Libraries First
12+
13+
There are a number of non-ipfs specific things that have been built for ipfs, that ipfs depends on. Implement these first
14+
15+
### The Multis
16+
17+
There are a number of self-describing protocols/formats in use all over ipfs.
18+
19+
- [multiaddr](https://github.com/jbenet/multiaddr)
20+
- [multihash](https://github.com/jbenet/multihash)
21+
- [multicodec](https://github.com/jbenet/multicodec)
22+
- [multistream](https://github.com/jbenet/multistream)
23+
24+
### libp2p
25+
26+
All complex peer-to-peer protocols for IPFS have been abstracted out into a separate library called libp2p. libp2p is a thin veneer over a wealth of modules that interface well with each other.
27+
28+
Implementations:
29+
- [go-libp2p](https://github.com/ipfs/go-libp2p)
30+
- [js-libp2p](https://github.com/ipfs/js-libp2p)
31+
32+
33+
libp2p may in fact be _the bulk_ of an ipfs implementation. the rest is very simple.
34+
35+
## Core Pieces
36+
37+
### IPLD
38+
39+
IPLD is the format for IPFS objects, but it can be used outside of ipfs (hence a module). Its layered on top of multihash and multicodec, and provides the heart of ipfs: the merkledag.
40+
41+
Implementations:
42+
- [go-ipld](https://github.com/ipfs/go-ipld)
43+
- [js-ipld](https://github.com/ipfs/js-ipld)
44+
45+
### IPRS
46+
47+
IPRS is the record system for IPFS, but it can be used outside of ipfs (hence a module). This deals with p2p system records -- it is also used by libp2p.
48+
49+
Implementations:
50+
- [go-iprs](https://github.com/ipfs/go-iprs)
51+
- [js-iprs](https://github.com/ipfs/js-iprs)
52+
53+
### IPNS
54+
55+
IPNS provides name resolution on top of IPRS -- and a choice of record routing system.
56+
57+
### IPFS-Repo
58+
59+
The IFPS-Repo is an IPFS Node's "local storage" or "database", though the storage may not be in a database nor local at all (e.g. s3-repo). There are common formats so that multiple implementations can read and write to the same repos. Though today we only have one repo format, more are easy to add so that we can create IPFS nodes on top of other storage solutions.
60+
61+
Implementations:
62+
- [go-ipfs-repo](https://github.com/ipfs/go-ipfs-repo)
63+
- [go-ipfs-repo-fs](https://github.com/ipfs/go-ipfs-repo/fs) - in filesystem
64+
- [go-ipfs-repo-s3](https://github.com/ipfs/go-ipfs-repo/s3) - in amazon s3
65+
- [js-ipfs-repo](https://github.com/ipfs/js-ipfs-repo)
66+
- [js-ipfs-repo-fs](https://github.com/ipfs/js-ipfs-repo/fs) - in filesystem
67+
- [js-ipfs-repo-browser](https://github.com/ipfs/js-ipfs-repo/browser) - in local storage
68+
69+
## IPFS Core
70+
71+
The Core of IPFS is an interface of functions layered over all of the other pieces.
72+
73+
### IPFS Node
74+
75+
The IPFS Node is an entity that bundles all the other pieces, and implements the interface (below). In its most basic sense, an IPFS node is really just:
76+
77+
```go
78+
type ipfs.Node struct {
79+
80+
Config // has a configuration
81+
repo.Repo // has a Repo for storing all the local data
82+
libp2p.Node // has an embedded libp2p.Node, and thus a peer.ID, and keys.
83+
dag.Store // has a DAG Store (over the repo + network)
84+
85+
}
86+
```
87+
88+
IPFS itself is very, very simple. The complexity lies within `libp2p.Node` and how the different IPFS commands should run depending on the `libp2p.Node` configuration.
89+
90+
### IPFS Node Config
91+
92+
IPFS Nodes can be configured. The basic configuration format is a JSON file, and so naturally converters to other formats can be made. eventually, the configuration will be an ipfs object itself.
93+
94+
The config is stored in the IPFS Repo, but is separate because some implementations may give it knowledge of other packages (like routing, http, etc)
95+
96+
### IPFS Interface or API
97+
98+
The IPFS Interface or API (not to be confused with the IPFS HTTP API) is the set of functions that IPFS Nodes must support. These are classified into sections, like _node, network, data, util_ etc.
99+
100+
The IPFS Interface can be implemented:
101+
- as a library - first and foremost
102+
- as a commandline toolchain, so users can use it directly
103+
- as RPC API, so that other programs could use it
104+
- over HTTP (the IPFS HTTP API)
105+
- over unix domain sockets
106+
- over IPC
107+
108+
One goal for the core interface libraries is to produce an interface that could operate on a local or a remote node. This means that, for example:
109+
110+
```go
111+
func Cat(n ipfs.Node, p ipfs.Path) io.Reader { ... }
112+
```
113+
should be able to work whether `n` represents a local node (in-process, local storage), or a remote node (over an RPC API, say HTTP).
114+
115+
_**For now, i list these from the commandline, but the goal is to produce a proper typed function interface/API that we can all agree on.**_
116+
117+
#### Node Commands
118+
119+
These are the for the node itself.
120+
121+
- ipfs init
122+
- ipfs config
123+
- ipfs repo
124+
- ipfs repo gc
125+
- ipfs stats
126+
- ipfs diag
127+
128+
#### Data Commands
129+
130+
- ipfs block
131+
- ipfs object
132+
- ipfs {cat, ls, refs}
133+
- ipfs pin
134+
- ipfs files
135+
- ipfs tar
136+
- ipfs resolve
137+
138+
#### Network Commands
139+
140+
These are carried over from libp2p, so ideally the libp2p implementations do the heavy lifting here.
141+
142+
- ipfs id
143+
- ipfs ping
144+
- ipfs swarm
145+
- ipfs exchange
146+
- ipfs routing
147+
- ipfs bitswap
148+
- ipfs bootstrap
149+
150+
#### Naming commands
151+
152+
These are carried over from IPNS (can make that its own tool/lib).
153+
154+
- ipfs dns
155+
- ipfs name
156+
157+
#### Tool Commands
158+
159+
- ipfs log
160+
- ipfs update
161+
- ipfs version
162+
- ipfs tour
163+
- ipfs daemon
164+
165+
## IPFS Datastructures and Data Handling
166+
167+
There are many useful datastructures on top of IPFS. Things like `unixfs`, `tar`, `keychain`, etc. And there are a number of ways of importing data -- whether posix files or not.
168+
169+
### IPLD Data Importing
170+
171+
Importing data into IPFS can be done in a variety of ways. These are use-case specific, produce different datastructures, produce different graph topologies, and so on. These are not _strictly_ needed in an IPFS implementation, but definitely make it more useful. They are really tools on top of IPLD though, so these can be generic and separate from IPFS itself.
172+
173+
- graph topologies - shape of the graphs
174+
- balanced - dumb, dead simple
175+
- trickledag - optimized for seeking
176+
- live stream
177+
- database indices
178+
- file chunking - how to split a continuous stream/file
179+
- fixed size
180+
- rabin fingerprinting
181+
- format chunking (use knowledge of formats, e.g. audio, video, etc)
182+
- special format datastructures
183+
- tar
184+
- document formats - pdf, doc, etc
185+
- audio and video formats - ogg, mpeg, etc
186+
- container and vm images
187+
- and many more
188+
189+
### `unixfs` datastructure
190+
191+
It's worth mentioning the `unixfs` datastructure, as it provides support for representing unix (posix) files in ipfs. It's simple, but powerful And it is first class, in that several basic commands make use of it.
192+
193+
### Interesting Data Structure questions
194+
195+
**interfacing with a variety of data structures**
196+
197+
We are still figuring out good ways to make all the different data structures play well with various commands -- there is some complexity when it comes to implementing things like `ipfs cat` -- it currently outputs the data of a `unixfs.File`, but it could do something for other graph objects too. Ideally, we could figure out common ways of making this work, If you have ideas, please discuss.
198+
199+
**graph mapping**
200+
201+
Sometimes one graph maps to another, for example a unixfs graph shards big files and big directories into smaller units and transparently presents them to the user for commands such as `ipfs cat` and `ipfs ls`.
202+
203+
**mixing data structures**
204+
205+
Some data structures are meant to be interspersed with others, meaning that they provide meaning to arbitrary things. One example is a `keychain.Signature` a cryptographic signature on any other object. Another example is a `versioning.Commit` which represents a specific revision in a version history over any other object. It is still not entirely clear how to build nice tooling that handles these transparently.
206+

0 commit comments

Comments
 (0)