-
Notifications
You must be signed in to change notification settings - Fork 26
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
169 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,169 @@ | ||
| <placeholder> | ||
| # IPFS Go Core Dev Team - Roadmap 2019 | ||
| The Go IPFS Working Group is responsible for the development of the Go IPFS implementation and participating in the planning and development of IPFS in general. | ||
|
|
||
| ### Responsibilities include: | ||
| * Implement new features in the Go IPFS implementation | ||
| * Support Go IPFS, fixing bugs and improving stability and performance | ||
| * Develop specifications for new IPFS protocols and capabilities | ||
|
|
||
| ## Current Projects | ||
| * go-ipfs | ||
| * IPLD | ||
| * gx | ||
| * Data Exchange/Transfer - improve performance of data transfer | ||
| * Data Management - improve ability of Go IPFS to manage large data sets | ||
| * Provider Strategies - improve efficiency of DHT use | ||
| * Commands Lib | ||
| * API | ||
| * Base32 CIDv1 | ||
|
|
||
| ## Major 2019 goals to achieve | ||
| 1. Benchmarks for Go IPFS that measure IPFS behavior across data lifecycle for a broad set of inputs. | ||
| 2. ‘Indefinite’ refactors are complete; code is maintainable and understandable | ||
| 3. Improve data transfer speed so that it is at least 0.8 times that of bittorrent/rsync, (whichever is faster) for a given set of peers and data | ||
| 4. Support adding/getting/pinning large volumes of data efficiently. Go-IPFS can add (and provide) 10TB of small files (< 1MB each) at > 0.5 times the speed of a simple dd] copy. | ||
| 5. Make storing, looking up, and retrieving provider data in the DHT reliable and fast. | ||
| 6. Complete, documented, performant, stable APIs for interacting with a running Go IPFS node. | ||
|
|
||
| ## Milestones | ||
|
|
||
| ### 📦 Package managers | ||
| * M (P0): The IPFS core team understands the needs of package management communities and engages their interest | ||
| * Meet people from various package management communities -- both distros and language tooling -- and learn about their needs | ||
| * Support interested package manager communities with DRIs | ||
| * F-droid | ||
| * Guix | ||
| * Snapshots.debian.org ? | ||
| * M (P0): We have understanding and documentation of what kind of scale (in terms of size at one time; and change in size over time) mirroring a package manager archive requires. | ||
| * Ex: document how long mirroring npm into IPFS takes | ||
| * Ex: VictorBjelkholm/arch-mirror also exists, let’s make sure we gather stats | ||
| * M (P1): Develop documentation for what Package Distribution with IPFS should look like. | ||
| * There is documentation and guidelines to help maintainers from existing package management communities learn about IPFS, explore common alternatives, and figure out what would work for their system | ||
| * Document and highlight best practices for dealing with the common hurdles in moving to immutable-first: e.g. “save packages in a merkle tree; build a package index merkle tree; use IPNS to handle updates of the tip of package index tree”. | ||
| * M (P1): There is a way to prioritize different peers to optimize network utilization | ||
| * Bitswap peer prioritization supports pinning local / responsive peers | ||
|
|
||
|
|
||
| ### 🗂 Large Files | ||
| Go-IPFS is fast, efficient and stable when handling and transferring large amounts of data. (“large files” = 10GB - 1TB, “many small files” = 10K-1M 1MB files). The goals in this section are exclusive of finding the data (DHT/providers etc.). This is just the speed of transferring data when already connected to IPFS nodes that have that data. | ||
| * M (P0): Awesome go-ipfs benchmark test suite exists comparing ipfs performance and transfer size relative to bittorrent, rsync, http, cp, dd | ||
| * M (P0): Go-IPFS bitswap (or equivalent) can transfer large files from many peers at 0.8 times the speed of BitTorrent. | ||
| * Bitswap session improvements that improve transfer efficiency by reducing duplicate blocks and efficiently fetching related data from many peers simultaneously. | ||
| * Graphsync selector implementation that can understand non-overlapping subsets of data to select from multiple hosts and the driver that can build up graphsync requests for many peers to maximize efficiency and throughput. | ||
| * M (P0): Go-IPFS can transfer many small files at 0.8 times the speed of rsync (for single peers) or BitTorrent (for many peers) | ||
| * Bitswap session improvements | ||
| * Graph Exchange that can route requests efficiently | ||
| * Graphsync | ||
| * Flexible provider strategies | ||
| * M (P1): Go-IPFS can transfer large files from a single peer at 0.8 times the speed of HTTP. | ||
| * Flexible provider strategies to use the DHT efficiently (and therefore not slow IPFS down while adding/fetching files) while still preserving the reachability of data. | ||
| * M (P1): Go-IPFS can add a directory of many small files at 50% the speed of a simple local hard drive copy (or dd) (10k x 1MB files) | ||
| * Flexible provider strategies to limit the number of DHT provider advertisements we make | ||
| * Faster datastore (Badger or equivalent) that doesn’t slow down as the number of blocks stored grows large | ||
| * GC improvements (transactions, speed) | ||
| * M (P2): Data transferred over the network is no more than 1.2x the size of the data being transferred | ||
| * Bitswap session improvements to reduce duplicates | ||
| * Graphsync | ||
|
|
||
|
|
||
| ### 🔄 Decentralized Web | ||
| go-ipfs is the go-to backend for building DApps. | ||
| * M (P0): Provider lookups are efficient (<1 sec) in real-world network conditions (ex between physically distant peers) | ||
| * * M (P0): ipfs://, ipns:// work in web browsers | ||
| * Base32 CIDs by default | ||
| * Base32 IPNS keys (or make IPNS keys CIDs) | ||
| * M (PX): <1 sec IPNS resolution for any IPNS record | ||
| * QUIC | ||
| * NAT detection/traversal | ||
| * Only "reachable" nodes join the DHT | ||
| * M (P1): IPFS provides a clean way to persist data (dapps need to store stuff) | ||
| * M (PX): IPFS supports basic dapp requirements for security and private content | ||
| * dapps are secure (api security, multi tenancy). Apps can't read each other's state | ||
| * There are examples and documentation for ipfs users to create private-content dapps using IPFS (aka handling encryption on the client side) | ||
| * M (PX): IPFS is usable for building wordpress-level decentralized sites | ||
| * API doesn't suck (and is fast) | ||
| * We can build an interactive blogging engine with a built-in editor/publisher entirely in a webapp (backed by a go-ipfs instance). And it's *easy* to do this. | ||
|
|
||
| ### 🤝 IPFS Contributors and Developers | ||
| Developers using IPFS can rely on go-ipfs as a platform on which to build their product, app, or ecosystem, and new contributors are well supported | ||
| * M (P0): (all) Go-IPFS is approachable as a new contributor | ||
| * Go-IPFS internals are well documented | ||
| * Technical debt is paid off, all "indefinitely-in-progress" refactors are completed. | ||
| * Cleanup/rethink abstraction layers with 20-20 hindsight. | ||
| * Re-structure repos into reusable but not fragmented components. | ||
| * M (P0): (all) Go-IPFS is usable without gx | ||
| * Fix the versions (make them all sub-0) | ||
| * Add go.mod files to every package | ||
| * Merge #5435. | ||
| * M (P0): (all) Users can reliably transfer data between any two nodes. | ||
| * Reliable NAT traversal (AutoNAT, relay, TURN, etc). | ||
| * Connection manager doesn't kill useful connections. | ||
| * Scalable content routing | ||
| * Reliable DHT: unreliable nodes don't join the DHT. | ||
| * M (P0): (dapps) ipfs:// and ipns:// work in web browsers | ||
| * Base32 CIDs | ||
| * Base32 IPNS or IPNS uses CIDs | ||
| * M (P1): (all) Go-IPFS is approachable as a user/app developer | ||
| * User documentation. | ||
| * Well designed API interface | ||
| * Well designed API transport | ||
| * M (P1): (Dapps) Fast (< 3s), mutable name resolution (IPNS) | ||
| * Reliable DHT. | ||
| * QUIC (for fast connection establishment). | ||
| * Better protocol negotiation (multistream-2.0) | ||
| * Delegated Routing | ||
| * M (P1): (Dapps) IPFS Realtime Story is as good as using a centralized service (e.g. Socket.io, Pusher, etc) | ||
| * The API for doing secure (authenticated) broadcast updates exists | ||
| * M (P2): Go-ipfs can handle large datasets (>1TiB, >1M nodes) | ||
| * Scalable content routing (providing) | ||
| * DagSync (or at least better bitswap) | ||
| * Reliable, performant datastore | ||
| * M (P2): (all, Dapps) IPFS is pluggable and extensible (9/10 users don't need custom features) | ||
| * The go-ipfs plugins system is expanded to support new datastores, exchanges, etc | ||
| * Exchanges support multiple protocols (that diagram from hack week) | ||
| * Multi-DHT (https://github.com/ipfs/notes/issues/291#issuecomment-414495124) | ||
| * UnixFS-V2 is implemented | ||
| * M (P2): (machine-learning/package managers) IPFS can handle (and transfer) large (>1M entries) sharded indexes (objects, directories) | ||
| * Bitswap Improvements (sessions, prediction, etc) | ||
| * UnixFS-V2 (better directory structure) | ||
| * M (P3): (Dapps) IPFS can locally share data without a shared network | ||
| * A bluetooth (or like) transport | ||
| * M (P4): (anti-censorship) Our IPFS implementations are secure and don't leak sensitive information | ||
| * Get a security audit | ||
| * Implement the privacy preserving DHT (https://github.com/ipfs/notes/issues/291#issuecomment-396003860) | ||
| * Add a download-only mode (avoid serving local data) | ||
| * Investigate a privacy-preserving transport | ||
|
|
||
| ### 🧠 Strategic goals | ||
| * M (P0): We have a stable core that people can add things to without breaking changes | ||
| * UnixFS is stable and extensible | ||
| * UnixFS v2 (with a good HAMT) | ||
| * The DHT is extensible | ||
| * Multi-DHT: https://github.com/ipfs/notes/issues/291#issuecomment-414495124 | ||
| * M (P0): Have a solution for testing/benchmarking | ||
| * Available for offline testing. | ||
| * Reproducibility (across different platforms). | ||
| * Provide realistic Internet/WAN simulations (with delays and dropped packets). | ||
| * M (P0): Support allow- and deny-lists for content | ||
| * Required for exposing data by libraries and archives (allowlist) | ||
| * Required for ease of Copyright compliance (denylist) | ||
| * M (P1): Support read only gateways (not-offline) | ||
| * Might be more fitting for partners | ||
| * M (P1): Interfaces are future-proof enough for 1.0 | ||
| * CoreAPI | ||
| * Plugins | ||
| * M (P1): Grow our own team processes / active contributors to add more project coordination support | ||
| * M (P2): Investigate IPNS that would continue working offline/in small network segment (for future goals) | ||
| * M (P2): Certificate based private networks | ||
|
|
||
| ## ⏳ Timeline | ||
|
|
||
| - Q1 | ||
| - Q2 | ||
| - Q3 | ||
| - Q4 | ||
|
|
||
|
|
||
| ## ⁉️ Want to get involved? | ||
| - File an issue here with questions, or make a PR to suggest an alternate path or addition | ||
| - Want to help make this happen? [Open an issue on the Go IPFS Working Group repo!](https://github.com/ipfs/go-ipfs/issues) |