Feat/module docs #83

README.md

@@ -3,40 +3,36 @@
 [![CircleCI](https://circleci.com/gh/filecoin-project/go-fil-markets.svg?style=svg)](https://circleci.com/gh/filecoin-project/go-fil-markets)
 [![codecov](https://codecov.io/gh/filecoin-project/go-fil-markets/branch/master/graph/badge.svg)](https://codecov.io/gh/filecoin-project/go-fil-markets)
 
-This repository contains modular implementations of the [storage and retrieval market subsystems](https://filecoin-project.github.io/specs/#systems__filecoin_markets) of Filecoin. They are guided by the [v1.0 and 1.1 Filecoin specification updates](https://filecoin-project.github.io/specs/#intro__changelog). 
+This repository contains modular implementations of the [storage and retrieval market subsystems](https://filecoin-project.github.io/specs/#systems__filecoin_markets) of Filecoin. 
+They are guided by the [v1.0 and 1.1 Filecoin specification updates](https://filecoin-project.github.io/specs/#intro__changelog). 
 
 Separating implementations into a blockchain component and one or more mining and market components presents an opportunity to encourage implementation diversity while reusing non-security-critical components.
 
 ## Components
 
-* **[filestore](./filestore)**: a submodule branch that is a side effect of using the ffi to generate commP
-* **[pieceio](./pieceio)**: utilities that take IPLD graphs and turn them into pieces
-* **[piecestore](/.piecestore)**:  
-* **[storage market](./storagemarket)**: for finding, negotiating, and consummating deals to store data between clients and providers (storage miners)
-* **[retrieval market](./retrievalmarket)**: for finding, negotiating, and consummating deals to retrieve data between clients and providers (retrieval miners)
+* **[storagemarket](./storagemarket)**: for finding, negotiating, and consummating deals to
+ store data between clients and providers (storage miners).
+* **[retrievalmarket](./retrievalmarket)**: for finding, negotiating, and consummating deals to
+ retrieve data between clients and providers (retrieval miners).
+* **[filestore](./filestore)**: a wrapper around os.File for use by pieceio, storagemarket, and retrievalmarket.
+* **[pieceio](./pieceio)**: utilities that take IPLD graphs and turn them into pieces. Used by storagemarket.
+* **[piecestore](./piecestore)**:  a database for storing deal-related PieceInfo and CIDInfo. 
+Used by storagemarket and retrievalmarket.
 
 Related components in other repos:
-* **[data transfer](https://github.com/filecoin-project/go-data-transfer)**: for exchanging piece data between clients and miners, used by storage & retrieval market modules.
+* **[go-data-transfer](https://github.com/filecoin-project/go-data-transfer)**: for exchanging piece data between clients and miners, used by storage & retrieval market modules.
 
 ### Background reading
-* The [Markets in Filecoin](https://filecoin-project.github.io/specs/#systems__filecoin_markets) section of the Filecoin Specification contains the canonical spec
-* The [Storage Market Module design doc](https://docs.google.com/document/d/1FfMUpW8vanR9FrXsybxBBbba7DzeyuCIN2uAXgE7J8U) is a more specific overview of these component implementations
+* The [Markets in Filecoin](https://filecoin-project.github.io/specs/#systems__filecoin_markets) 
+section of the Filecoin Specification contains the canonical spec.
 
-## Usage
-
-**Requires go 1.13**
-
-Install the go-fil-markets module(s) that you want to use.  Available modules are:
-* `filestore`
-* `pieceio`
-* `piecestore`
-* `retrievalmarket`
-* `storagemarket`
+## Installation
+```bash
+go get "github.com/filecoin-project/go-fil-markets/<MODULENAME>"`
+```
 
-Install with:
-`go get "github.com/filecoin-project/go-fil-markets/<MODULENAME>"`
-
-TODO: usage for each module (maybe in subdirectories)
+## Usage
+Documentation is in the README for each module, listed in [Components](#Components).
 
 ## Contributing
 Issues and PRs are welcome! Please first read the [background reading](#background-reading) and [CONTRIBUTING](.go-fil-markets/CONTRIBUTING.md) guide, and look over the current code. PRs against master require approval of at least two maintainers. 

filestore/README.md

@@ -0,0 +1,29 @@
+# filestore
+
+The `filestore` module is a simple wrapper for os.File. It is used by [pieceio](../pieceio),
+[retrievialmarket](../retrievalmarket), and [storagemarket](../storagemarket).
+
+## Installation
+```bash
+go get github.com/filecoin-project/go-fil-markets/filestore
+```
+
+## FileStore
+FileStore is the primary export of this module.
+
+### Usage
+To create a new local filestore mounted on a given local directory path, use:
+```go
+package filestore
+
+func NewLocalFileStore(basedirectory OsPath) (FileStore, error) 
+```
+
+A FileStore provides the following functions:
+* [`Open`](filestore.go)
+* [`Create`](filestore.go)
+* [`Store`](filestore.go)
+* [`Delete`](filestore.go)
+* [`CreateTemp`](filestore.go)
+
+Please the [tests](filestore_test.go) for more information about expected behavior.

pieceio/README.md

@@ -0,0 +1,54 @@
+# pieceio
+
+The pieceio module is a collection of structs for generating piece commitments (a.k.a. CommP) and 
+storing pieces for storage market deals. It is used by the 
+[`storagemarket`](../storagemarket) module.
+
+## Installation
+```bash
+go get github.com/filecoin-project/go-fil-markets/pieceio
+```
+
+## PieceIO
+`PieceIO` is used by [`storagemarket`](../storagemarket) client for proposing deals. 
+
+**To initialize a PieceIO:**
+```go
+package pieceio
+
+func NewPieceIO(carIO CarIO, bs blockstore.Blockstore) PieceIO
+```
+**Parameters**
+* `carIO` is a [CarIO](#CarIO) from this module
+* `bs` is an IPFS blockstore for storing and retrieving data for deals. See
+ [github.com/ipfs/go-ipfs-blockstore](github.com/ipfs/go-ipfs-blockstore).
+
+## PieceIOWithStore
+`PieceIOWithStore` is `PieceIO` with a [`filestore`](../filestore). It is used by 
+[`storagemarket`](../storagemarket) provider to store pieces, and to generate and store piece commitments
+ and piece metadata for deals. 
+
+**To initialize a PieceIOWithStore:**
+
+```go
+package pieceio
+
+func NewPieceIOWithStore(carIO CarIO, store filestore.FileStore, bs blockstore.Blockstore) PieceIOWithStore
+```
+**Parameters**
+* `carIO` is a [CarIO](#CarIO) from this module
+* `store` is a [FileStore](../filestore) from this go-fil-markets repo.
+* `bs` is an IPFS blockstore for storing and retrieving data for deals. See
+ [github.com/ipfs/go-ipfs-blockstore](github.com/ipfs/go-ipfs-blockstore).
+
+## CarIO
+CarIO is a utility module that wraps [github.com/ipld/go-car](https://github.com/ipld/go-car) for use by storagemarket.
+
+**To initialize a CarIO:**
+```go
+package cario
+
+func NewCarIO() pieceio.CarIO
+```
+
+Please the [tests](pieceio_test.go) for more information about expected behavior.

piecestore/README.md

@@ -0,0 +1,48 @@
+# piecestore
+
+The `piecestore` module is a simple encapsulation of two data stores, one for `PieceInfo` and
+ another for `CIDInfo`.  The piecestore's main goal is to help 
+ [storagemarket module](../storagemarket) and [retrievalmarket module](../retrievalmarket)
+ find where sealed data lives inside of sectors. Storage market writes the
+ data, and retrieval market reads it.
+
+Both markets use `CIDInfo` to look up a Piece that contains the payload, and then
+ use `PieceInfo` to find the sector that contains the piece.
+
+The storage market has to write this data before it completes the deal in order to later
+ look up the payload when the data is served.
+
+## Installation
+```bash
+go get github.com/filecoin-project/go-fil-markets/piecestore
+```
+
+### PieceStore
+`PieceStore` is primary export of this module. It is a database 
+of piece info that can be modified and queried. The PieceStore 
+interface is implemented in [piecestore.go](./piecestore.go).
+
+It has two stores, one for `PieceInfo` keyed by `pieceCID`, and another for 
+`CIDInfo`, keyed by `payloadCID`. These keys are of type `cid.CID`; see 
+[github.com/ipfs/go-cid](https://github.com/ipfs/go-cid).
+
+**To initialize a PieceStore**
+```go
+func NewPieceStore(ds datastore.Batching) PieceStore
+```
+
+**Parameters**
+* `ds datastore.Batching` is a datastore for the deal's state machine. It is
+ typically the node's own datastore that implements the IPFS datastore.Batching interface.
+ See
+  [github.com/ipfs/go-datastore](https://github.com/ipfs/go-datastore).
+
+
+`PieceStore` implements the following functions:
+
+* [`AddDealForPiece`](./piecestore.go)
+* [`AddPieceBlockLocations`](./piecestore.go)
+* [`GetPieceInfo`](./piecestore.go)
+* [`GetCIDInfo`](./piecestore.go)
+
+Please the [tests](piecestore_test.go) for more information about expected behavior.

piecestore/piecestore.go

@@ -26,6 +26,7 @@ type pieceStore struct {
 	cidInfos *statestore.StateStore
 }
 
+// Store `dealInfo` in the PieceStore with key `pieceCID`.
 func (ps *pieceStore) AddDealForPiece(pieceCID cid.Cid, dealInfo DealInfo) error {
 	return ps.mutatePieceInfo(pieceCID, func(pi *PieceInfo) error {
 		for _, di := range pi.Deals {
@@ -38,6 +39,7 @@ func (ps *pieceStore) AddDealForPiece(pieceCID cid.Cid, dealInfo DealInfo) error
 	})
 }
 
+// Store the map of blockLocations in the PieceStore's CIDInfo store, with key `pieceCID`
 func (ps *pieceStore) AddPieceBlockLocations(pieceCID cid.Cid, blockLocations map[cid.Cid]BlockLocation) error {
 	for c, blockLocation := range blockLocations {
 		err := ps.mutateCIDInfo(c, func(ci *CIDInfo) error {
@@ -56,6 +58,7 @@ func (ps *pieceStore) AddPieceBlockLocations(pieceCID cid.Cid, blockLocations ma
 	return nil
 }
 
+// Retrieve the PieceInfo associated with `pieceCID` from the piece info store.
 func (ps *pieceStore) GetPieceInfo(pieceCID cid.Cid) (PieceInfo, error) {
 	var out PieceInfo
 	if err := ps.pieces.Get(pieceCID).Get(&out); err != nil {
@@ -64,6 +67,7 @@ func (ps *pieceStore) GetPieceInfo(pieceCID cid.Cid) (PieceInfo, error) {
 	return out, nil
 }
 
+// Retrieve the CIDInfo associated with `pieceCID` from the CID info store.
 func (ps *pieceStore) GetCIDInfo(payloadCID cid.Cid) (CIDInfo, error) {
 	var out CIDInfo
 	if err := ps.cidInfos.Get(payloadCID).Get(&out); err != nil {

piecestore/types.go

@@ -7,7 +7,7 @@ import (
 
 //go:generate cbor-gen-for PieceInfo DealInfo BlockLocation PieceBlockLocation CIDInfo
 
-// DealInfo is information about a single deal for a give piece
+// DealInfo is information about a single deal for a given piece
 type DealInfo struct {
 	DealID   abi.DealID
 	SectorID uint64