Feat/module docs #83

README.md

@@ -9,34 +9,32 @@ Separating implementations into a blockchain component and one or more mining an
 
 ## Components
 
-* **[filestore](./filestore)**: a submodule branch that is a side effect of using the ffi to generate commP
-* **[pieceio](./pieceio)**: utilities that take IPLD graphs and turn them into pieces
-* **[piecestore](/.piecestore)**:  
-* **[storage market](./storagemarket)**: for finding, negotiating, and consummating deals to store data between clients and providers (storage miners)
-* **[retrieval market](./retrievalmarket)**: for finding, negotiating, and consummating deals to retrieve data between clients and providers (retrieval miners)
+* **[filestore](./filestore)**: a submodule branch that is a side effect of using the ffi to
+ generate commP.
+* **[pieceio](./pieceio)**: utilities that take IPLD graphs and turn them into pieces.
+* **[piecestore](./piecestore)**:  a database for storing deal-related PieceInfo and CIDInfo. 
+* **[storagemarket](./storagemarket)**: for finding, negotiating, and consummating deals to
+ store data between clients and providers (storage miners).
+* **[retrievalmarket](./retrievalmarket)**: for finding, negotiating, and consummating deals to
+ retrieve data between clients and providers (retrieval miners).
 
 Related components in other repos:
 * **[data transfer](https://github.com/filecoin-project/go-data-transfer)**: for exchanging piece data between clients and miners, used by storage & retrieval market modules.
 
 ### Background reading
 * The [Markets in Filecoin](https://filecoin-project.github.io/specs/#systems__filecoin_markets) section of the Filecoin Specification contains the canonical spec
-* The [Storage Market Module design doc](https://docs.google.com/document/d/1FfMUpW8vanR9FrXsybxBBbba7DzeyuCIN2uAXgE7J8U) is a more specific overview of these component implementations
-
-## Usage
-
-**Requires go 1.13**
-
-Install the go-fil-markets module(s) that you want to use.  Available modules are:
-* `filestore`
-* `pieceio`
-* `piecestore`
-* `retrievalmarket`
-* `storagemarket`
+* The [Storage Market Module design doc](https://docs.google.com/document/d
+/1FfMUpW8vanR9FrXsybxBBbba7DzeyuCIN2uAXgE7J8U) is a more specific overview of the storage market
+ component implementations
+* The 
+[Retrieval Market Module design doc](https://docs.google.com/document/d/1SyUDXzbGwYwoKMUWwE9_8IIjHshecLo_k7PdKQ0WK9g/edit#heading=h.uq51khvyisgr) 
+is a more specific overview of the retrieval market component implementations
 
 Install with:
 `go get "github.com/filecoin-project/go-fil-markets/<MODULENAME>"`
 
-TODO: usage for each module (maybe in subdirectories)
+## Usage
+Documentation linked in each listed module in [Components](#Components).
 
 ## Contributing
 Issues and PRs are welcome! Please first read the [background reading](#background-reading) and [CONTRIBUTING](.go-fil-markets/CONTRIBUTING.md) guide, and look over the current code. PRs against master require approval of at least two maintainers. 

filestore/README.md

@@ -0,0 +1,2 @@
+# How to use the filestore module
+

pieceio/README.md

@@ -0,0 +1 @@
+# How to use the pieceio module

piecestore/README.md

@@ -0,0 +1,75 @@
+# The piecestore module
+
+The piecestore module is  a simple encapsulation of two data stores, one for `PieceInfo` and
+ another for `CIDInfo`.  It is currently used by the [storagemarket module](../storagemarket) for
+  storing information needed for storage deals.
+
+## Installation
+```bash
+go get github.com/filecoin-project/go-fil-markets/piecestore
+```
+
+## Types
+
+* [`DealInfo`](./types.go) - information about a single deal for a given piece
+* [`BlockLocation`](./types.go) - information about where a given block is relative to the
+ overall
+ piece
+* [`PieceBlockLocation`](./types.go) - block information plus the pieceCID of the piece the
+ block is inside of.
+* [`CIDInfo`](./types.go) - where a given CID will live inside a piece
+* [`PieceInfo`](./types.go) - metadata about a piece a provider may be storing
+ based on
+ its PieceCID -- so that, given a pieceCID during retrieval, the miner can determine how to unseal it if needed
+* [`PieceStore`](#PieceStore) - a saved database of piece info that can be modified and
+ queried
+
+### PieceStore
+`PieceStore` is a saved database of piece info that can be modified 
+and queried.  The PieceStore interface is implemented in [piecestore.go](./piecestore.go).
+
+It has two stores, one for `PieceInfo` and another for `CIDInfo`, each keyed by the `pieceCID`,
+ which is a `cid.CID`.
+
+Please see the [tests](./piecestore_test.go) for more detail about how a `PieceStore` is 
+expected to behave. 
+
+#### To create a new PieceStore
+```go
+func NewPieceStore(ds datastore.Batching) PieceStore
+```
+
+`PieceStore` implements the following functions:
+
+* [`AddDealForPiece`](#AddDealForPiece)
+* [`AddPieceBlockLocations`](#AddPieceBlockLocations)
+* [`GetPieceInfo`](#GetPieceInfo)
+* [`GetCIDInfo`](#GetCIDInfo)
+
+#### AddDealForPiece
+```go
+func AddDealForPiece(pieceCID cid.Cid, dealInfo DealInfo) error
+```
+
+Store `dealInfo` in the PieceStore's piece info store, with key `pieceCID`.
+
+#### AddPieceBlockLocations
+```go
+func AddPieceBlockLocations(pieceCID cid.Cid, blockLocations map[cid.Cid]BlockLocation) error
+```
+
+Store the map of blockLocations in the PieceStore's CIDInfo store, with key `pieceCID`
+
+#### GetPieceInfo
+```go
+func GetPieceInfo(pieceCID cid.Cid) (PieceInfo, error)
+```
+
+Retrieve the PieceInfo associated with `pieceCID` from the piece info store.
+
+#### GetCIDInfo
+```go
+func GetCIDInfo(payloadCID cid.Cid) (CIDInfo, error)
+```
+
+Retrieve the CIDInfo associated with `pieceCID` from the CID info store.

piecestore/piecestore.go

@@ -26,6 +26,7 @@ type pieceStore struct {
 	cidInfos *statestore.StateStore
 }
 
+// Store `dealInfo` in the PieceStore with key `pieceCID`.
 func (ps *pieceStore) AddDealForPiece(pieceCID cid.Cid, dealInfo DealInfo) error {
 	return ps.mutatePieceInfo(pieceCID, func(pi *PieceInfo) error {
 		for _, di := range pi.Deals {

piecestore/types.go

@@ -7,7 +7,7 @@ import (
 
 //go:generate cbor-gen-for PieceInfo DealInfo BlockLocation PieceBlockLocation CIDInfo
 
-// DealInfo is information about a single deal for a give piece
+// DealInfo is information about a single deal for a given piece
 type DealInfo struct {
 	DealID   abi.DealID
 	SectorID uint64

retrievalmarket/README.md

@@ -0,0 +1,124 @@
+# How to use the RetrievalMarket module
+
+## Background reading
+Please see the 
+[Filecoin Retrieval Market Specification](https://filecoin-project.github.io/specs/#systems__filecoin_markets__retrieval_market).
+
+## For Implementers
+You will need to implement all of the required Client and Provider API functions in 
+[retrievalmarket/types.go](./types.go), described below:
+
+### PeerResolver
+PeerResolver is an interface for looking up providers that may have a piece.
+
+#### GetPeers
+```go
+func GetPeers(payloadCID cid.Cid) ([]RetrievalPeer, error)
+```
+Return a slice of RetrievalPeers that store data referenced by `payloadCID`.
+
+---
+### RetrievalClientNode
+
+`RetrievalClientNode` contains the node dependencies for a RetrievalClient.
+
+* [`AllocateLane`](#AllocateLane)
+* [`GetChainHead`](#GetChainHead)
+* [`GetOrCreatePaymentChannel`](#GetOrCreatePaymentChannel)
+* [`CreatePaymentVoucher`](#CreatePaymentVoucher)
+* [`WaitForPaymentChannelAddFunds`](#WaitForPaymentChannelAddFunds)
+* [`WaitForPaymentChannelCreation`](#WaitForPaymentChannelCreation)
+
+#### AllocateLane
+```go
+func AllocateLane(paymentChannel address.Address) (uint64, error)
+```
+
+Create a lane within `paymentChannel` so that calls to CreatePaymentVoucher will 
+automatically make vouchers only for the difference in total. Note that payment channel 
+Actors have a
+[lane limit](https://github.com/filecoin-project/specs-actors/blob/0df536f7e461599c818231aa0effcdaccbb74900/actors/builtin/paych/paych_actor.go#L20).
+
+#### CreatePaymentVoucher
+```go
+func CreatePaymentVoucher(ctx context.Context, paymentChannel address.Address, 
+                         amount abi.TokenAmount, lane uint64, tok shared.TipSetToken
+                         ) (*paych.SignedVoucher, error)
+```
+Create a new payment voucher for `paymentChannel` with `amount`, for lane `lane`, given chain
+state at `tok`.
+
+#### GetChainHead
+```go
+func GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error)
+```
+Get the current chain head. Return the head TipSetToken and abi.ChainEpoch for 
+which it is the Head.
+
+#### GetOrCreatePaymentChannel
+```go
+func GetOrCreatePaymentChannel(ctx context.Context, clientAddress, minerAddress address.Address, 
+                          amount abi.TokenAmount, tok shared.TipSetToken
+                         ) (address.Address, cid.Cid, error)
+```
+If there is a current payment channel for deals between `clientAddress` and `minerAddress`, 
+add `amount` to the channel, then return the payment channel address and `cid.Undef`.
+
+If there isn't, construct a new payment channel actor with `amount` funds by posting 
+the corresponding message on chain, then return `address.Undef` and the posted message `cid.Cid`.
+
+#### WaitForPaymentChannelAddFunds
+```go
+func WaitForPaymentChannelAddFunds(messageCID cid.Cid) error
+```
+Wait for message with CID `messageCID` on chain that funds have been sent to a payment channel.
+
+#### WaitForPaymentChannelCreation
+```go
+func WaitForPaymentChannelCreation(messageCID cid.Cid) (address.Address, error)
+```
+Wait for a message on chain with CID `messageCID` that a payment channel has been created.
+
+---
+### RetrievalProviderNode
+`RetrievalProviderNode` contains the node dependencies for a RetrievalProvider.
+
+* [`GetChainHead`](#GetChainHead)
+* [`GetMinerWorkerAddress`](#GetMinerWorkerAddress)
+* [`UnsealSector`](#UnsealSector)
+* [`SavePaymentVoucher`](#SavePaymentVoucher)
+
+#### GetChainHead
+```go
+func GetChainHead(ctx context.Context) (shared.TipSetToken, abi.ChainEpoch, error)
+```
+Get the current chain head. Return the head TipSetToken and its abi.ChainEpoch.
+
+#### GetMinerWorkerAddress
+```go
+func GetMinerWorkerAddress(ctx context.Context, addr address.Address, tok shared.TipSetToken,
+                     ) (address.Address, error)
+```
+Get the miner worker address for the given miner owner, as of `tok`.
+
+#### UnsealSector
+```go
+func UnsealSector(ctx context.Context, sectorID uint64, offset uint64, length uint64,
+             ) (io.ReadCloser, error)
+```
+Unseal `length` data contained in `sectorID`, starting at `offset`.  Return an `io.ReadCloser
+` for accessing the data.
+
+#### SavePaymentVoucher
+```go
+func SavePaymentVoucher(ctx context.Context, paymentChannel address.Address, 
+                   voucher *paych.SignedVoucher, proof []byte, expectedAmount abi.TokenAmount, 
+                   tok shared.TipSetToken) (abi.TokenAmount, error)
+```
+
+Save the provided `paych.SignedVoucher` for `paymentChannel`. The RetrievalProviderNode
+implementation should validate the SignedVoucher using the provided `proof`, `
+expectedAmount`, based on  the chain state referenced by `tok`.  The value of the
+voucher should be equal or greater than the largest previous voucher by 
+ `expectedAmount`. It returns the actual difference.
+

retrievalmarket/types.go

@@ -197,7 +197,8 @@ type RetrievalClientNode interface {
 
 	// GetOrCreatePaymentChannel sets up a new payment channel if one does not exist
 	// between a client and a miner and insures the client has the given amount of funds available in the channel
-	GetOrCreatePaymentChannel(ctx context.Context, clientAddress address.Address, minerAddress address.Address, clientFundsAvailable abi.TokenAmount, tok shared.TipSetToken) (address.Address, cid.Cid, error)
+	GetOrCreatePaymentChannel(ctx context.Context, clientAddress, minerAddress address.Address,
+		clientFundsAvailable abi.TokenAmount, tok shared.TipSetToken) (address.Address, cid.Cid, error)
 
 	// Allocate late creates a lane within a payment channel so that calls to
 	// CreatePaymentVoucher will automatically make vouchers only for the difference
@@ -207,7 +208,8 @@ type RetrievalClientNode interface {
 	// CreatePaymentVoucher creates a new payment voucher in the given lane for a
 	// given payment channel so that all the payment vouchers in the lane add up
 	// to the given amount (so the payment voucher will be for the difference)
-	CreatePaymentVoucher(ctx context.Context, paymentChannel address.Address, amount abi.TokenAmount, lane uint64, tok shared.TipSetToken) (*paych.SignedVoucher, error)
+	CreatePaymentVoucher(ctx context.Context, paymentChannel address.Address, amount abi.TokenAmount,
+		lane uint64, tok shared.TipSetToken) (*paych.SignedVoucher, error)
 
 	// WaitForPaymentChannelAddFunds waits for a message on chain that funds have
 	// been sent to a payment channel