cleanup: miner: remove markets and deal-making from Lotus Miner

[aarshkshah1992]

Related Issues

Closes #12004

Proposed Changes

The use of legacy markets was deprecated from Lotus Miner a long time ago in favour of Boost. This PR removes all legacy markets related code from the Lotus Miner. Lotus no longer depends on go-graphsync, go-data-transfer, go-fil-markets, dagstore and index-provider .

All Markets related functionality has been removed from the Lotus Miner CLI and the API. Specifically, the following Market APIs have been removed from a Lotus Miner API as part of this PR:

         MarketImportDealData(ctx context.Context, propcid cid.Cid, path string) error //perm:write
	// MarketListRetrievalDeals is deprecated, returns empty list
	MarketListRetrievalDeals(ctx context.Context) ([]struct{}, error)                                                                                                                    //perm:read
	MarketGetDealUpdates(ctx context.Context) (<-chan storagemarket.MinerDeal, error)                                                                                                    //perm:read
	MarketListIncompleteDeals(ctx context.Context) ([]storagemarket.MinerDeal, error)                                                                                                    //perm:read
	MarketSetAsk(ctx context.Context, price types.BigInt, verifiedPrice types.BigInt, duration abi.ChainEpoch, minPieceSize abi.PaddedPieceSize, maxPieceSize abi.PaddedPieceSize) error //perm:admin
	MarketGetAsk(ctx context.Context) (*storagemarket.SignedStorageAsk, error)                                                                                                           //perm:read
	MarketSetRetrievalAsk(ctx context.Context, rask *retrievalmarket.Ask) error                                                                                                          //perm:admin
	MarketGetRetrievalAsk(ctx context.Context) (*retrievalmarket.Ask, error)                                                                                                             //perm:read
	MarketListDataTransfers(ctx context.Context) ([]DataTransferChannel, error)                                                                                                          //perm:write
	MarketDataTransferUpdates(ctx context.Context) (<-chan DataTransferChannel, error)                                                                                                   //perm:write
	// MarketDataTransferDiagnostics generates debugging information about current data transfers over graphsync
	MarketDataTransferDiagnostics(ctx context.Context, p peer.ID) (*TransferDiagnostics, error) //perm:write
	// MarketRestartDataTransfer attempts to restart a data transfer with the given transfer ID and other peer
	MarketRestartDataTransfer(ctx context.Context, transferID datatransfer.TransferID, otherPeer peer.ID, isInitiator bool) error //perm:write
	// MarketCancelDataTransfer cancels a data transfer with the given transfer ID and other peer
	MarketCancelDataTransfer(ctx context.Context, transferID datatransfer.TransferID, otherPeer peer.ID, isInitiator bool) error //perm:write
	MarketPendingDeals(ctx context.Context) (PendingDealInfo, error)                                                             //perm:write
	MarketPublishPendingDeals(ctx context.Context) error                                                                         //perm:admin
	MarketRetryPublishDeal(ctx context.Context, propcid cid.Cid) error                                                           //perm:admin

	// DagstoreListShards returns information about all shards known to the
	// DAG store. Only available on nodes running the markets subsystem.
	DagstoreListShards(ctx context.Context) ([]DagstoreShardInfo, error) //perm:read

	// DagstoreInitializeShard initializes an uninitialized shard.
	//
	// Initialization consists of fetching the shard's data (deal payload) from
	// the storage subsystem, generating an index, and persisting the index
	// to facilitate later retrievals, and/or to publish to external sources.
	//
	// This operation is intended to complement the initial migration. The
	// migration registers a shard for every unique piece CID, with lazy
	// initialization. Thus, shards are not initialized immediately to avoid
	// IO activity competing with proving. Instead, shard are initialized
	// when first accessed. This method forces the initialization of a shard by
	// accessing it and immediately releasing it. This is useful to warm up the
	// cache to facilitate subsequent retrievals, and to generate the indexes
	// to publish them externally.
	//
	// This operation fails if the shard is not in ShardStateNew state.
	// It blocks until initialization finishes.
	DagstoreInitializeShard(ctx context.Context, key string) error //perm:write

	// DagstoreRecoverShard attempts to recover a failed shard.
	//
	// This operation fails if the shard is not in ShardStateErrored state.
	// It blocks until recovery finishes. If recovery failed, it returns the
	// error.
	DagstoreRecoverShard(ctx context.Context, key string) error //perm:write

	// DagstoreInitializeAll initializes all uninitialized shards in bulk,
	// according to the policy passed in the parameters.
	//
	// It is recommended to set a maximum concurrency to avoid extreme
	// IO pressure if the storage subsystem has a large amount of deals.
	//
	// It returns a stream of events to report progress.
	DagstoreInitializeAll(ctx context.Context, params DagstoreInitializeAllParams) (<-chan DagstoreInitializeAllEvent, error) //perm:write

	// DagstoreGC runs garbage collection on the DAG store.
	DagstoreGC(ctx context.Context) ([]DagstoreShardResult, error) //perm:admin

	// DagstoreRegisterShard registers a shard manually with dagstore with given pieceCID
	DagstoreRegisterShard(ctx context.Context, key string) error //perm:admin

	// IndexerAnnounceDeal informs indexer nodes that a new deal was received,
	// so they can download its index
	IndexerAnnounceDeal(ctx context.Context, proposalCid cid.Cid) error //perm:admin

	// IndexerAnnounceAllDeals informs the indexer nodes aboutall active deals.
	IndexerAnnounceAllDeals(ctx context.Context) error //perm:admin

	// DagstoreLookupPieces returns information about shards that contain the given CID.
	DagstoreLookupPieces(ctx context.Context, cid cid.Cid) ([]DagstoreShardInfo, error) //perm:admin
	
	DealsImportData(ctx context.Context, dealPropCid cid.Cid, file string) error //perm:admin
	DealsConsiderOnlineStorageDeals(context.Context) (bool, error)               //perm:admin
	DealsSetConsiderOnlineStorageDeals(context.Context, bool) error              //perm:admin
	DealsConsiderOnlineRetrievalDeals(context.Context) (bool, error)             //perm:admin
	DealsSetConsiderOnlineRetrievalDeals(context.Context, bool) error            //perm:admin
	DealsPieceCidBlocklist(context.Context) ([]cid.Cid, error)                   //perm:admin
	DealsSetPieceCidBlocklist(context.Context, []cid.Cid) error                  //perm:admin
	DealsConsiderOfflineStorageDeals(context.Context) (bool, error)              //perm:admin
	DealsSetConsiderOfflineStorageDeals(context.Context, bool) error             //perm:admin
	DealsConsiderOfflineRetrievalDeals(context.Context) (bool, error)            //perm:admin
	DealsSetConsiderOfflineRetrievalDeals(context.Context, bool) error           //perm:admin
	DealsConsiderVerifiedStorageDeals(context.Context) (bool, error)             //perm:admin
	DealsSetConsiderVerifiedStorageDeals(context.Context, bool) error            //perm:admin
	DealsConsiderUnverifiedStorageDeals(context.Context) (bool, error)           //perm:admin
	DealsSetConsiderUnverifiedStorageDeals(context.Context, bool) error          //perm:admin

	PiecesListPieces(ctx context.Context) ([]cid.Cid, error)                                 //perm:read
	PiecesListCidInfos(ctx context.Context) ([]cid.Cid, error)                               //perm:read
	PiecesGetPieceInfo(ctx context.Context, pieceCid cid.Cid) (*piecestore.PieceInfo, error) //perm:read
	PiecesGetCIDInfo(ctx context.Context, payloadCid cid.Cid) (*piecestore.CIDInfo, error)   //perm:read

However, the following APIs have still been kept around to enable users to interact with the built-in Market Actor.

        // StateMarketBalance looks up the Escrow and Locked balances of the given address in the Storage Market
	StateMarketBalance(context.Context, address.Address, types.TipSetKey) (MarketBalance, error) //perm:read
	// StateMarketParticipants returns the Escrow and Locked balances of every participant in the Storage Market
	StateMarketParticipants(context.Context, types.TipSetKey) (map[string]MarketBalance, error) //perm:read
	// StateMarketDeals returns information about every deal in the Storage Market
	StateMarketDeals(context.Context, types.TipSetKey) (map[string]*MarketDeal, error) //perm:read
	// StateMarketStorageDeal returns information about the indicated deal
	StateMarketStorageDeal(context.Context, abi.DealID, types.TipSetKey) (*MarketDeal, error) //perm:read
	
        // MarketListDeals enlists all deals in the storage market actor
	MarketListDeals(ctx context.Context) ([]*MarketDeal, error) //perm:read
        DealsList(ctx context.Context) ([]*MarketDeal, error) //perm:admin
	
        // MarketAddBalance adds funds to the market actor
	MarketAddBalance(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error) //perm:sign
	// MarketGetReserved gets the amount of funds that are currently reserved for the address
	MarketGetReserved(ctx context.Context, addr address.Address) (types.BigInt, error) //perm:sign
	// MarketReserveFunds reserves funds for a deal
	MarketReserveFunds(ctx context.Context, wallet address.Address, addr address.Address, amt types.BigInt) (cid.Cid, error) //perm:sign
	// MarketReleaseFunds releases funds reserved by MarketReserveFunds
	MarketReleaseFunds(ctx context.Context, addr address.Address, amt types.BigInt) error //perm:sign
	// MarketWithdraw withdraws unlocked funds from the market actor
	MarketWithdraw(ctx context.Context, wallet, addr address.Address, amt types.BigInt) (cid.Cid, error) //perm:sign

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green

[github-actions]

All checks have passed

[ZenGround0]

You can (and imo should) remove the graphsync module from the fullnode with markets gone. There are comments saying it's used by other subsystems but it's just markets afaik.

[aarshkshah1992]

@ZenGround0 Yup. You will see it all in the final draft once it is ready for review :)

[aarshkshah1992]

Tested this on a local devnet node and things look good. Was able to seal and activate sectors (submit WindowPosts for them)

aarshshah@Aarshs-MacBook-Pro lotus % lotus-miner sectors list
ID  State    OnChain  Active  Expiration                    Deals  
0   Proving  YES      YES     3675537 (in 24 weeks 2 days)  CC     
1   Proving  YES      YES     3675537 (in 24 weeks 2 days)  CC     
2   Proving  YES      YES     1549540 (in 10 weeks 1 day)   CC     
3   Proving  YES      YES     1549548 (in 10 weeks 1 day)   CC

aarshshah@Aarshs-MacBook-Pro lotus % lotus-miner proving check 0
deadline  partition  sector  status
0         0          0       good
0         0          1       good
aarshshah@Aarshs-MacBook-Pro lotus % lotus-miner proving check 1
deadline  partition  sector  status
1         0          2       good
1         0          3       good

aarshshah@Aarshs-MacBook-Pro lotus % lotus-miner proving deadlines
Miner: t01000
deadline  open      partitions  sectors (faults)  proven partitions
0         17:03:03  1           2 (0)             0
1         17:07:03  1           2 (0)             0
2         17:11:03  0           0 (0)             0
3         17:15:03  0           0 (0)             0  (current)
4         17:19:03  0           0 (0)             0
5         17:23:03  0           0 (0)             0
6         17:27:03  0           0 (0)             0
7         17:31:03  0           0 (0)             0
8         17:35:03  0           0 (0)             0
9         17:39:03  0           0 (0)             0
10        17:43:03  0           0 (0)             0
11        17:47:03  0           0 (0)             0
12        17:51:03  0           0 (0)             0
13        17:55:03  0           0 (0)             0
14        17:59:03  0           0 (0)             0
15        18:03:03  0           0 (0)             0
16        18:07:03  0           0 (0)             0
17        18:11:03  0           0 (0)             0
18        18:15:03  0           0 (0)             0
19        18:19:03  0           0 (0)             0
20        18:23:03  0           0 (0)             0
21        18:27:03  0           0 (0)             0
22        18:31:03  0           0 (0)             0
23        18:35:03  0           0 (0)             0
24        18:39:03  0           0 (0)             0
25        18:43:03  0           0 (0)             0
26        18:47:03  0           0 (0)             0
27        18:51:03  0           0 (0)             0
28        18:55:03  0           0 (0)             0
29        18:59:03  0           0 (0)             0
30        19:03:03  0           0 (0)             0
31        19:07:03  0           0 (0)             0
32        19:11:03  0           0 (0)             0
33        19:15:03  0           0 (0)             0
34        19:19:03  0           0 (0)             0
35        19:23:03  0           0 (0)             0
36        19:27:03  0           0 (0)             0
37        19:31:03  0           0 (0)             0
38        19:35:03  0           0 (0)             0
39        19:39:03  0           0 (0)             0
40        19:43:03  0           0 (0)             0
41        19:47:03  0           0 (0)             0
42        19:51:03  0           0 (0)             0
43        19:55:03  0           0 (0)             0
44        19:59:03  0           0 (0)             0
45        20:03:03  0           0 (0)             0
46        20:07:03  0           0 (0)             0
47        20:11:03  0           0 (0)             0

[aarshkshah1992]

The CI test failure is a flaky.

[rvagg]

love it