Improve deal making efficiency & Lotus v1.1.3 update & fixes #705

jsign · 2020-11-12T13:26:53Z

This PR:

Update Lotus client & docker image to v1.1.3
Leverage a PR included in 1.1.3 I made in Lotus some time ago that allows removing the replace directive we had in go.mod; GO clients will be happier.
Leverage new APIs to only calculate once deal size and CommP. This makes Job execution load on the Lotus node be independent of the replication factor. That is if the replication factor was 5, this PR decreases the total cumulative load in the Job execution by 5x in Lotus.
The TotalActiveDeals is removed from the miner's index. Unfortunately, this was derived from an API that lists all the sectors of miners to do some extra calculation, but for big miners, this can have a huge output in the API call which isn't paged thus trigger OOMs. Total active deals were nice metrics, but honestly is very easy to fake anyway so not really convincing for decisions anyway. If we come up to think this was important later, we should research/escalate a more efficient way to derive this. Unblocks Consider making Powergate miner index data public textile#397.
Make the index raw json output bind-address be configurable, and bind to dynamic free ports to avoid collisions with other running things in the same GH runner.

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign · 2020-11-13T15:07:42Z

deals/module/deals.go

+				PieceCid:     &pieceCid,
+				PieceSize:    pieceSize.Unpadded(),


We delegate to the client to provide pieceSize and pieceCid. This allows Lotus to avoid calculating these values, and just trusting they're correct. This is important for clients such as FFS, which create multiple deals with the same data.

jsign · 2020-11-13T15:08:38Z

deals/module/deals.go

-	dsz, err := lapi.ClientDealSize(ctx, c)
+	dsz, err := lapi.ClientDealPieceCID(ctx, c)


We use a new API that calculates both size and PieceCID.

jsign · 2020-11-13T15:10:10Z

ffs/filcold/filcold.go

@@ -81,14 +81,14 @@ func (fc *FilCold) Fetch(ctx context.Context, pyCid cid.Cid, piCid *cid.Cid, wad
 	return ffs.FetchInfo{RetrievedMiner: miner, FundsSpent: fundsSpent}, nil
 }

-func (fc *FilCold) calculatePieceSize(ctx context.Context, c cid.Cid) (api.DataSize, error) {
+func (fc *FilCold) calculateDealPiece(ctx context.Context, c cid.Cid) (abi.PaddedPieceSize, cid.Cid, error) {


I preferred to avoid using api namespace here since is a Lotus package.
Also, I decided using abi.PaddedPieceSize since using uint64 is ambiguous regarding if the size includes padding or not. It's a technical detail that matters in using the right size type for calculating price. For other minor things, we can still use uint64. It's just a type-safety mesaure.

Saying it differently: abi.PaddedPieceSize always indicates padded data which is the real size it will take in the miner's sector.

jsign · 2020-11-13T15:12:04Z

ffs/integrationtest/scheduler/scheduler_test.go

-	require.Equal(t, "failed to start deal: cannot propose a deal whose piece size (4096) is greater than sector size (2048)", de.Message)
+	require.Equal(t, "data doesn't fit in a sector", de.Message)


Lotus API changed error string.

jsign · 2020-11-13T15:13:08Z

go.mod

@@ -60,5 +61,3 @@ require (
 )

 replace github.com/dgraph-io/badger/v2 => github.com/dgraph-io/badger/v2 v2.2007.2
-
-replace github.com/filecoin-project/filecoin-ffi => github.com/filecoin-project/filecoin-ffi v0.30.4-0.20200910194244-f640612a1a1f


jsign · 2020-11-13T15:13:24Z

reputation/reputation.go

-	score := 50*faultsScore + 20*powerScore + 20*externalScore + 10*askScore + 1000*float64(miner.ActiveDeals)
+	score := 50*faultsScore + 20*powerScore + 20*externalScore + 100*askScore


Still arbitrary.

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

asutula

Awesome!

jsign added enhancement New feature or request patch labels Nov 12, 2020

jsign self-assigned this Nov 12, 2020

jsign force-pushed the jsign/upandeff branch from 17c9101 to 6685316 Compare November 12, 2020 13:29

jsign changed the title ~~Improve deal making efficiency & Lotus v1.1.3 update~~ Improve deal making efficiency & Lotus v1.1.3 update & fixes Nov 12, 2020

jsign added 14 commits November 13, 2020 11:31

update things

2138e69

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

add missing deal

848b673

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix changed error message

2086230

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update devnet

df1acb7

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update devnet

fb6a70b

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

revert change

ecbce59

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

remove sector list from miner index

ad04787

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

avoid port collisions

bc520c0

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

configurable index raw json output bind addr

56f25c4

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

use random free port

1789fff

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

fix

73ac136

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

lints

9bc9df2

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update devnet to 1.1.3 unnamed

b11b943

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

update tags

48032c6

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign force-pushed the jsign/upandeff branch from 5550387 to 48032c6 Compare November 13, 2020 14:32

jsign requested a review from asutula November 13, 2020 14:58

jsign marked this pull request as ready for review November 13, 2020 14:58

jsign added minor and removed patch labels Nov 13, 2020

update to definitive tag

e2858ff

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

jsign commented Nov 13, 2020

View reviewed changes

fmt nit

b3c1245

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

asutula approved these changes Nov 13, 2020

View reviewed changes

jsign merged commit fd1029c into master Nov 13, 2020

jsign deleted the jsign/upandeff branch November 13, 2020 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve deal making efficiency & Lotus v1.1.3 update & fixes #705

Improve deal making efficiency & Lotus v1.1.3 update & fixes #705

jsign commented Nov 12, 2020 •

edited

Loading

jsign Nov 13, 2020

jsign Nov 13, 2020

jsign Nov 13, 2020

jsign Nov 13, 2020

jsign Nov 13, 2020

jsign Nov 13, 2020

asutula left a comment

		dsz, err := lapi.ClientDealSize(ctx, c)
		dsz, err := lapi.ClientDealPieceCID(ctx, c)

		require.Equal(t, "failed to start deal: cannot propose a deal whose piece size (4096) is greater than sector size (2048)", de.Message)
		require.Equal(t, "data doesn't fit in a sector", de.Message)

		score := 50faultsScore + 20powerScore + 20externalScore + 10askScore + 1000*float64(miner.ActiveDeals)
		score := 50faultsScore + 20powerScore + 20externalScore + 100askScore

Improve deal making efficiency & Lotus v1.1.3 update & fixes #705

Improve deal making efficiency & Lotus v1.1.3 update & fixes #705

Conversation

jsign commented Nov 12, 2020 • edited Loading

jsign Nov 13, 2020

Choose a reason for hiding this comment

jsign Nov 13, 2020

Choose a reason for hiding this comment

jsign Nov 13, 2020

Choose a reason for hiding this comment

jsign Nov 13, 2020

Choose a reason for hiding this comment

jsign Nov 13, 2020

Choose a reason for hiding this comment

jsign Nov 13, 2020

Choose a reason for hiding this comment

asutula left a comment

Choose a reason for hiding this comment

jsign commented Nov 12, 2020 •

edited

Loading