SR2 miner selector & safer default StorageConfig config #614
Conversation
| var ms ffs.MinerSelector | ||
| var err error | ||
|
|
||
| switch conf.MinerSelector { |
There was a problem hiding this comment.
MinerSelector configuration depending on flags/envs.
cmd/powd/main.go
Outdated
| @@ -301,6 +308,8 @@ func setupFlags() error { | |||
| pflag.String("maxminddbfolder", ".", "Path of the folder containing GeoLite2-City.mmdb") | |||
| pflag.String("mongouri", "", "Mongo URI to connect to MongoDB database. (Optional: if empty, will use Badger)") | |||
| pflag.String("mongodb", "", "Mongo database name. (if --mongouri is used, is mandatory") | |||
| pflag.String("ffsminerselector", "sr2", "Miner selector to be used by FFS: 'sr2', 'reputation'") | |||
There was a problem hiding this comment.
MinerSelector can be:
sr2: the default now. Does what was commented in PR description.reputation: The previous miner selector that uses the reputation index, and still directly targets peter's miners as trusted.
| Enabled: true, | ||
| Enabled: false, |
There was a problem hiding this comment.
Decided with @asutula to avoid disk usage explosion.
If hte user want's HotStorage enabled, should be set explicitly on pushing a config.
| @@ -0,0 +1,143 @@ | |||
| package sr2 | |||
There was a problem hiding this comment.
Summary of the selector:
- Fetch the json file
- Parse it
- Get covered on some possibly weird values (amount < 0, etc)
- Use the ask cache index as much as possible. Some miners might not be there if they have 0 power. So instead of changing the ask index to include also miners with 0 power (which are a lot and would affect index building times), I decided that if a miner in this trusted list hasn't a cached ask, we'd do that directly here.
| } | ||
|
|
||
| // GetMiners returns miners from SR2. | ||
| func (ms *MinerSelector) GetMiners(n int, f ffs.MinerSelectorFilter) ([]ffs.MinerProposal, error) { |
There was a problem hiding this comment.
Important fact. This MinerSelector will ignore n and f.
The replication factor will be the sum of Amount of buckets, and not filtering is done.
So this is a very aggressive miner selector kind of forcing what was decided in SR2. No custom StorageConfig will change this fact, unless the user runs with --ffsminerselector=reputation.
There was a problem hiding this comment.
I can't remember, but I suppose the replication factor provided in the storage config is passed into a miner selector via ffs.MinerSelectorFilter? So this means that user provided replication factor is ignored? That is fine, just want to be sure I understand.
There was a problem hiding this comment.
Yes, is ignored. Now will be controled in this remote file now that --ffsminerselector=sr2 default miner selector.
If someone wants to come back to not being ignored, they could specify --ffsminerselector=reputation and things will work as before.
jsign
changed the title
ffs/scheduler/scheduler_storage.go
Outdated
| // ToDo: this is a hot-fix to force a big timeout until we have a | ||
| // migration tool to make this tunable again. | ||
| sctx, cancel := context.WithTimeout(ctx, time.Second*300) |
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
| } | ||
|
|
||
| // GetMiners returns miners from SR2. | ||
| func (ms *MinerSelector) GetMiners(n int, f ffs.MinerSelectorFilter) ([]ffs.MinerProposal, error) { |
There was a problem hiding this comment.
I can't remember, but I suppose the replication factor provided in the storage config is passed into a miner selector via ffs.MinerSelectorFilter? So this means that user provided replication factor is ignored? That is fine, just want to be sure I understand.
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
This PR creates a new
MinerSelectorimplementation, which executes the planned strategy for selecting miners in SR2.In a nutshell, when this miner selector is used:
Amountrandom miners from each miners bucket.This allows to:
github.com/filecoin-project/slingshotrepository, and automatically update every running Powerate without restarts. Every time a list of miners needs to be selected, it will fetch the currentlist.jsonin themasterbranch of that repo.Amountattribute of each miners bucket in the json file.By default now we configure Poweragate to use this miner selector strategy. A new flag was included if someone wants to keep using the previous one; so at the end of the day is configurable. The URL where this JSON will be fetched is also configurable via flags/envs, so if someone wants to use their own JSON file, that's OK too.
So in short, the default now is to have the right configuration for the SR2 contest.
Some other lateral changes:
AddTimeoutis hardcoded to a bigger value ignoring the StorageConfig. This is a temporal solution until we have a bit more control to make general StorageConfig changes in the Hub.lotus-devnetwas updated to v0.7.2.reputationminerselector, since SR2 want to hold a bit using the new one created in this PR.