Network selection and protocol upgrades should be handled at runtime

[q9f]

instead of compile-time; one binary should be able to handle all configurations.

Adding here some sub-tasks:

• builtin calibnet/mainnet selection via a cli flag
• test that both networks still sync
• optional include of genesis bytes for builtin network
• support for file-based network configuration
• one folder per chain db (Add network specific databases #1418)
• fix of Add typesafe version in configuration file #1498

[willeslau]

@q9f is there more info to this issue?

[q9f]

Yes, let me add some pointers.

Filecoin does have different networks for mainnet and testing and these networks have different network versions ("hard forks" or "protocol upgrades").

Currently, there is no easy way to configure a filecoin network for the forest client. Instead, we have compile-time features that enable certain chain specifications:

cargo build --features "interopnet"
cargo build --features "devnet"
cargo build --features "calibnet"

(See Makefile for examples.)

This is not optimal, it results in different binaries for different network specifications.

Now, the ultimate goal of abstraction should be:

1. one binary for all networks
2. all network configurations should be selectable by switches
3. all network versions should be selectable by switches
4. everything should function at runtime

E.g.:

forest --chain devnet

Or:

forest --chain calibnet --override-network-version 13

Or:

forest --chain ./path/to/custom/chain.toml

These are just ideas rather than expectations. I hope this clarifies what this issue is about.

[q9f]

Take a look at the calibnet pr to see how it is currently done: #1370

[q9f]

This probably involves major refactoring. Related to #1396, we should be able to transition over network upgrades, sync from genesis, and eventually, allow for dynamic specification of chains. This would allow us to add support for any past and upcoming testnet by simply providing a toml file.

[elmattic]

Some remarks:

If env var FOREST_CONFIG_PATH is not set, then there's today no default location where forest is looking for the toml config file.

Also if a config file is present in cwd then this file should/could take precedence.

To sum-up parameters resolution could work as follows:

• Try to load config from file:

  1. Config file is in cwd, load it.
  2. If env var FOREST_CONFIG_PATH is set, look for config file in this path and load it.
  3. Look in the default user path (could be something like ~/.local/share/forest/config.toml or ~/.forest/config.toml on linux)
  4. Load default config coded in forest binary

• If a cli param is used, then override the one from config file (like we do today)

To select a chain/network type, cli parameter --chain <type> could be used to select the kind of chain to run:

[chain.calibnet]

bootstrap_nodes = [
  "/dns4/bootstrap-0.calibration.fildev.network/tcp/1347/p2p/12D3KooWJkikQQkxS58spo76BYzFt4fotaT5NpV2zngvrqm4u5ow",
  "/dns4/bootstrap-1.calibration.fildev.network/tcp/1347/p2p/12D3KooWLce5FDHR4EX4CrYavphA5xS3uDsX6aoowXh5tzDUxJav",
  ...
]

min_storage_pow = 2147483648

# The schedule is a list of network upgrades + genesis car file
# An upgrade could contain “hard consensus” or any “soft consensus” features (e.g. an optimization)
upgrades = [
  { name = "ohsnap",
    version = 15,
    height = 682006,
    features = ["fip_0023", "some_feature"]
  },
  { name = "chocolate",
    version = 14,
    height = 312746
  },
  ...
]

genesis.path = <location>
genesis.cid = <cid>

[chain.mainnet]
...

The existing parameters of today like data_dir could be set using the network/chain type as well.

[calibnet]
data_dir = "~/.forest/calibnet/db"

Some questions:
Should we have a dedicated toml/json config file per network/chain type rather than conflate everything in one single file?

Also we could have the network/chain configuration where genesis block is created from the same json like output code by command lotus-seed:

./lotus-seed genesis new localnet.json

See https://docs.filecoin.io/build/local-devnet/#manual-set-up.

[q9f]

If env var FOREST_CONFIG_PATH is not set, then there's today no default location where forest is looking for the toml config file.

We should default to $XDG_CONFIG_HOME and if not present, create a default config.toml.

• https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables

Also if a config file is present in cwd then this file should/could take precedence.

No, this would be confusing for users. If a different config were to be used, it should be explicitly overridden by some kind of --config flag.

• Try to load config from file:

  1. Config file is in cwd, load it.
  2. If env var FOREST_CONFIG_PATH is set, look for config file in this path and load it.
  3. Look in the default user path (could be something like ~/.local/share/forest/config.toml or ~/.forest/config.toml on linux)
  4. Load default config coded in forest binary

• If a cli param is used, then override the one from config file (like we do today)

The order should be:

1. $XDG_CONFIG_HOME
2. location specified by --config
3. create Forest-default config in $FOREST_CONFIG_PATH if 1 and 2 are not available

Not sure what $FOREST_CONFIG_PATH was used for in the past, do we really need it?

Some questions: Should we have a dedicated toml/json config file per network/chain type rather than conflate everything in one single file?

Yes, if possible one config per chain specification.

Also we could have the network/chain configuration where genesis block is created from the same json like output code by command lotus-seed:

That would be amazing! 👍

[connormullett]

Not sure what $FOREST_CONFIG_PATH was used for in the past, do we really need it?

This was an old attempt at doing $XDG_CONFIG_HOME without $XDG_CONFIG_HOME lol

I think we should break off from $FOREST_CONFIG_PATH and stick to using $XDG_CONFIG_HOME such that if a location isn't specified and a configuration doesnt exist there, we create one in $XDG_CONFIG_HOME