Merge branch 'main' into dependabot/npm_and_yarn/docs/prism-react-ren…

…derer-2.4.1

.github/workflows/forest.yml

@@ -565,3 +565,29 @@ jobs:
       - name: Dump docker logs
         if: always()
         uses: jwalton/gh-docker-logs@v2
+  # Umbrella job to aggregate all integration tests and get their status
+  integration-tests-status:
+    needs:
+      - build-macos
+      - build-ubuntu
+      - cargo-publish-dry-run
+      - forest-cli-check
+      - calibnet-check
+      - calibnet-stateless-mode-check
+      - state-migrations-check
+      - calibnet-wallet-check
+      - calibnet-export-check
+      - calibnet-no-discovery-checks
+      - calibnet-kademlia-checks
+      - calibnet-eth-mapping-check
+      - db-migration-checks
+      - db-migration-checks-car-db  
+      - local-devnet-check
+      - local-devnet-curio-check
+      - calibnet-rpc-checks
+      - bootstrap-checks-forest
+      - bootstrap-checks-lotus
+    name: Integration tests status
+    runs-on: ubuntu-24.04
+    steps:
+      - run: echo "All integration tests passed."

CHANGELOG.md

@@ -35,6 +35,8 @@
 
 ### Fixed
 
+- [#5111](https://github.com/ChainSafe/forest/issues/5111) Make F3 work when the node Kademlia is disabled.
+
 ## Forest v.0.23.3 "Plumber"
 
 Mandatory release for calibnet node operators. It fixes a sync error at epoch 2281645.
@@ -46,6 +48,9 @@ Mandatory release for calibnet node operators. It fixes a sync error at epoch 22
 - [#5020](https://github.com/ChainSafe/forest/issues/5020) Add support for the
   `Filecoin.EthGetTransactionByBlockNumberAndIndex` RPC method.
 
+- [#4907](https://github.com/ChainSafe/forest/issues/4907) Add support for the
+  `Filecoin.StateMinerInitialPledgeForSector` RPC method.
+
 ### Changed
 
 ### Removed

CONTRIBUTING.md

@@ -37,7 +37,7 @@ forward to your contributions. 🎉
 ## I Have a Question
 
 > If you want to ask a question, we assume that you have read the available
-> [Documentation](https://forest-docs-v2.pages.dev/).
+> [Documentation](https://docs.forest.chainsafe.io/).
 
 Before you ask a question, it is best to search for existing
 [Issues](https://github.com/ChainSafe/forest/issues) or
@@ -82,7 +82,7 @@ steps in advance to help us fix any potential bug as fast as possible.
 - Make sure that you are using the latest version.
 - Determine if your bug is really a bug and not an error on your side e.g. using
   incompatible environment components/versions (Make sure that you have read the
-  [documentation](https://forest-docs-v2.pages.dev/). If you are looking for
+  [documentation](https://docs.forest.chainsafe.io/). If you are looking for
   support, you might want to check [this section](#i-have-a-question)).
 - To see if other users have experienced (and potentially already solved) the
   same issue you are having, check if there is not already a bug report existing
@@ -143,7 +143,7 @@ community to understand your suggestion and find related suggestions.
 #### 🎯 Before Submitting an Enhancement
 
 - Make sure that you are using the latest version.
-- Read the [documentation](https://forest-docs-v2.pages.dev/) carefully and find
+- Read the [documentation](https://docs.forest.chainsafe.io/) carefully and find
   out if the functionality is already covered, maybe by an individual
   configuration.
 - Perform a [search](https://github.com/ChainSafe/forest/issues) to see if the
@@ -225,7 +225,7 @@ eternally grateful and hope you will continue to contribute to the project.
 ### 📚 Improving The Documentation
 
 The documentation is currently hosted on
-[forest-docs-v2.pages.dev](https://forest-docs-v2.pages.dev/). If you find any
+[docs.forest.chainsafe.io](https://docs.forest.chainsafe.io/). If you find any
 issues with the documentation, please create an issue as the repository is not
 public yet.
 

docs/README.md

@@ -2,32 +2,85 @@
 
 This directory contains a Docusaurus documentation website for both user and developer documentation.
 
+## Getting Started
+
+> Note: This project uses [Yarn](https://yarnpkg.com/getting-started/install)
+
 ### Installation
 
+Install the required dependencies:
+
 ```
 $ yarn
 ```
 
 ### Local Development
 
+Start local development server:
+
 ```
 $ yarn start
 ```
 
-This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.
+### Build
 
-### Spellcheck
+To compile an optimized production build:
 
 ```
-yarn spellcheck
+$ yarn build
 ```
 
-You can add unknown words to `dictionary.txt`.
+### CI Checks
 
-### Build
+These commands are recommended to run before commiting code. They are run as checks in CI.
 
 ```
-$ yarn build
+yarn spellcheck # Checks spelling
+yarn format     # Run prettier to fix formatting issues
+yarn typecheck  # Validate typescript files
 ```
 
-This command generates static content into the `build` directory and can be served using any static contents hosting service.
+> **How to Fix Spellcheck Errors:** You can add unknown words to `dictionary.txt`.
+
+### Deployment
+
+The documentation site is continuously deployed to CloudFlare Pages, triggered on every commit to `main`. [This workflow](/.github/workflows/docs-deploy.yml) defines the deployment process.
+
+## Site Structure
+
+The site it comprised of two sub-sites - user documentation (`/`) and developer documentation (`/developers`). There is intentionally no link from the user docs to the developer docs, to avoid overwhelming users with unnecessary info.
+
+### User Docs
+
+Available at the root (`/`), default location for anyone visiting the documentation domain. Source files are under `docs/user`
+
+Follows the [Diátaxis](https://diataxis.fr/) framework for structuring documentation. The site is divided into four types of documentation: tutorials (Getting Started), how-to guides (Guides), explanations (Knowledge Base) and reference (Reference).
+
+#### CLI Docs
+
+These docs are automatically generated from the Forest CLI. See [script](/docs/docs/users/reference/cli.sh).
+
+#### JSON-RPC Docs
+
+We use the OpenRPC document from Forest to populate the documentation for each method. For this we use `@metamask/docusaurus-openrpc`.
+
+### Developer Docs
+
+Available at `/developers`, source code is located in `docs/developers`. Comprised of a collection of documents aimed at contributors. May be relevant to power users.
+
+> Note: As a general rule of thumb, if it involes reading or writing Rust, it should live under the Developer documentation.
+
+## Contributing
+
+### References
+
+- [Docusaurus Guide](https://docusaurus.io/docs/category/guides)
+- [Docusaurus Configuration Docs](https://docusaurus.io/docs/api/docusaurus-config)
+- [Forest Contributor Guidelines](../CONTRIBUTING.md)
+
+### Useful Features
+
+- Admonitions (eg. Info, Warning, etc): https://docusaurus.io/docs/markdown-features/admonitions
+- Mermaid Diagrams: https://docusaurus.io/docs/markdown-features/diagrams
+- MDX (embedding JavaScript): https://docusaurus.io/docs/markdown-features/react
+- Code Blocks: https://docusaurus.io/docs/markdown-features/code-blocks

f3-sidecar/p2p.go

@@ -14,9 +14,10 @@ import (
 const ListenAddr = "/ip4/127.0.0.1/tcp/0"
 
 type P2PHost struct {
-	Host   host.Host
-	DHT    *dht.IpfsDHT
-	PubSub *pubsub.PubSub
+	Host      host.Host
+	DHT       *dht.IpfsDHT
+	BackupDHT *dht.IpfsDHT
+	PubSub    *pubsub.PubSub
 }
 
 func createP2PHost(ctx context.Context, networkName string) (*P2PHost, error) {
@@ -36,6 +37,17 @@ func createP2PHost(ctx context.Context, networkName string) (*P2PHost, error) {
 		return nil, err
 	}
 
+	backupDthOpts := []dht.Option{
+		dht.Mode(dht.ModeAutoServer),
+		dht.ProtocolPrefix(protocol.ID(fmt.Sprintf("/fil/kad/f3-sidecar/%s", networkName))),
+		dht.DisableProviders(),
+		dht.DisableValues(),
+	}
+	backupHostDHT, err := dht.New(ctx, host, backupDthOpts...)
+	if err != nil {
+		return nil, err
+	}
+
 	ps, err := pubsub.NewGossipSub(ctx, host,
 		pubsub.WithPeerExchange(true),
 		pubsub.WithFloodPublish(true),
@@ -44,5 +56,5 @@ func createP2PHost(ctx context.Context, networkName string) (*P2PHost, error) {
 		return nil, err
 	}
 
-	return &P2PHost{host, hostDHT, ps}, nil
+	return &P2PHost{host, hostDHT, backupHostDHT, ps}, nil
 }

scripts/tests/calibnet_no_discovery_check.sh

@@ -5,11 +5,21 @@ set -euxo pipefail
 
 source "$(dirname "$0")/harness.sh"
 
-$FOREST_PATH --chain calibnet --encrypt-keystore false --mdns false --kademlia false --auto-download-snapshot --log-dir "$LOG_DIRECTORY" --detach --save-token ./admin_token
+$FOREST_PATH --chain calibnet --encrypt-keystore false --mdns false --kademlia false --auto-download-snapshot --save-token ./admin_token --exit-after-init
+$FOREST_PATH --chain calibnet --encrypt-keystore false --mdns false --kademlia false --auto-download-snapshot --log-dir "$LOG_DIRECTORY" &
+FOREST_NODE_PID=$!
+
 FULLNODE_API_INFO="$(cat admin_token):/ip4/127.0.0.1/tcp/2345/http"
 export FULLNODE_API_INFO
 
 # Verify that one of the seed nodes has been connected to
 until $FOREST_CLI_PATH net peers | grep "calib"; do
     sleep 1s;
 done
+
+# Verify F3 is getting certificates from the network
+until [[ $($FOREST_CLI_PATH f3 certs get --output json | jq '.GPBFTInstance') -gt 100 ]]; do
+    sleep 1s;
+done
+kill -KILL $FOREST_NODE_PID
+wait $FOREST_NODE_PID || true

src/f3/mod.rs

@@ -100,6 +100,7 @@ pub fn run_f3_sidecar_if_enabled(
     if is_sidecar_ffi_enabled(chain_config) {
         #[cfg(all(f3sidecar, not(feature = "no-f3-sidecar")))]
         {
+            tracing::info!("Starting F3 sidecar service ...");
             GoF3NodeImpl::run(
                 _rpc_endpoint,
                 _jwt,

src/libp2p/discovery.rs

@@ -35,6 +35,8 @@ use crate::utils::version::FOREST_VERSION_STRING;
 pub struct DerivedDiscoveryBehaviour {
     /// Kademlia discovery.
     kademlia: Toggle<kad::Behaviour<kad::store::MemoryStore>>,
+    /// Kademlia discovery for bootstrapping F3 sidecar when the main Kademlia is disabled.
+    kademlia_f3_sidecar: kad::Behaviour<kad::store::MemoryStore>,
     /// Discovers nodes on the local network.
     mdns: Toggle<Mdns>,
     /// [`identify::Behaviour`] needs to be manually hooked up with [`kad::Behaviour`] to make discovery work. See <https://docs.rs/libp2p/latest/libp2p/kad/index.html#important-discrepancies>
@@ -158,6 +160,12 @@ impl<'a> DiscoveryConfig<'a> {
         } else {
             None
         };
+        let kademlia_f3_sidecar = new_kademlia(
+            local_peer_id,
+            StreamProtocol::try_from_owned(format!(
+                "/fil/kad/f3-sidecar/{network_name}/kad/1.0.0"
+            ))?,
+        );
 
         let mdns_opt = if enable_mdns {
             Some(Mdns::new(Default::default(), local_peer_id).expect("Could not start mDNS"))
@@ -168,6 +176,7 @@ impl<'a> DiscoveryConfig<'a> {
         Ok(DiscoveryBehaviour {
             discovery: DerivedDiscoveryBehaviour {
                 kademlia: kademlia_opt.into(),
+                kademlia_f3_sidecar,
                 mdns: mdns_opt.into(),
                 identify: identify::Behaviour::new(
                     identify::Config::new("ipfs/0.1.0".into(), local_public_key)
@@ -441,6 +450,11 @@ impl NetworkBehaviour for DiscoveryBehaviour {
                                         kademlia.add_address(peer_id, address.clone());
                                     }
                                 }
+                                for address in &info.listen_addrs {
+                                    self.discovery
+                                        .kademlia_f3_sidecar
+                                        .add_address(peer_id, address.clone());
+                                }
                             }
                         }
                         DerivedDiscoveryBehaviourEvent::Autonat(_) => {}
@@ -470,6 +484,7 @@ impl NetworkBehaviour for DiscoveryBehaviour {
                                 trace!("Libp2p => Unhandled Kademlia event: {:?}", other)
                             }
                         },
+                        DerivedDiscoveryBehaviourEvent::KademliaF3Sidecar(_) => {}
                         DerivedDiscoveryBehaviourEvent::Mdns(ev) => match ev {
                             MdnsEvent::Discovered(list) => {
                                 if self.n_node_connected >= self.target_peer_count {

src/rpc/methods/state.rs

@@ -2800,3 +2800,94 @@ impl TryFrom<&ChainConfig> for ForkUpgradeParams {
         })
     }
 }
+
+pub enum StateMinerInitialPledgeForSector {}
+impl RpcMethod<4> for StateMinerInitialPledgeForSector {
+    const NAME: &'static str = "Filecoin.StateMinerInitialPledgeForSector";
+    const PARAM_NAMES: [&'static str; 4] = [
+        "sector_duration",
+        "sector_size",
+        "verified_size",
+        "tipset_key",
+    ];
+    const API_PATHS: ApiPaths = ApiPaths::V1;
+    const PERMISSION: Permission = Permission::Read;
+
+    type Params = (ChainEpoch, SectorSize, u64, ApiTipsetKey);
+    type Ok = TokenAmount;
+
+    async fn handle(
+        ctx: Ctx<impl Blockstore + Send + Sync + 'static>,
+        (sector_duration, sector_size, verified_size, ApiTipsetKey(tsk)): Self::Params,
+    ) -> Result<Self::Ok, ServerError> {
+        if sector_duration <= 0 {
+            return Err(anyhow::anyhow!("sector duration must be greater than 0").into());
+        }
+        if verified_size > sector_size as u64 {
+            return Err(
+                anyhow::anyhow!("verified deal size cannot be larger than sector size").into(),
+            );
+        }
+
+        let ts = ctx.chain_store().load_required_tipset_or_heaviest(&tsk)?;
+
+        let power_state: power::State = ctx.state_manager.get_actor_state(&ts)?;
+        let power_smoothed = power_state.total_power_smoothed();
+        let pledge_collateral = power_state.total_locked();
+
+        let reward_state: reward::State = ctx.state_manager.get_actor_state(&ts)?;
+
+        let genesis_info = GenesisInfo::from_chain_config(ctx.chain_config().clone());
+        let circ_supply = genesis_info.get_vm_circulating_supply_detailed(
+            ts.epoch(),
+            &ctx.store_owned(),
+            ts.parent_state(),
+        )?;
+
+        let deal_weight = BigInt::from(0);
+        let verified_deal_weight = BigInt::from(verified_size) * sector_duration;
+        let sector_weight = qa_power_for_weight(
+            sector_size.into(),
+            sector_duration,
+            &deal_weight,
+            &verified_deal_weight,
+        );
+
+        let (epochs_since_start, duration) = get_pledge_ramp_params(&ctx, ts.epoch(), &ts)?;
+
+        let initial_pledge: TokenAmount = reward_state
+            .initial_pledge_for_power(
+                &sector_weight,
+                pledge_collateral,
+                power_smoothed,
+                &circ_supply.fil_circulating.into(),
+                epochs_since_start,
+                duration,
+            )?
+            .into();
+
+        let (value, _) = (initial_pledge * INITIAL_PLEDGE_NUM).div_rem(INITIAL_PLEDGE_DEN);
+        Ok(value)
+    }
+}
+
+fn get_pledge_ramp_params(
+    ctx: &Ctx<impl Blockstore + Send + Sync + 'static>,
+    height: ChainEpoch,
+    ts: &Tipset,
+) -> Result<(ChainEpoch, u64), anyhow::Error> {
+    let state_tree = ctx.state_manager.get_state_tree(ts.parent_state())?;
+
+    let power_state: power::State = state_tree
+        .get_actor_state()
+        .context("loading power actor state")?;
+
+    if power_state.ramp_start_epoch() > 0 {
+        Ok((
+            height - power_state.ramp_start_epoch(),
+            power_state.ramp_duration_epochs(),
+        ))
+    } else {
+        Ok((0, 0))
+    }
+}