Migrate Forest to jsonrpsee crate

[elmattic]

Summary of changes

Changes introduced in this pull request:

• Migrate to jsonrpsee server

$ ./forest-tool api compare --lotus /ip4/127.0.0.1/tcp/1234/http --forest /ip4/127.0.0.1/tcp/2345/http forest_snapshot_calibnet_2024-02-05_height_1326491.forest.car.zst

| RPC Method                                     | Forest              | Lotus               |
|------------------------------------------------|---------------------|---------------------|
| Filecoin.BeaconGetEntry                        | Valid               | Valid               |
| Filecoin.ChainGetBlock                         | Valid               | Valid               |
| Filecoin.ChainGetBlockMessages (61)            | Valid               | Valid               |
| Filecoin.ChainGetGenesis                       | Valid               | Valid               |
| Filecoin.ChainGetMessage (2)                   | Valid               | Valid               |
| Filecoin.ChainGetMessagesInTipset (20)         | Valid               | Valid               |
| Filecoin.ChainGetParentMessages (61)           | Valid               | Valid               |
| Filecoin.ChainGetParentReceipts (61)           | Valid               | Valid               |
| Filecoin.ChainGetTipSet                        | Valid               | Valid               |
| Filecoin.ChainGetTipSetByHeight                | Valid               | Valid               |
| Filecoin.ChainHasObj                           | Valid               | Valid               |
| Filecoin.ChainHead                             | Valid               | Valid               |
| Filecoin.ChainReadObj                          | Valid               | Valid               |
| Filecoin.EthAccounts                           | Valid               | Valid               |
| Filecoin.EthBlockNumber                        | Valid               | Valid               |
| Filecoin.EthChainId                            | Valid               | Valid               |
| Filecoin.EthGasPrice                           | Valid               | Valid               |
| Filecoin.EthGetBalance (4)                     | Valid               | Valid               |
| Filecoin.MinerGetBaseInfo (7)                  | InvalidResponse     | Valid               |
| Filecoin.MinerGetBaseInfo (54)                 | Valid               | Valid               |
| Filecoin.MpoolGetNonce (2)                     | Valid               | Valid               |
| Filecoin.MpoolPending                          | Valid               | Valid               |
| Filecoin.MsigGetAvailableBalance               | Valid               | Valid               |
| Filecoin.MsigGetPending                        | Valid               | Valid               |
| Filecoin.NetAddrsListen                        | Valid               | Valid               |
| Filecoin.NetPeers                              | Valid               | Valid               |
| Filecoin.Session                               | Valid               | Valid               |
| Filecoin.StartTime                             | Valid               | Valid               |
| Filecoin.StateAccountKey (4)                   | Valid               | Valid               |
| Filecoin.StateCall (7)                         | Valid               | Valid               |
| Filecoin.StateCirculatingSupply (20)           | Valid               | Valid               |
| Filecoin.StateGetActor                         | Valid               | Valid               |
| Filecoin.StateGetRandomnessFromBeacon          | Valid               | Valid               |
| Filecoin.StateGetRandomnessFromTickets         | Valid               | Valid               |
| Filecoin.StateListMessages (2)                 | InternalServerError | InternalServerError |
| Filecoin.StateListMessages (4)                 | Valid               | Valid               |
| Filecoin.StateListMiners                       | Valid               | Valid               |
| Filecoin.StateLookupID (4)                     | Valid               | Valid               |
| Filecoin.StateMinerActiveSectors (62)          | Valid               | Valid               |
| Filecoin.StateMinerDeadlines (61)              | Valid               | Valid               |
| Filecoin.StateMinerFaults (61)                 | Valid               | Valid               |
| Filecoin.StateMinerInfo (2)                    | InvalidResponse     | Valid               |
| Filecoin.StateMinerInfo (59)                   | Valid               | Valid               |
| Filecoin.StateMinerPower (61)                  | Valid               | Valid               |
| Filecoin.StateMinerProvingDeadline (61)        | Valid               | Valid               |
| Filecoin.StateMinerRecoveries (61)             | Valid               | Valid               |
| Filecoin.StateMinerSectorCount (61)            | Valid               | Valid               |
| Filecoin.StateNetworkName                      | Valid               | Valid               |
| Filecoin.StateNetworkVersion                   | Valid               | Valid               |
| Filecoin.StateReadState (2)                    | Valid               | Valid               |
| Filecoin.StateSearchMsg                        | InternalServerError | InternalServerError |
| Filecoin.StateSearchMsg                        | Valid               | InternalServerError |
| Filecoin.StateSearchMsgLimited                 | InternalServerError | Valid               |
| Filecoin.StateSearchMsgLimited                 | InvalidResponse     | Valid               |
| Filecoin.StateSectorGetInfo                    | Valid               | Valid               |
| Filecoin.StateVMCirculatingSupplyInternal (20) | Valid               | Valid               |
| Filecoin.StateVerifiedClientStatus (2)         | Valid               | Valid               |
| Filecoin.StateWaitMsg                          | Timeout             | Timeout             |
| Filecoin.StateWaitMsg                          | Valid               | Timeout             |
| Filecoin.Version                               | Valid               | Valid               |
| Filecoin.WalletBalance                         | Valid               | Valid               |
| Filecoin.WalletValidateAddress                 | Valid               | Valid               |
| Filecoin.WalletVerify                          | Valid               | Valid               |

Websocket transport flag should work as well:

$ ./forest-tool api compare --ws --lotus /ip4/127.0.0.1/tcp/1234/ws --forest /ip4/127.0.0.1/tcp/2345/ws forest_snapshot_calibnet_2024-02-14_height_1352592.forest.car.zst

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

[aatifsyed]

I don't understand why we need Arc<Arc<T>> here - could you justify or remove?

[elmattic]

The first Arc is needed to share the state between two RPC modules.

forest/src/rpc/mod.rs

Line 69 in ce33815

let state = Arc::new(state);

We only have one module now, but to implement specific pubsub code, we will need to implement a dedicated RpcModule.
In the context of async methods, jsonrpsee creates a second Arc around the Context type:

pub struct RpcModule<Context> {
    ctx: Arc<Context, Global>,
    methods: Methods,
}

[ruseinov]

Quick question: do we really need to share the state between modules?
And if so - perhaps the best approach is to type alias this Arc<Arc stuff away and keep it in one place instead of declaring it on every method.

[elmattic]

Yes, because all methods in a module need access to the state.

In our case, we will have to call chain_api::chain_notify to get the subscriber:

async fn chain_notify<DB: Blockstore>(
     data: Arc<RPCState<DB>>,
 ) -> Result<Subscriber<HeadChange>, JsonRpseeError> {
     let head_change = data.chain_store.publisher().subscribe();

     Ok(head_change)
 }

[elmattic]

Hum, there's only one Arc here.

[elmattic]

Ah I see why, ignore my comment.

[aatifsyed]

What's the resolution here? this is the only thing blocking an approval from me I think

[elmattic]

I think the double Arc<Arc<T>> is ok for now (I've hidden the ugliness using the Data type alias). This should resolve at the jsonrpsee crate level. Maybe with something like:

pub fn from_arc(ctx: Arc<Context>) -> Self {
  Self { ctx, methods: Default::default() }
}

[aatifsyed]

I'm grumpy about this, but if there's truly no other way then fine

[elmattic]

My from_arc works and I can remove one Arc level: pub type Data<T> = Arc<T>;

This will require change in upstream repository though.

[ruseinov]

Seems good apart from two nits already mentioned by @aatifsyed.

Perhaps the error codes could be improved, so that we at least have a way of knowing which error type was returned.
And the double Arc thing has to be either avoided or type aliased if unavoidable.

[elmattic]

Perhaps the error codes could be improved, so that we at least have a way of knowing which error type was returned.

Yes, will do. Error codes could be much more precise.

And we don't always validate params; for example, in eth_chain_id; if we send incorrect params, Lotus answers with:

    JsonRpcError {
        code: 500,
        message: "{\"jsonrpc\":\"2.0\",\"id\":0,\"error\":{\"code\":-32602,\"message\":\"wrong param count (method 'Filecoin.EthChainId'): 2 != 0\"}}\n",
    },

While for Forest, it's valid. This can be addressed in a follow-up PR, though.