mpm integration-test support fork from remote and call-api #3600

pause125 · 2022-08-02T15:14:19Z

Pull request type

Please check the type of change your PR introduces:

What is the current behavior?

Issue Number: N/A

What is the new behavior?

实现 MockChain 和 MockStateDB，从给定的区块高度 fork；后续查询时，block_number 小于 fork number 时，从远端节点读取，否则从本地 Storage 读取
StateApi 新增接口：给定 StateTree 的 key（HashValue形式），直接从 StateTree 中读取内容

Other information

已完成内容：

整体结构和流程
少部分的 ChainApi 和 StateApi 接口，用于测试

未完成内容：

大部分 ChainApi 和 StateApi 的接口具体实现
单元测试。对于 fork 后新增的状态和区块，需要进行哪些测试？目前，仅可以测试最新 state_root 和最新区块里的 state_root 的一致性。
代码整理。

pause125 · 2022-08-02T15:19:49Z

主要流程基本完成，剩下 API 接口的实现，请老师帮忙看看是否有问题？ @jolestar

rpc/api/src/state/mod.rs

vm/starcoin-transactional-test-harness/src/lib.rs

vm/starcoin-transactional-test-harness/src/fork_chain.rs

pause125 · 2022-08-06T05:34:13Z

目前存在的问题：

ChainApi 中一些 transaction 相关的接口，需要 block 信息，但是 mpm integration-test 中 transaction 是独立于 block 的，因此这些接口没有实现。还有一些感觉不适合的接口也没有实现。
RPC 新增 StateApi::get_state_node_by_node_hash，单元测试中有的测试项从 main/banard 网络 fork，需要该接口，但是目前 main/barnard 网络没有该接口，因此这些测试 case 无法跑通，先注释掉了。这些测试 case 已经在 fork 本地 dev 网络的情景下测试过了。

codecov · 2022-08-06T06:01:34Z

Codecov Report

Merging #3600 (a68e949) into master (06d5321) will decrease coverage by 0.09%.
The diff coverage is 0.18%.

@@            Coverage Diff             @@
##           master    #3600      +/-   ##
==========================================
- Coverage   28.95%   28.87%   -0.08%     
==========================================
  Files         589      591       +2     
  Lines       49864    50342     +478     
  Branches    23450    23715     +265     
==========================================
+ Hits        14434    14529      +95     
- Misses      21274    21655     +381     
- Partials    14156    14158       +2

Flag	Coverage Δ
unittests	`28.87% <0.18%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
rpc/api/src/state/mod.rs	`31.92% <0.00%> (-0.69%)`	⬇️
rpc/api/src/types.rs	`13.72% <0.00%> (-0.23%)`	⬇️
rpc/server/src/module/state_rpc.rs	`4.72% <0.00%> (-0.18%)`	⬇️
...starcoin-transactional-test-harness/src/context.rs	`0.00% <0.00%> (ø)`
...rcoin-transactional-test-harness/src/fork_chain.rs	`0.00% <0.00%> (ø)`
...rcoin-transactional-test-harness/src/fork_state.rs	`0.00% <0.00%> (ø)`
...oin-transactional-test-harness/src/remote_state.rs	`0.00% <0.00%> (ø)`
vm/starcoin-transactional-test-harness/src/lib.rs	`0.18% <1.50%> (-<0.01%)`	⬇️
txpool/src/pool/listener/tests.rs	`10.91% <0.00%> (-3.63%)`	⬇️
miner/src/generate_block_event_pacemaker.rs	`51.17% <0.00%> (-2.32%)`	⬇️
... and 70 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 06d5321...a68e949. Read the comment docs.

jolestar

LGTM

baichuan3 · 2022-08-09T02:51:03Z

目前存在的问题：

ChainApi 中一些 transaction 相关的接口，需要 block 信息，但是 mpm integration-test 中 transaction 是独立于 block 的，因此这些接口没有实现。还有一些感觉不适合的接口也没有实现。

RPC 新增 StateApi::get_state_node_by_node_hash，单元测试中有的测试项从 main/banard 网络 fork，需要该接口，但是目前 main/barnard 网络没有该接口，因此这些测试 case 无法跑通，先注释掉了。这些测试 case 已经在 fork 本地 dev 网络的情景下测试过了。

准备在DAO里接入这部分代码，目前只支持fork dev网络进行集成测试？

jolestar · 2022-08-09T06:24:22Z

目前存在的问题：

ChainApi 中一些 transaction 相关的接口，需要 block 信息，但是 mpm integration-test 中 transaction 是独立于 block 的，因此这些接口没有实现。还有一些感觉不适合的接口也没有实现。

RPC 新增 StateApi::get_state_node_by_node_hash，单元测试中有的测试项从 main/banard 网络 fork，需要该接口，但是目前 main/barnard 网络没有该接口，因此这些测试 case 无法跑通，先注释掉了。这些测试 case 已经在 fork 本地 dev 网络的情景下测试过了。

准备在DAO里接入这部分代码，目前只支持fork dev网络进行集成测试？

halley 也行，不过 barnard 和 main 需要等发布新版本后部署后

baichuan3 · 2022-08-09T08:36:57Z

在halley上测试call-api方法，integration test 会一直hang住 @jolestar ：

//# init --rpc http://halley.seed.starcoin.org --block-number 76107

//# faucet --addr creator --amount 100000000000

//# call-api state.get_with_proof_by_root_raw ["0x6bfb460477adf9dd0455d3de2fc7f211/1/0x00000000000000000000000000000001::IdentifierNFT::IdentifierNFT<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMember<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>,0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMemberBody<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>>","0xd5cd5dc44799c989a84b7d4a810259f373b13a9bf8ee21ecbed0fab264e2090d"]

//TODO how to convert the type.
//# run --signers creator --args {{$.call-api[0]}}
script{
use StarcoinFramework::Debug;

fun main(_sender: signer, snpashot_raw_proofs: vector<u8>){
    let expect_raw_proofs = x"0145016bfb460477adf9dd0455d3de2fc7f21101000000000000000664616f313031000a69616d67655f64617461005704000000000000640000000000000000000000000000000145020120cc969848619e507450ebf01437155ab5f2dbb554fe611cb71958855a1b2ec664012035f2374c333a51e46b62b693ebef25b9be2cefde8d156db08bff28f9a0b87742012073837fcf4e69ae60e18ea291b9f25c86ce8053c6ee647a2b287083327c67dd7e20dac300fb581a6df034a44a99e8eb1cd55c7dd8f0a9e78f1114a6b4eda01d834e0e2078179a07914562223d40068488a0d65673b3b2681642633dc33904575f8f070b20b7c2a21de200ca82241e5e480e922258591b219dd56c24c4d94299020ee8299a204f541db82477510f699c9d75dd6d8280639a1ec9d0b90962cbc0c2b06514a78b20cacfce99bb564cfb70eec6a61bb76b9d56eb1b626d7fa231338792e1f572a8df20da6c1337ca5d8f0fa18b2db35844c610858a710edac35206ef0bf52fd32a4ac920ef6fb8f82d32ca2b7c482b7942505e6492bffa3ed14dd635bae16a14b4ac32e6202ad7b36e08e7b5d208de8eec1ef1964dc8433ccca8ac4632f36054926e858ac32027d5ccc8aa57b964ad50334f62821188b89945ae999ad0bb31cdc16df1763f8120afa983813953d6aa9563db12d5e443c9e8114c3482867c95a661a240d6f0e0ec206466ac318f5d9deb7b64b12622a0f4bed2f19379667d18b6ccfeaa84171d812f20eb45021b7b39887925a5b49018cdc8ad44c14835a42e5775666315f4a3e0ba42204c2b365a78e4615873772a0f039a7326150472b4923d40640863dbe42a2351eb20825d0c21dd1105faf528934842419f8661d695fc72ec6ef8036f5d03359126d3205d806c027eecdfbc3960e68c5997718a0709a6079f96e1af3ffe21878ada2b830120fa60f8311936961f5e9dee5ccafaea83ed91c6eaa04a7dea0b85a38cf84d8564207ef6a85019523861474cdf47f4db8087e5368171d95cc2c1e57055a72ca39cb704208db1e4e4c864882bd611b1cda02ca30c43b3c7bc56ee7cb174598188da8b49ef2063b3f1e4f05973830ba40e0c50c4e59f31d3baa5643d19676ddbacbf797bf6b720b39d31107837c1751d439706c8ddac96f8c148b8430ac4f40546f33fb9871e4320fb0ad035780bb8f1c6481bd674ccad0948cd2e8e6b97c08e582f67cc26918fb3";
    Debug::print(&snpashot_raw_proofs);
    assert!(expect_raw_proofs == snpashot_raw_proofs, 8001);
}

}
// check: EXECUTED

pause125 · 2022-08-09T08:55:36Z

在halley上测试call-api方法，integration test 会一直hang住 @jolestar ：

//# init --rpc http://halley.seed.starcoin.org --block-number 76107

//# faucet --addr creator --amount 100000000000

//# call-api state.get_with_proof_by_root_raw ["0x6bfb460477adf9dd0455d3de2fc7f211/1/0x00000000000000000000000000000001::IdentifierNFT::IdentifierNFT<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMember<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>,0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMemberBody<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>>","0xd5cd5dc44799c989a84b7d4a810259f373b13a9bf8ee21ecbed0fab264e2090d"]

//TODO how to convert the type. //# run --signers creator --args {{$.call-api[0]}} script{ use StarcoinFramework::Debug;

fun main(_sender: signer, snpashot_raw_proofs: vector<u8>){
    let expect_raw_proofs = x"0145016bfb460477adf9dd0455d3de2fc7f21101000000000000000664616f313031000a69616d67655f64617461005704000000000000640000000000000000000000000000000145020120cc969848619e507450ebf01437155ab5f2dbb554fe611cb71958855a1b2ec664012035f2374c333a51e46b62b693ebef25b9be2cefde8d156db08bff28f9a0b87742012073837fcf4e69ae60e18ea291b9f25c86ce8053c6ee647a2b287083327c67dd7e20dac300fb581a6df034a44a99e8eb1cd55c7dd8f0a9e78f1114a6b4eda01d834e0e2078179a07914562223d40068488a0d65673b3b2681642633dc33904575f8f070b20b7c2a21de200ca82241e5e480e922258591b219dd56c24c4d94299020ee8299a204f541db82477510f699c9d75dd6d8280639a1ec9d0b90962cbc0c2b06514a78b20cacfce99bb564cfb70eec6a61bb76b9d56eb1b626d7fa231338792e1f572a8df20da6c1337ca5d8f0fa18b2db35844c610858a710edac35206ef0bf52fd32a4ac920ef6fb8f82d32ca2b7c482b7942505e6492bffa3ed14dd635bae16a14b4ac32e6202ad7b36e08e7b5d208de8eec1ef1964dc8433ccca8ac4632f36054926e858ac32027d5ccc8aa57b964ad50334f62821188b89945ae999ad0bb31cdc16df1763f8120afa983813953d6aa9563db12d5e443c9e8114c3482867c95a661a240d6f0e0ec206466ac318f5d9deb7b64b12622a0f4bed2f19379667d18b6ccfeaa84171d812f20eb45021b7b39887925a5b49018cdc8ad44c14835a42e5775666315f4a3e0ba42204c2b365a78e4615873772a0f039a7326150472b4923d40640863dbe42a2351eb20825d0c21dd1105faf528934842419f8661d695fc72ec6ef8036f5d03359126d3205d806c027eecdfbc3960e68c5997718a0709a6079f96e1af3ffe21878ada2b830120fa60f8311936961f5e9dee5ccafaea83ed91c6eaa04a7dea0b85a38cf84d8564207ef6a85019523861474cdf47f4db8087e5368171d95cc2c1e57055a72ca39cb704208db1e4e4c864882bd611b1cda02ca30c43b3c7bc56ee7cb174598188da8b49ef2063b3f1e4f05973830ba40e0c50c4e59f31d3baa5643d19676ddbacbf797bf6b720b39d31107837c1751d439706c8ddac96f8c148b8430ac4f40546f33fb9871e4320fb0ad035780bb8f1c6481bd674ccad0948cd2e8e6b97c08e582f67cc26918fb3";
    Debug::print(&snpashot_raw_proofs);
    assert!(expect_raw_proofs == snpashot_raw_proofs, 8001);
}

} // check: EXECUTED

我看看

jolestar · 2022-08-09T09:41:55Z

@baichuan3 是不是网络问题，用 curl 测试一下网络

baichuan3 · 2022-08-09T09:50:38Z

curl --location --request POST 'https://halley-seed.starcoin.org'
--header 'Content-Type: application/json'
--data-raw '{
"id":200,
"jsonrpc":"2.0",
"method":"state.get_with_proof_by_root_raw",
"params":["0x6bfb460477adf9dd0455d3de2fc7f211/1/0x00000000000000000000000000000001::IdentifierNFT::IdentifierNFT<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMember<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>, 0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::DaoMemberBody<0x6bfb460477adf9dd0455d3de2fc7f211::SBTModule::SbtTestDAO>>","0x7c0c3b5d037241d4487f4eaf77a56c3f9600441b7524df6b9ad3f34f91364ed2"]
}'

{
"jsonrpc": "2.0",
"result": "0x0145016bfb460477adf9dd0455d3de2fc7f21101000000000000000664616f313031000a69616d67655f64617461005704000000000000640000000000000000000000000000000145020120cc969848619e507450ebf01437155ab5f2dbb554fe611cb71958855a1b2ec6640120e9036d04ecb8da65ec2f576e3d44a7e3b361615665528d5b3a24cd55d331c1be012073837fcf4e69ae60e18ea291b9f25c86ce8053c6ee647a2b287083327c67dd7e201e488652be3576bbdf5f767ac4c45ea65b626a7185af04c600f3a7b7582199cc0720bd651193de5be5fd19c40b9dbc26d52d22bc40b65f92112cb9dc69560b811a37205350415253455f4d45524b4c455f504c414345484f4c4445525f484153480000205350415253455f4d45524b4c455f504c414345484f4c4445525f484153480000200911f753d151afd303488aa85cb246986fbf18cc055f29f5aa18e2159af38ac42053ef9425d82ac8137c11554fbde3cc938c6ee1302a8953f3a83ae97c33a5779b203ac06fbab3073f4c710197531e67e606998f31041c43c6b05becb7efb82c65e6204483b32b742a84982c2f97493f8eec4064f4e53d11c5f8103983993f0d350cb50120fa60f8311936961f5e9dee5ccafaea83ed91c6eaa04a7dea0b85a38cf84d8564207ef6a85019523861474cdf47f4db8087e5368171d95cc2c1e57055a72ca39cb70420ac2f136cddd462a3df5555a94ead3b1e09d401d768a36c625fc1e9da948f90fd205350415253455f4d45524b4c455f504c414345484f4c4445525f48415348000020284c597ee0618821ea22fecfd2e6da6f3e5a451ee13870e6338e840e50a025fb20e18365cf9b9eb20b0df4a5c08e94e648d8706860fd7891ca3742667a3e55690c",
"id": 200
}

integration-test 则会一直hang住：

mpm integration-test snapshot
BUILDING StarcoinFramework

running 1 tests
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^C

baichuan3 · 2022-08-09T09:54:03Z

看之前integration-test例子里写的是halley.seed，curl里写的是halley-seed，跟这个写法有关吗？

//# init --rpc http://halley.seed.starcoin.org/ --block-number 76107
curl --location --request POST 'https://halley-seed.starcoin.org/'

jolestar · 2022-08-09T16:26:38Z

看之前integration-test例子里写的是halley.seed，curl里写的是halley-seed，跟这个写法有关吗？

//# init --rpc http://halley.seed.starcoin.org/ --block-number 76107
curl --location --request POST 'https://halley-seed.starcoin.org/'

halley-seed 是 https, halley.seed 是 http

baichuan3 · 2022-08-20T09:00:45Z

测试下来call api在某些场景下会失败，测试case如下 @jolestar ：

//# call-api chain.info

//# call-api chain.get_block_by_hash ["{{$.call-api[0].head.block_hash}}",{"raw":true}]

//# run --signers alice  --args {{$.call-api[1].raw.header}}

同样的rpc接口，将参数改为parent_hash，call api会失败。

//# call-api chain.info

//# call-api chain.get_block_by_hash ["{{$.call-api[0].head.parent_hash}}",{"raw":true}]

//# run --signers alice  --args {{$.call-api[1].raw.header}}

用上同样的参数，在命令行curl下调用chain.get_block_by_hash，传入block_hash和parent_hash，都能正常返回。

curl --location --request POST 'https://halley-seed.starcoin.org'
--header 'Content-Type: application/json'
--data-raw '{
"id":101,
"jsonrpc":"2.0",
"method":"chain.get_block_by_hash",
"params":["0x56a503f5c608a1b803fea481c069d90380d495d1535f87079b689ade2eba7f17"]
}'

完整测试case：https://github.com/starcoinorg/starcoin-framework/blob/dao_integration_test/integration-tests/block/checkpoints_test.move

pause125 requested a review from jolestar August 2, 2022 15:14

jolestar reviewed Aug 2, 2022

View reviewed changes

rpc/api/src/state/mod.rs Outdated Show resolved Hide resolved

jolestar reviewed Aug 2, 2022

View reviewed changes

vm/starcoin-transactional-test-harness/src/lib.rs Outdated Show resolved Hide resolved

vm/starcoin-transactional-test-harness/src/fork_chain.rs Outdated Show resolved Hide resolved

pause125 added 13 commits August 5, 2022 14:24

fork from rpc and call_api

35b67e6

unify fork and dev net

1a002be

add fork state

2b99deb

add state api, state db

7fb7587

clean

6296f4a

add MockChainStateAsyncService

4be649f

fix MockChainStateAsyncService

e62fd89

fix block state root

cc1e687

fix

4966f45

api rename get_state_node_by_node_hash

fe6480c

implemented chain api

fc33ba0

fix fmt

b87d393

fix

a68e949

pause125 force-pushed the rpc-fork branch from 367e4d2 to a68e949 Compare August 6, 2022 05:25

pause125 marked this pull request as ready for review August 6, 2022 05:29

pause125 requested review from nanne007 and sanlee42 as code owners August 6, 2022 05:29

jolestar approved these changes Aug 6, 2022

View reviewed changes

jolestar merged commit e526915 into starcoinorg:master Aug 6, 2022

pause125 mentioned this pull request Aug 13, 2022

Fix mpm integration-test hangs in rpc-fork mode, update test case on halley chain #3653

Merged

7 tasks

pause125 mentioned this pull request Aug 21, 2022

mpm fix parent hash #3681

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mpm integration-test support fork from remote and call-api #3600

mpm integration-test support fork from remote and call-api #3600

pause125 commented Aug 2, 2022

pause125 commented Aug 2, 2022

pause125 commented Aug 6, 2022

codecov bot commented Aug 6, 2022

jolestar left a comment

baichuan3 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

pause125 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 20, 2022

mpm integration-test support fork from remote and call-api #3600

mpm integration-test support fork from remote and call-api #3600

Conversation

pause125 commented Aug 2, 2022

Pull request type

What is the current behavior?

What is the new behavior?

Other information

pause125 commented Aug 2, 2022

pause125 commented Aug 6, 2022

codecov bot commented Aug 6, 2022

Codecov Report

jolestar left a comment

Choose a reason for hiding this comment

baichuan3 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

pause125 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

baichuan3 commented Aug 9, 2022

jolestar commented Aug 9, 2022

baichuan3 commented Aug 20, 2022