new rpc-client for JD-Server mempool #761
Conversation
ec5aec9 to
7bec385
Compare
|
| Report | Tue, February 20, 2024 at 18:16:30 UTC |
| Project | Stratum v2 (SRI) |
| Branch | jd-server-mini-rpc-client |
| Testbed | sv1 |
| Benchmark | Latency | Latency Results nanoseconds (ns) | (Δ%) | Latency Upper Boundary nanoseconds (ns) | (%) |
|---|---|---|---|
| client-submit-serialize | ✅ (view plot) | 6839.500 (+0.92%) | 7138.100 (95.82%) |
| client-submit-serialize-deserialize | ✅ (view plot) | 7898.300 (+2.11%) | 8233.916 (95.92%) |
| client-submit-serialize-deserialize-handle/client-submit-serialize-deserialize-handle | ✅ (view plot) | 8350.900 (+0.58%) | 8748.024 (95.46%) |
| client-sv1-authorize-serialize-deserialize-handle/client-sv1-authorize-serialize-deserialize-handle | ✅ (view plot) | 888.770 (-1.35%) | 939.247 (94.63%) |
| client-sv1-authorize-serialize-deserialize/client-sv1-authorize-serialize-deserialize | ✅ (view plot) | 689.260 (-1.42%) | 723.029 (95.33%) |
| client-sv1-authorize-serialize/client-sv1-authorize-serialize | ✅ (view plot) | 246.260 (+0.17%) | 249.034 (98.89%) |
| client-sv1-get-authorize/client-sv1-get-authorize | ✅ (view plot) | 158.140 (+0.64%) | 158.963 (99.48%) |
| client-sv1-get-submit | ✅ (view plot) | 6748.500 (+2.36%) | 6961.398 (96.94%) |
| client-sv1-get-subscribe/client-sv1-get-subscribe | ✅ (view plot) | 283.310 (+0.25%) | 293.494 (96.53%) |
| client-sv1-subscribe-serialize-deserialize-handle/client-sv1-subscribe-serialize-deserialize-handle | ✅ (view plot) | 741.360 (-1.24%) | 783.142 (94.66%) |
| client-sv1-subscribe-serialize-deserialize/client-sv1-subscribe-serialize-deserialize | ✅ (view plot) | 610.800 (-0.38%) | 631.136 (96.78%) |
| client-sv1-subscribe-serialize/client-sv1-subscribe-serialize | ✅ (view plot) | 218.020 (+4.75%) | 220.591 (98.83%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
| Report | Mon, March 4, 2024 at 12:44:20 UTC |
| Project | Stratum v2 (SRI) |
| Branch | jd-server-mini-rpc-client |
| Testbed | sv1 |
| Benchmark | Estimated Cycles | Estimated Cycles Results estimated cycles | (Δ%) | Estimated Cycles Upper Boundary estimated cycles | (%) | Instructions | Instructions Results instructions | (Δ%) | Instructions Upper Boundary instructions | (%) | L1 Accesses | L1 Accesses Results accesses | (Δ%) | L1 Accesses Upper Boundary accesses | (%) | L2 Accesses | L2 Accesses Results accesses | (Δ%) | L2 Accesses Upper Boundary accesses | (%) | RAM Accesses | RAM Accesses Results accesses | (Δ%) | RAM Accesses Upper Boundary accesses | (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| get_authorize | ✅ (view plot) | 8432.000 (-0.50%) | 8554.979 (98.56%) | ✅ (view plot) | 3746.000 (-0.04%) | 3761.117 (99.60%) | ✅ (view plot) | 5252.000 (-0.01%) | 5275.764 (99.55%) | ✅ (view plot) | 6.000 (-16.88%) | 9.205 (65.18%) | ✅ (view plot) | 90.000 (-1.13%) | 93.071 (96.70%) |
| get_submit | ✅ (view plot) | 95499.000 (-0.10%) | 95699.581 (99.79%) | ✅ (view plot) | 59439.000 (-0.01%) | 59487.259 (99.92%) | ✅ (view plot) | 85354.000 (-0.01%) | 85433.826 (99.91%) | ✅ (view plot) | 55.000 (-0.28%) | 57.895 (95.00%) | ✅ (view plot) | 282.000 (-0.86%) | 287.370 (98.13%) |
| get_subscribe | ✅ (view plot) | 7999.000 (-0.27%) | 8128.228 (98.41%) | ✅ (view plot) | 2841.000 (-0.02%) | 2845.070 (99.86%) | ✅ (view plot) | 3969.000 (-0.01%) | 3975.943 (99.83%) | ✅ (view plot) | 15.000 (+1.05%) | 16.709 (89.77%) | ✅ (view plot) | 113.000 (-0.55%) | 116.645 (96.88%) |
| serialize_authorize | ✅ (view plot) | 12147.000 (-0.58%) | 12348.176 (98.37%) | ✅ (view plot) | 5317.000 (-0.03%) | 5332.117 (99.72%) | ✅ (view plot) | 7417.000 (+0.01%) | 7438.790 (99.71%) | ✅ (view plot) | 8.000 (-18.73%) | 12.582 (63.58%) | ✅ (view plot) | 134.000 (-1.31%) | 139.117 (96.32%) |
| serialize_deserialize_authorize | ✅ (view plot) | 24464.000 (-0.13%) | 24609.153 (99.41%) | ✅ (view plot) | 9898.000 (-0.03%) | 9928.235 (99.70%) | ✅ (view plot) | 13959.000 (-0.03%) | 14012.865 (99.62%) | ✅ (view plot) | 36.000 (-0.95%) | 40.003 (89.99%) | ✅ (view plot) | 295.000 (-0.24%) | 298.809 (98.73%) |
| serialize_deserialize_handle_authorize | ✅ (view plot) | 30177.000 (-0.11%) | 30310.878 (99.56%) | ✅ (view plot) | 12101.000 (-0.01%) | 12116.117 (99.88%) | ✅ (view plot) | 17117.000 (-0.02%) | 17146.587 (99.83%) | ✅ (view plot) | 64.000 (+2.14%) | 66.393 (96.40%) | ✅ (view plot) | 364.000 (-0.28%) | 368.269 (98.84%) |
| serialize_deserialize_handle_submit | ✅ (view plot) | 126390.000 (-0.05%) | 126568.737 (99.86%) | ✅ (view plot) | 73224.000 (-0.01%) | 73272.259 (99.93%) | ✅ (view plot) | 104950.000 (-0.00%) | 105029.716 (99.92%) | ✅ (view plot) | 116.000 (-3.96%) | 127.117 (91.25%) | ✅ (view plot) | 596.000 (-0.19%) | 600.790 (99.20%) |
| serialize_deserialize_handle_subscribe | ✅ (view plot) | 27519.000 (-0.00%) | 27632.412 (99.59%) | ✅ (view plot) | 9643.000 (-0.00%) | 9647.070 (99.96%) | ✅ (view plot) | 13634.000 (-0.01%) | 13643.878 (99.93%) | ✅ (view plot) | 68.000 (+1.68%) | 69.798 (97.42%) | ✅ (view plot) | 387.000 (-0.03%) | 390.504 (99.10%) |
| serialize_deserialize_submit | ✅ (view plot) | 115011.000 (-0.03%) | 115190.544 (99.84%) | ✅ (view plot) | 68001.000 (-0.02%) | 68097.519 (99.86%) | ✅ (view plot) | 97556.000 (-0.02%) | 97720.234 (99.83%) | ✅ (view plot) | 68.000 (+0.28%) | 70.958 (95.83%) | ✅ (view plot) | 489.000 (-0.08%) | 492.616 (99.27%) |
| serialize_deserialize_subscribe | ✅ (view plot) | 22896.000 (-0.05%) | 23014.337 (99.49%) | ✅ (view plot) | 8195.000 (-0.01%) | 8203.140 (99.90%) | ✅ (view plot) | 11541.000 (-0.01%) | 11557.242 (99.86%) | ✅ (view plot) | 38.000 (-0.33%) | 43.531 (87.29%) | ✅ (view plot) | 319.000 (-0.09%) | 322.371 (98.95%) |
| serialize_submit | ✅ (view plot) | 99786.000 (-0.11%) | 100049.347 (99.74%) | ✅ (view plot) | 61483.000 (-0.01%) | 61531.259 (99.92%) | ✅ (view plot) | 88201.000 (-0.01%) | 88279.787 (99.91%) | ✅ (view plot) | 56.000 (+1.30%) | 57.865 (96.78%) | ✅ (view plot) | 323.000 (-0.98%) | 329.720 (97.96%) |
| serialize_subscribe | ✅ (view plot) | 11298.000 (-0.39%) | 11480.619 (98.41%) | ✅ (view plot) | 4188.000 (-0.01%) | 4192.070 (99.90%) | ✅ (view plot) | 5828.000 (-0.01%) | 5834.225 (99.89%) | ✅ (view plot) | 16.000 (+6.44%) | 16.592 (96.43%) | ✅ (view plot) | 154.000 (-0.88%) | 159.341 (96.65%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
| Report | Mon, March 4, 2024 at 12:44:23 UTC |
| Project | Stratum v2 (SRI) |
| Branch | jd-server-mini-rpc-client |
| Testbed | sv2 |
| Benchmark | Latency | Latency Results nanoseconds (ns) | (Δ%) | Latency Upper Boundary nanoseconds (ns) | (%) |
|---|---|---|---|
| client_sv2_handle_message_common | ✅ (view plot) | 44.168 (+0.01%) | 44.812 (98.56%) |
| client_sv2_handle_message_mining | ✅ (view plot) | 71.737 (+7.28%) | 72.063 (99.55%) |
| client_sv2_mining_message_submit_standard | ✅ (view plot) | 14.661 (-0.00%) | 14.703 (99.72%) |
| client_sv2_mining_message_submit_standard_serialize | ✅ (view plot) | 274.120 (+2.70%) | 286.732 (95.60%) |
| client_sv2_mining_message_submit_standard_serialize_deserialize | ✅ (view plot) | 582.280 (-0.59%) | 619.402 (94.01%) |
| client_sv2_open_channel | ✅ (view plot) | 169.290 (+1.13%) | 176.253 (96.05%) |
| client_sv2_open_channel_serialize | ✅ (view plot) | 294.500 (+3.62%) | 297.406 (99.02%) |
| client_sv2_open_channel_serialize_deserialize | ✅ (view plot) | 392.190 (+4.53%) | 399.880 (98.08%) |
| client_sv2_setup_connection | ✅ (view plot) | 164.120 (-0.54%) | 179.033 (91.67%) |
| client_sv2_setup_connection_serialize | ✅ (view plot) | 459.510 (-3.30%) | 505.724 (90.86%) |
| client_sv2_setup_connection_serialize_deserialize | ✅ (view plot) | 982.170 (+0.52%) | 1032.807 (95.10%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
|
| Report | Mon, March 4, 2024 at 12:44:20 UTC |
| Project | Stratum v2 (SRI) |
| Branch | jd-server-mini-rpc-client |
| Testbed | sv2 |
| Benchmark | Estimated Cycles | Estimated Cycles Results estimated cycles | (Δ%) | Estimated Cycles Upper Boundary estimated cycles | (%) | Instructions | Instructions Results instructions | (Δ%) | Instructions Upper Boundary instructions | (%) | L1 Accesses | L1 Accesses Results accesses | (Δ%) | L1 Accesses Upper Boundary accesses | (%) | L2 Accesses | L2 Accesses Results accesses | (Δ%) | L2 Accesses Upper Boundary accesses | (%) | RAM Accesses | RAM Accesses Results accesses | (Δ%) | RAM Accesses Upper Boundary accesses | (%) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| client_sv2_handle_message_common | ✅ (view plot) | 2090.000 (+2.54%) | 2106.431 (99.22%) | ✅ (view plot) | 469.000 (+0.19%) | 478.569 (98.00%) | ✅ (view plot) | 730.000 (+0.15%) | 745.809 (97.88%) | ✅ (view plot) | 6.000 (-5.88%) | 8.875 (67.61%) | ✅ (view plot) | 38.000 (+4.11%) | 38.357 (99.07%) |
| client_sv2_handle_message_mining | ✅ (view plot) | 8198.000 (+1.11%) | 8248.124 (99.39%) | 🚨 (view plot | view alert) | 2133.000 (+2.59%) | 2109.208 (101.13%) | 🚨 (view plot | view alert) | 3158.000 (+2.99%) | 3116.861 (101.32%) | ✅ (view plot) | 35.000 (-18.72%) | 47.212 (74.13%) | ✅ (view plot) | 139.000 (+0.79%) | 141.858 (97.99%) |
| client_sv2_mining_message_submit_standard | ✅ (view plot) | 6318.000 (-1.61%) | 6555.474 (96.38%) | ✅ (view plot) | 1750.000 (+0.02%) | 1758.599 (99.51%) | ✅ (view plot) | 2553.000 (+0.27%) | 2560.009 (99.73%) | ✅ (view plot) | 18.000 (-15.91%) | 24.660 (72.99%) | ✅ (view plot) | 105.000 (-2.47%) | 111.724 (93.98%) |
| client_sv2_mining_message_submit_standard_serialize | ✅ (view plot) | 14637.000 (-0.71%) | 14870.140 (98.43%) | ✅ (view plot) | 4694.000 (+0.01%) | 4702.599 (99.82%) | ✅ (view plot) | 6757.000 (+0.09%) | 6764.634 (99.89%) | ✅ (view plot) | 50.000 (-4.42%) | 57.381 (87.14%) | ✅ (view plot) | 218.000 (-1.27%) | 224.425 (97.14%) |
| client_sv2_mining_message_submit_standard_serialize_deserialize | ✅ (view plot) | 27218.000 (-0.81%) | 27630.184 (98.51%) | ✅ (view plot) | 10537.000 (+0.00%) | 10545.599 (99.92%) | ✅ (view plot) | 15338.000 (+0.07%) | 15341.495 (99.98%) | ✅ (view plot) | 87.000 (-4.89%) | 97.649 (89.09%) | ✅ (view plot) | 327.000 (-1.81%) | 338.213 (96.68%) |
| client_sv2_open_channel | ✅ (view plot) | 4497.000 (-1.37%) | 4675.999 (96.17%) | ✅ (view plot) | 1461.000 (+0.06%) | 1470.569 (99.35%) | ✅ (view plot) | 2152.000 (+0.08%) | 2163.169 (99.48%) | ✅ (view plot) | 14.000 (+12.85%) | 15.874 (88.20%) | ✅ (view plot) | 65.000 (-3.08%) | 70.044 (92.80%) |
| client_sv2_open_channel_serialize | ✅ (view plot) | 14104.000 (-0.23%) | 14267.961 (98.85%) | ✅ (view plot) | 5064.000 (+0.02%) | 5073.569 (99.81%) | ✅ (view plot) | 7319.000 (+0.02%) | 7331.876 (99.82%) | ✅ (view plot) | 41.000 (+1.86%) | 42.647 (96.14%) | ✅ (view plot) | 188.000 (-0.58%) | 193.009 (97.40%) |
| client_sv2_open_channel_serialize_deserialize | ✅ (view plot) | 22448.000 (-0.55%) | 22768.070 (98.59%) | ✅ (view plot) | 7979.000 (+0.01%) | 7988.569 (99.88%) | ✅ (view plot) | 11613.000 (+0.04%) | 11626.175 (99.89%) | ✅ (view plot) | 74.000 (+1.20%) | 78.202 (94.63%) | ✅ (view plot) | 299.000 (-1.26%) | 308.496 (96.92%) |
| client_sv2_setup_connection | ✅ (view plot) | 4783.000 (+1.41%) | 4809.296 (99.45%) | ✅ (view plot) | 1502.000 (+0.06%) | 1511.569 (99.37%) | ✅ (view plot) | 2273.000 (-0.03%) | 2290.652 (99.23%) | ✅ (view plot) | 12.000 (+0.52%) | 14.009 (85.66%) | ✅ (view plot) | 70.000 (+2.80%) | 70.684 (99.03%) |
| client_sv2_setup_connection_serialize | ✅ (view plot) | 16160.000 (+0.26%) | 16198.379 (99.76%) | ✅ (view plot) | 5963.000 (+0.01%) | 5972.569 (99.84%) | ✅ (view plot) | 8660.000 (+0.02%) | 8674.288 (99.84%) | ✅ (view plot) | 44.000 (-3.16%) | 49.300 (89.25%) | ✅ (view plot) | 208.000 (+0.65%) | 209.315 (99.37%) |
| client_sv2_setup_connection_serialize_deserialize | ✅ (view plot) | 35432.000 (+0.03%) | 35515.944 (99.76%) | ✅ (view plot) | 14806.000 (+0.01%) | 14815.569 (99.94%) | ✅ (view plot) | 21757.000 (+0.04%) | 21764.423 (99.97%) | ✅ (view plot) | 89.000 (-8.07%) | 104.111 (85.49%) | ✅ (view plot) | 378.000 (+0.32%) | 379.568 (99.59%) |
Bencher - Continuous Benchmarking
View Public Perf Page
Docs | Repo | Chat | Help
There was a problem hiding this comment.
I tested this (on top of #752) against the TP on a custom Signet (using -debug=rpc).
I saw successful calls to getrawmempool.
I created a transaction to ensure that getrawtransaction is called.
I waited for it to find a block, which causes submitblock to be called (on one of the nodes, the other one got the solution via the SubmitSolution sv2 message).
I did not look at the code.
7bec385 to
3145739
Compare
|
why not use |
There was a problem hiding this comment.
SRI has been criticized about our lack of discipline in the way we leave TODO comments on the code
for example: #723 (comment)
I understand the notion of PR atomicity and I fully agree that sometimes there are things that maybe fall out of the scope of some specific PR, so I don't think writing TODO comments on a PR is a cardinal sin.
However, I believe we can raise the bar on how we manage these kind of practices. My suggestion is: for every TODO comment, open an issue with a meaningful description, and paste the link to the issue along with the TODO comment.
That way, future tasks are still manageable in our roadmap and external eyes have a little bit of context on how the code is evolving.
Because it imported the json-rpc library, which in turn imported serde in a way that it is not compatible with the way it is imported ion the SRI. Moreover, import bitcoincore rpc or just using vendored json-rpc is worse from a performance point of view |
Completely agree with that. I must say that the todo that I left are very marginal, by the way. |
af76e26 to
798ec79
Compare
GitGab19
left a comment
There was a problem hiding this comment.
Review done.
I left some questions to clarify some doubts I have (especially in the submit_solution_handler part)
There was a problem hiding this comment.
I ran it and I got:
Running fmt on: benches
error: toolchain 'nightly-x86_64-apple-darwin' is not installed
Fmt failed in: benches
Why do you use +nightly ?
There was a problem hiding this comment.
we need nightly to run cargo fmt
There was a problem hiding this comment.
we need nightly to run cargo fmt
Why? I always ran it locally with simply cargo fmt
There was a problem hiding this comment.
this is very strange in the config file we have:
imports_indent = "Block"
imports_layout = "Mixed"
imports_granularity = "Crate"that use unstable features available only on nighlty
9eb85a8 to
0ee5082
Compare
192f441 to
0d9b165
Compare
fd68321 to
014ad86
Compare
utils/rpc/src/mini_rpc_client.rs
Outdated
| // TODO | ||
| // - WHEN UPGRADING TO 1.0.1 Client is in hyper-utils: | ||
| // Struct hyper_util::client::legacy::Client | ||
| // - use https for security reasons |
There was a problem hiding this comment.
The Bitcoin Core RPC does not support https. You should connect on the same machine (or via a secure tunnel). There's work in progress to support unix domain sockets: bitcoin/bitcoin#27375 (for now I think that's limited to p2p, not rpc)
There was a problem hiding this comment.
Thank you for pointing this out!
| } | ||
| let merkle_root = | ||
| merkle_root_from_path(&coinbase_pre[..], &coinbase_suf[..], &extranonce[..], &path) | ||
| .expect("Invalid coinbase"); |
There was a problem hiding this comment.
maybe you should use try_from instead of panicing
There was a problem hiding this comment.
you are right but before I must make the code work. I leave this conversation open. When works I will go back and fix here!
A new bitcoin rpc custom rpc client is added. Before at the end of the trait method implementation handle_submit_solution in jd-server/src/lib/job_declarator/message_handlers.rs there was the part of the code responsible for submitting the block to the bitcoin node via rpc. This was a blocker. Now when submit_solution arrives, it is sent though a channel to the mempool, that submits it to the node. This task is managed asychronically in the main, so that we don't have to wait that the rpc client submits the block to the bitcoincore node. The error management of JdsMempoolError enum is moved in mempool::error module whenever possible in order to keep main clean.
old vendored json-rpc removed last fixes
- moved mini rpc client in /utils
- removed unused dependencies in the jds-server
- the channel (submit_solution_sender, submit_solution_receiver) =
unbounded() is now bounded but there is no management of when the
limit is received
- new method of JorDeclaretorDownstream (server) that take submit
solution and calculates the block hex. This method relies on a new
function in roles-logic-sv2::utils::submit_solution_to_block() that
calculates the block given the list of transactions, the
DeclareMiningJob and the SubmitSolution
- better error management in jds-server main
apply suggestions of the second round of review. - removed unnecessary Arc<Mutex<>> - in the handler implementation for the jds, when a submit_block is received, returns a SendTo::None(Some(SubmitSolution)). Accordingly changes on JobDeclaratorDownstream:::Start - close connection if a message SubmitSolution is sent but there is no job declared in the downstream - frunction roles-logic::utils::submit_solution_to_block is sobstituted with the implementation of From on a wrapper struct - removed catch-all arms in pattern match of handle_message_job_declaration in JobDeclaratorDownstream::start - removed explicit dependancy of library bytes, now relies on hyper::body::Bytes (that uses the same library under the hood, btw)
removed comment and /benches/Cargo.lock
5da7bf5 to
0a99214
Compare
Since the task that listen for new blocks is now performed independently in main, if the url rpc config is the jds is left empty the mempool the method `get_client` called in `JDsMemppol::on_submit()` returns `NoClient` error. Since we do not want rpc connections in message generator, the url config is left empty, resulting in a lot of `NoCLient` errors in the JD-server, which led the MG tests involving jd-server to fail. Also changed some variables name to make them more intuitive
lorbax
left a comment
There was a problem hiding this comment.
addressed this suggestion by @plebhash:
#782
This PR we through three rounds of reviews, that are visible in the commit history
This is a new and rebased version of PR716
What is in this PR.
A new Rpc client, called MiniRpcClient. This client communicates with a bitcoin node, and the requests that can do are just the following: ask for the mempool, ask for a specific transaction, submit a bitcoin block. This client is used to make the jd-server mempool communicate with the bitcoin node. The previous client was depending upon json-rpc library, which caused some issues because imported serde in a way that the SRI "doesn't like". This rpc client is made on top of the latest hyper 1.1.0 library for http connections.
The code is asynchronous and non blocking. Before was "almost" non blocking, as the function "submit_block" that concerned submitting a block to the bitcoin node, was blocking.
I also took advantage of introduce some proper error management in the jds mempool module.
This PR has been tested in regtest and needs to be tested in testnet.