mm2 tries to fetch 2M block headers from electrum

[cipig]

mm2 tries to request 2M block headers and gets error from electrumx servers because response is too large (over 2M bytes)

26 18:55:32, coins:utxo_arc_builder:261] ERROR Error rpc_clients:1915] JsonRpcError { client_info: "coin: CRYPTO", request: JsonRpcRequest { jsonrpc: "2.0", id: "506", method: "blockchain.block.headers", params: [Number(1), Number(2062652)] }, error: Response(electrum2.cipig.net:10008, Object({"code": Number(-32600), "message": String("response too large (over 2,000,000 bytes")})) } on retrieving the latest headers from rpc!

the doc for the blockchain.block.headers call: https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html#blockchain-block-headers

[shamardy]

@borngraced this happens only for coins with "enable_spv_proof": true in their config due to trying to retrieve all the headers at once here https://github.com/KomodoPlatform/atomicDEX-API/blob/5dbc5de4f40ad93e93b08a6ac63605bd51def5b9/mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs#L255
Note: this doesn't happen with BTC/tBTC as the electrum server sends only 2016 block headers at a time for BTC, I suspect the electrums for BTC use a different electrum code that still have MAX_CHUNK_SIZE = 2016. We can limit the number of blocks requested to 2016 for now in mm2, if any other coin needed more than that for block difficulty calculations in the future we can make this a parameter in the coin config.

[borngraced]

@borngraced this happens only for coins with "enable_spv_proof": true in their config due to trying to retrieve all the headers at once here

https://github.com/KomodoPlatform/atomicDEX-API/blob/5dbc5de4f40ad93e93b08a6ac63605bd51def5b9/mm2src/coins/utxo/utxo_builder/utxo_arc_builder.rs#L255

Note: this doesn't happen with BTC/tBTC as the electrum server sends only 2016 block headers at a time for BTC, I suspect the electrums for BTC use a different electrum code that still have MAX_CHUNK_SIZE = 2016. We can limit the number of blocks requested to 2016 for now in mm2, if any other coin needed more than that for block difficulty calculations in the future we can make this a parameter in the coin config.

ok cool thanks for the info!

[sergeyboyko0791]

@cipig could you please check if this error has gone?

[cipig]

looks good... i reenabled spv_proof for KMD and it works
but i get this errors on one node

18 12:36:02, coins:utxo_arc_builder:290] ERROR Error rpc_clients:1966] Internal("No headers available") on retrieving the latest headers from rpc! 2 attempts left

it shows up every 10s, but doesn't say which coin

[sergeyboyko0791]

@borngraced could you please check what's going on?

[borngraced]

@borngraced could you please check what's going on?

I guess this behaviour is expected as we are suppose to stop the requests if there's no more headers to fetch hence we call notify on permanent error sync_status_loop_handle.notify_on_permanent_error(error.to_string()) but now we are trying to attempts the requests again in case of different errors but now we didn't know the error is going to beNo more headers available. and I assume the full headers for KMD coin has been fetched for that node.

perhaps I need to check for this kind of error specifically and avoid attempting to refetch the headers

[shamardy]

I guess this behaviour is expected as we are suppose to stop the requests if there's no more headers to fetch hence we call notify on permanent error sync_status_loop_handle.notify_on_permanent_error(error.to_string()) but now we are trying to attempts the requests again in case of different errors but now we didn't know the error is going to beNo more headers available. and I assume the full headers for KMD coin has been fetched for that node.

We need to continue fetching headers because new headers appear onchain and we need these new headers to validate transactions against them, that's why we spawn a loop to continue fetching new headers. notify_on_permanent_error is meant only for when an error happens when activating the coin you can see that this notification is received only here https://github.com/KomodoPlatform/atomicDEX-API/blob/4de3cf2b24b914935b0d3f7364676b38275084fe/mm2src/coins_activation/src/utxo_activation/init_utxo_standard_activation.rs#L100

[shamardy]

I believe the attempts shouldn't be reduced if the error is No headers available, so we need to filter for this error too if we are gonna have a number of attempts. Also when No headers available appears we should call notify_sync_finished to continue the coin activation. I see there is a PR to fix this I will have a look at it @borngraced :)

[borngraced]

I believe the attempts shouldn't be reduced if the error is No headers available, so we need to filter for this error too if we are gonna have a number of attempts. Also when No headers available appears we should call notify_sync_finished to continue the coin activation. I see there is a PR to fix this I will have a look at it @borngraced :)

yes you should review it now.

[borngraced]

@cipig do you have an update on this? pls...

[cipig]

used latest mm2 from dev, enabled spv_proof for KMD by setting "enable_spv_proof": true in coins file
swap started, mm2 shows error 09 23:34:41, coins:spv:46] ERROR Failed spv proof validation for transaction e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60 with error: rpc_clients:2070] rpc_clients:2002] InvalidHeight("Transaction block header is not found in storage"), retrying in 10 seconds. every 10s, but tx has 32 confirmations: https://kmdexplorer.io/tx/e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60

swap is basically stuck at on maker side

      {
         "event" : {
            "type" : "TakerPaymentReceived",
            "data" : {
               "tx_hash" : "e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60",
               "tx_hex" : "0400008085202f8902e4886edf3fa60d16bb7afd97dfd604b242233c49be009de9beb6a42884c80c73010000006a4730440220363780c0260ff22c17e422cf8d90b19554f80a0195e4c28a3753eceb5cafa6f1022023149bf061c92873489d4f5471304c2a8a7d8dbb43f99e0ab02e07625ae55b3f0121023c5ba1d7ef6fa015eb33defb3aba2a961898a51bbb7ff30344d07ba75ad3f289ffffffff1e7ea07ca95d608e954917c18b119ffca3c5cceea53a7606d730fc2f6823c43d020000006b4830450221009c9658aff7d3e818a8ce44692414eca04638c7edd4b86a03d4e3db3ffb0982d7022070bf1559c5a72e522e70d102ae4845774622cde824d32afe84e30dba679fd6660121023c5ba1d7ef6fa015eb33defb3aba2a961898a51bbb7ff30344d07ba75ad3f289ffffffff03db916e660000000017a91493ec9093cdf6ca3dc967e26d141bf5879f71f3c0870000000000000000166a14b6f138f49c3eafded95ba35ba2b7f8963a37293be066fb09000000001976a91484c74592ed8ac05340906784d277ca4d4e0af08e88ac3cb69363000000000000000000000000000000"
            }
         },
         "timestamp" : 1670626875124
      },
      {
         "timestamp" : 1670626875129,
         "event" : {
            "type" : "TakerPaymentWaitConfirmStarted"
         }
      }

the initial problem with response too large error from electrumx is gone, so maybe the above is actually a separate issue

[borngraced]

used latest mm2 from dev, enabled spv_proof for KMD by setting "enable_spv_proof": true in coins file swap started, mm2 shows error 09 23:34:41, coins:spv:46] ERROR Failed spv proof validation for transaction e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60 with error: rpc_clients:2070] rpc_clients:2002] InvalidHeight("Transaction block header is not found in storage"), retrying in 10 seconds. every 10s, but tx has 32 confirmations: https://kmdexplorer.io/tx/e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60

swap is basically stuck at on maker side

      {
         "event" : {
            "type" : "TakerPaymentReceived",
            "data" : {
               "tx_hash" : "e14d1a6e109103b57ed1f8ff1843d64a35b37641882b0c07bea8156c1357fd60",
               "tx_hex" : "0400008085202f8902e4886edf3fa60d16bb7afd97dfd604b242233c49be009de9beb6a42884c80c73010000006a4730440220363780c0260ff22c17e422cf8d90b19554f80a0195e4c28a3753eceb5cafa6f1022023149bf061c92873489d4f5471304c2a8a7d8dbb43f99e0ab02e07625ae55b3f0121023c5ba1d7ef6fa015eb33defb3aba2a961898a51bbb7ff30344d07ba75ad3f289ffffffff1e7ea07ca95d608e954917c18b119ffca3c5cceea53a7606d730fc2f6823c43d020000006b4830450221009c9658aff7d3e818a8ce44692414eca04638c7edd4b86a03d4e3db3ffb0982d7022070bf1559c5a72e522e70d102ae4845774622cde824d32afe84e30dba679fd6660121023c5ba1d7ef6fa015eb33defb3aba2a961898a51bbb7ff30344d07ba75ad3f289ffffffff03db916e660000000017a91493ec9093cdf6ca3dc967e26d141bf5879f71f3c0870000000000000000166a14b6f138f49c3eafded95ba35ba2b7f8963a37293be066fb09000000001976a91484c74592ed8ac05340906784d277ca4d4e0af08e88ac3cb69363000000000000000000000000000000"
            }
         },
         "timestamp" : 1670626875124
      },
      {
         "timestamp" : 1670626875129,
         "event" : {
            "type" : "TakerPaymentWaitConfirmStarted"
         }
      }

the initial problem with response too large error from electrumx is gone, so maybe the above is actually a separate issue

thanks, I'm now looking into this.