feat(zcoin): ARRR WASM implementation

[borngraced]

This PR integrates zcash wallet db in WASM. All changes has been tested really well.

Tasks Completed in this Pull Request:

• Implemented all blocksdb I/O functions for WASM including unit tests
• Refactored the light_wallet_db_synch_loop function to be fully asynchronous and removed all uses of block_on and block_in_place.
• Renamed the scan_blocks function to scan_validate_and_update_blocks and refactored it into an asynchronous function.
• Refactored the validate_chain and scan_cached_block functions to be fully asynchronous. Simplified the function processes for better manageability.
• Implementation of walletdb + unit tests for wallets I/O functions for WASM.
• Implementation of zcash_params an unit tests using indexeddb
• Implemented tonic transport layer for ARRR that works almost same way with the current native transport and also support streaming data
• completed ARRR web integration with changes to z_coin mod.

Deps Additions

• base64(0.21.7) - Base64 encoding and decoding functionality used for WASM tonic client
• tower-service(0.3.2) - Defines the Service trait for building composable network services used for WASM tonic client.
• blake2b_simd(0.5.10) - Rust implementation of the BLAKE2b cryptographic hash function for working with z_coin transactions
• ff(0.8.0) - Finite field arithmetic operations for working with z_coin transactions.
• jubjub(0.5.1) - ubjub elliptic curve cryptography for working with z_coin transactions.
• time(0.3.20, features = ["wasm-bindgen"]) - time library for Rust, with WebAssembly support for working with z_coin transactions.

Deps Updated

• proc-macro2 1.0.63 -> 1.0.69
• quote 1.0.28 -> 1.0.33
• memoffset 0.5.4 -> 0.5.6
• syn 2.0.23 -> 2.0.38
• serde_derive 1.0.164 -> 1.0.189
• serde_urlencoded 0.7.0 -> 0.7.1
• time 0.3.11 -> 0.3.20
• time-macros 0.2.4 -> 0.2.8

NOTE

Some walletdb wasm tests are commented out because of long running time.

[smk762]

@borngraced I've just encountered a previously unseen error testing on wasm:

{
  "mmrpc": "2.0",
  "result": {
    "status": "InProgress",
    "details": {
      "TemporaryError": "z_rpc:286] status: Unknown, message: \"tonic_client:45] http:340] Invalid request: InvalidRequest(\\\"JsValue(TypeError: network error\\\\nTypeError: network error)\\\")\", details: [], metadata: MetadataMap { headers: {} }"
    }
  },
  "id": null
}

Is this a concern? I'll restart and try again, just curious what might cause this.

Some context:

Seems that though some calls to the grpc are successful, a few are returning a protocol error.

After clearing cache, next attempt was as expected:

{
  "mmrpc": "2.0",
  "result": {
    "status": "InProgress",
    "details": {
      "BuildingWalletDb": {
        "first_sync_block": {
          "requested": 2794017,
          "is_pre_sapling": false,
          "actual": 2794017
        },
        "current_scanned_block": 2794017,
        "latest_block": 2795458
      }
    }
  },
  "id": null
}

[borngraced]

@borngraced I've just encountered a previously unseen error testing on wasm:

{
  "mmrpc": "2.0",
  "result": {
    "status": "InProgress",
    "details": {
      "TemporaryError": "z_rpc:286] status: Unknown, message: \"tonic_client:45] http:340] Invalid request: InvalidRequest(\\\"JsValue(TypeError: network error\\\\nTypeError: network error)\\\")\", details: [], metadata: MetadataMap { headers: {} }"
    }
  },
  "id": null
}

Is this a concern? I'll restart and try again, just curious what might cause this.

Some context:

Seems that though some calls to the grpc are successful, a few are returning a protocol error.

After clearing cache, next attempt was as expected:

{
  "mmrpc": "2.0",
  "result": {
    "status": "InProgress",
    "details": {
      "BuildingWalletDb": {
        "first_sync_block": {
          "requested": 2794017,
          "is_pre_sapling": false,
          "actual": 2794017
        },
        "current_scanned_block": 2794017,
        "latest_block": 2795458
      }
    }
  },
  "id": null
}

this is simply a network error while fetching the request prob related to the server. I don't think it's an internal issue.

Note: When requesting for block ranges, we split the request into chunks of 20k headers each and fetch concurrently. So if one request fails, all request other requests will be cancelled and we try again. also, I have only experienced network error once when I had actual internet problem.

[smk762]

Thanks for clarifying, it is as I expected. We're on day 4 of a heatwave here and it seems hot enough to melt the copper my internet arrives on, so sounds like its a "me" problem.
If replicated by others, I'll review the server to see if any optimisations are possible.

[smk762]

Unfortunately I'm getting these errors pretty consistently, likely due to my location / connection (unless replicated).
After a network failed error, subsequent attempts to activate returned

{
  "mmrpc": "2.0",
  "result": {
    "status": "Error",
    "details": {
      "error": "Error on platform coin ARRR creation: Database 'MAIN::21605444b36ec72780bdf52a5ffbc18288893664::z_compactblocks_cache' is open already",
      "error_path": "lib.z_coin_activation.z_coin.z_rpc.blockdb_idb_storage.builder",
      "error_trace": "lib:104] z_coin_activation:245] z_coin:995] z_rpc:517] blockdb_idb_storage:80] builder:166]",
      "error_type": "CoinCreationError",
      "error_data": {
        "ticker": "ARRR",
        "error": "Database 'MAIN::21605444b36ec72780bdf52a5ffbc18288893664::z_compactblocks_cache' is open already"
      }
    }
  },
  "id": null
}

and browser tab needs to be refreshed before another attempt is possible.

I checked the lightwallet server, and there was no issues in logs. Browser console logs revealed that even after mm2 returns network error, downloading continues in the background.

cnx-fail.mp4

Would it be possible to recover and continue rather than be forced to start again from the beginning? Or stopping d/l and closing db connection for second attempt? If not, perhaps an error message indicating that a reload is required before trying again.

[smk762]

Perhaps not an issue, but I noticed that the response for zhtlc coin initialization once a coin has been activated remains static, i.e. if balance changes after activation completes, the balance at time of activation is still returned from the task::enable_z_coin::status request. This also applies to the returned current_block value.

It would be nice if the status returned live values for balance and block_height, though this would likely also apply to other coins so beyond the scope of this PR (I'll convert to issue).

Regression testing for TCP ZHTLC activation with a variety of optional param combinations was otherwise successful, with no defects found and no issues sending (with/without memo) or trading ZHTLC coins (as maker and taker).

Wasm tests still in progress

[smk762]

Given my issues with connection breaks for longer scans, I focused on the date/block param tests to (hopefully) have a connection stable long enough to activate. This was successful on first try, but the second one got stuck on activating, though console seemed to suggest it has completed. If clearing cache, I was once again able to activate.
Refreshing the page without clearing cache and trying again, I was stuck on once again activating.

Video below at 3x speed to keep filesize low:

forever-activating.webm

This was the third attempt of sequential leave cache / clear cache test. Note that the sync param was changed after the first successful attempt and the failed 2nd attempts before clearing cache. When the sync param does not change between attempts, activation completes successfully.

Retesting this case on TCP, it was fine on the second activation with new sync params without needing to rm MM2.db

[borngraced]

Given my issues with connection breaks for longer scans, I focused on the date/block param tests to (hopefully) have a connection stable long enough to activate. This was successful on first try, but the second one got stuck on activating, though console seemed to suggest it has completed. If clearing cache, I was once again able to activate. Refreshing the page without clearing cache and trying again, I was stuck on once again activating.

Video below at 3x speed to keep filesize low:

forever-activating.webm

This was the third attempt of sequential leave cache / clear cache test. Note that the sync param was changed after the first successful attempt and the failed 2nd attempts before clearing cache. When the sync param does not change between attempts, activation completes successfully.

Retesting this case on TCP, it was fine on the second activation with new sync params without needing to rm MM2.db

thanks for the well organized bug report 😄 .

can you try again now, I believe it shouldn't stuck anymore when you cancel and try to re-activate coin again without refreshing or clearing cache. @smk762

[smk762]

I believe it shouldn't stuck anymore when you cancel and try to re-activate coin again without refreshing or clearing cache.

Great work, thanks! Confirm I no longer faced issues attempting to activate zhtlc with wasm after changing sync params and not clearing cache.

[smk762]

Unfortunately I've been unable to perform withdrawals with ZHTLC coins in WASM.
In CLI, the https://komodo-docs-revamp-2023.pages.dev/en/docs/atomicdex/api/v20-dev/task_withdraw/ methods are used, but in wasm, this returns

{
  "mmrpc": "2.0",
  "result": {
    "status": "Error",
    "details": {
      "error": "'ARRR' coin doesn't support 'init_withdraw' yet. Consider using 'withdraw' request instead",
      "error_path": "init_withdraw",
      "error_trace": "init_withdraw:137]",
      "error_type": "CoinDoesntSupportInitWithdraw",
      "error_data": {
        "coin": "ARRR"
      }
    }
  },
  "id": 0
}

I also tried the v2 withdraw method, which returned

{
  "mmrpc": "2.0",
  "error": "Internal error: Zcoin doesn't support legacy withdraw",
  "error_path": "z_coin",
  "error_trace": "z_coin:1737]",
  "error_type": "InternalError",
  "error_data": "Zcoin doesn't support legacy withdraw",
  "id": 0
}

and the v1 withdraw method returns

{
  "error": "rpc:156] rpc:215] dispatcher_legacy:141] dispatcher_legacy:151] z_coin:1737] Internal error: Zcoin doesn't support legacy withdraw"
}

[borngraced]

Unfortunately I've been unable to perform withdrawals with ZHTLC coins in WASM. In CLI, the https://komodo-docs-revamp-2023.pages.dev/en/docs/atomicdex/api/v20-dev/task_withdraw/ methods are used, but in wasm, this returns

{
  "mmrpc": "2.0",
  "result": {
    "status": "Error",
    "details": {
      "error": "'ARRR' coin doesn't support 'init_withdraw' yet. Consider using 'withdraw' request instead",
      "error_path": "init_withdraw",
      "error_trace": "init_withdraw:137]",
      "error_type": "CoinDoesntSupportInitWithdraw",
      "error_data": {
        "coin": "ARRR"
      }
    }
  },
  "id": 0
}

I also tried the v2 withdraw method, which returned

{
  "mmrpc": "2.0",
  "error": "Internal error: Zcoin doesn't support legacy withdraw",
  "error_path": "z_coin",
  "error_trace": "z_coin:1737]",
  "error_type": "InternalError",
  "error_data": "Zcoin doesn't support legacy withdraw",
  "id": 0
}

and the v1 withdraw method returns

{
  "error": "rpc:156] rpc:215] dispatcher_legacy:141] dispatcher_legacy:151] z_coin:1737] Internal error: Zcoin doesn't support legacy withdraw"
}

@smk762, after the build is complete, you can use the rpc commands below to initiate a withdrawal
task::withdraw::init
task::withdraw::status
task::withdraw::cancel

[smk762]

Attempting to place a sell (in Firefox) order via wasm between DOC and ARR failed, with the following error seen in console
common:480] panicked at 'could not initialize thread_rng: All entropy sources failed (permanently unavailable); cause: OsRng: support for wasm32 requires emscripten, stdweb or wasm-bindgen (permanently unavailable)', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.6.5/src/rngs/thread.rs:80:17

Replicated with same result in opera and brave.

Placing a buy order worked, but using set_price returned the same error as above

[smk762]

I loaded up b28e3fa in wasm playground and tried to withdraw again, but now it is (incorrectly) telling me I'm too poor.

cant-spend.mp4

[borngraced]

I loaded up b28e3fa in wasm playground and tried to withdraw again, but now it is (incorrectly) telling me I'm too poor.

cant-spend.mp4

@smk762 withdraw has been enabled and tested, you can try withdrawing now without any issue
NOTE: It's a known issue for long(not too long) execution time when you send a request that involves generating tx like withdrawing #2000 (comment)

Attempting to place a sell (in Firefox) order via wasm between DOC and ARR failed, with the following error seen in console
common:480] panicked at 'could not initialize thread_rng: All entropy sources failed (permanently unavailable); cause: OsRng: support for wasm32 requires emscripten, stdweb or wasm-bindgen (permanently unavailable)', /github/home/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.6.5/src/rngs/thread.rs:80:17

This has been resolved too, you can now place sell or setprice orders..I made couple successful trades so it should work!.

[smk762]

Confirming withdraw is functioning as expected.
Also completed swaps using setprice, buy and sell methods. During swaps, at times the tab would freeze for a bit, I assume while generating the required transactions. cc: @CharlVS this is something we should keep an eye on once adding zhtlc to the web wallet.

Thanks for your efforts @borngraced on this great feature!