Upgrade to `rust-lightning` 0.0.116 and bump `rust-dlc` #1482

bonomat · 2023-10-13T14:31:06Z

~~As far as we can tell, positions opened before this update will not be loaded from storage. It is therefore recommended to close all positions before upgrading to this patch.~~

Some notes:

We've had to increase the payment amounts in the multi_hop_payment and just_in_time_channel_with_multiple_payments tests, as otherwise we would run into route not found errors. It's probably worth investigating why this is the case.
We have decided to eagerly send rust-dlc messages because: (1) after the upgrade they were not being automatically sent as fast as they used to and (2) we can speed up the protocols this way anyway.

holzeis

LGMT 🚀

We might want to schedule the release of this though 🙂

crates/ln-dlc-node/src/node/mod.rs

...le/native/native.xcodeproj/xcuserdata/bonomat.xcuserdatad/xcschemes/xcschememanagement.plist

CHANGELOG.md

crates/ln-dlc-node/src/dlc_custom_signer.rs

crates/ln-dlc-node/src/fee_rate_estimator.rs

crates/ln-dlc-node/src/ln/app_event_handler.rs

crates/ln-dlc-node/src/ln/coordinator_event_handler.rs

crates/ln-dlc-node/src/node/mod.rs

maker/src/ln/event_handler.rs

bonomat · 2023-10-19T14:38:29Z

@luckysori : feel free to grab it tomorrow and continue on it. It looks like some of the lnd_dlc tests are now failing.

luckysori · 2023-11-01T14:30:22Z

@bonomat, @holzeis: Please have another look at this. There was a slightly tricky rebase in between which forced me to touch the collaborative revert functionality that was recently introduced.

I've tested this a few times locally:

Open channel + position on 0.0.114.
Upgrade app and coordinator to 0.0.116.
Restart app and coordinator without hiccups.
Collaboratively revert the open subchannel, with the money arriving back in the app's on-chain wallet.

I also did the same (except the last step, since it's impossible) for a channel without an open position.

bonomat · 2023-11-01T19:32:26Z

Cargo.toml

-p2pd-oracle-client = { git = "https://github.com/p2pderivatives/rust-dlc", rev = "9815582" }
-dlc-trie = { git = "https://github.com/p2pderivatives/rust-dlc", rev = "9815582" }
-simple-wallet = { git = "https://github.com/p2pderivatives/rust-dlc", rev = "9815582" }
+dlc-manager = { git = "https://github.com/p2pderivatives/rust-dlc", rev = "d1e08a0" }


Note: we will need to bump this once more because this fix landed: p2pderivatives/rust-dlc#162

bonomat · 2023-11-01T19:42:09Z

crates/ln-dlc-node/src/node/dlc_channel.rs

+    // Enqueue the message.
+    dlc_message_handler.send_message(node_id, msg);
+
+    // According to the LDK docs, you don't _have_ to call this function explicitly if you are


bonomat · 2023-11-01T19:45:18Z

mobile/native/src/ln_dlc/mod.rs

+    // TODO: Should get the funding TXO from the coordinator, since we pass it in as an argument to
+    // the API to collaboratively revert anyway. This way we can avoid depending on the channel
+    // still being tracked by the `ChannelManager`.


Good point, we can definitely do this.

Mentioned in this ticket: #1546

bonomat · 2023-11-01T19:49:50Z

@bonomat, @holzeis: Please have another look at this. There was a slightly tricky rebase in between which forced me to touch the collaborative revert functionality that was recently introduced.

I've tested this a few times locally:
* Open channel + position on 0.0.114.

* Upgrade app and coordinator to 0.0.116.

* Restart app and coordinator without hiccups.

* Collaboratively revert the open subchannel, with the money arriving back in the app's on-chain wallet.
I also did the same (except the last step, since it's impossible) for a channel without an open position.

Unfortunately I cannot confirm that this works for me :(

start coordinator/app/maker on 0c0a33a
trade with app 1 keep position open
trade with app 2 open and close position
upgrade all to this PR
--> can't start the coordiantor:

     Running `target/debug/coordinator`
2023-11-01 19:46:27  INFO coordinator::logger: Initialized logger
2023-11-01 19:46:27  INFO coordinator: Data-dir: "/Users/bonomat/src/github/itchysats/10101/data/coordinator/regtest"
2023-11-01 19:46:27 DEBUG coordinator::cli: Skipping node announcement on '0.0.0.0'.
2023-11-01 19:46:27  INFO coordinator::cli: Adding node announcement within local network 192.168.178.56. Do not use for production!
2023-11-01 19:46:27  INFO ln_dlc_node::on_chain_wallet: Creating the wallet network=Regtest
2023-11-01 19:46:27  INFO ln_dlc_node::node::channel_manager: Found channel manager data on disk. Recovering from stored state
Error: Unknown required feature preventing decode

luckysori · 2023-11-02T02:41:10Z

Unfortunately I cannot confirm that this works for me :(

1. start coordinator/app/maker on [0c0a33a](https://github.com/get10101/10101/commit/0c0a33a157db4ad873147badb12ffea40d28b1bb)

2. trade with app 1 keep position open

3. trade with app 2 open and close position

4. upgrade all to this PR

5. --> can't start the coordiantor:

     Running `target/debug/coordinator`
2023-11-01 19:46:27  INFO coordinator::logger: Initialized logger
2023-11-01 19:46:27  INFO coordinator: Data-dir: "/Users/bonomat/src/github/itchysats/10101/data/coordinator/regtest"
2023-11-01 19:46:27 DEBUG coordinator::cli: Skipping node announcement on '0.0.0.0'.
2023-11-01 19:46:27  INFO coordinator::cli: Adding node announcement within local network 192.168.178.56. Do not use for production!
2023-11-01 19:46:27  INFO ln_dlc_node::on_chain_wallet: Creating the wallet network=Regtest
2023-11-01 19:46:27  INFO ln_dlc_node::node::channel_manager: Found channel manager data on disk. Recovering from stored state
Error: Unknown required feature preventing decode

Hmm, I haven't been able to reproduce your problem yet.

holzeis · 2023-11-02T08:00:55Z

I can confirm that it's working for me 🚀

just wipe
start coordinator, maker
just fund
start app / create wallet
fund wallet with faucet
open position
stop coordinator
change to head of branch chore/upgrade-to-rust-lightning-116
start coordinator / restart maker
just gen ios run
close position 🎉

bonomat · 2023-11-02T16:33:52Z

Unfortunately I have severe problems with this branch :(

Even if I just use this branch to do a restart roundtrip I get errors, e.g.

checkout this branch
start everything up on a new env
fund an open position through app
close coordinator
start coordinator

Get same error :(

2023-11-02 16:33:30  INFO coordinator::logger: Initialized logger
2023-11-02 16:33:30  INFO coordinator: Data-dir: "/Users/bonomat/src/github/itchysats/10101/data/coordinator/regtest"
2023-11-02 16:33:30 DEBUG coordinator::cli: Skipping node announcement on '0.0.0.0'.
2023-11-02 16:33:30  INFO coordinator::cli: Adding node announcement within local network 192.168.178.56. Do not use for production!
2023-11-02 16:33:30  INFO ln_dlc_node::on_chain_wallet: Creating the wallet network=Regtest
2023-11-02 16:33:30  INFO ln_dlc_node::node::channel_manager: Found channel manager data on disk. Recovering from stored state
2023-11-02 16:33:30  INFO lightning::ln::channelmanager: Successfully loaded channel 1653c2e484433eb51a01a0550bd3a22032f6ffff08658b4311dd8c687eebb992 at update_id 0 against monitor at update id 0
Error: Unknown required feature preventing decode

Edit: I thought I'll give upgrading rust to 1.73 a change but without luck.

bonomat · 2023-11-02T18:45:18Z

I did a full wipe of all cached dependencies, data and docker and now I get a new error 😅

2023-11-02 18:40:46  INFO coordinator::logger: Initialized logger
2023-11-02 18:40:46  INFO coordinator: Data-dir: "/Users/bonomat/src/github/10101/10101/data/coordinator/regtest"
2023-11-02 18:40:46 DEBUG coordinator::cli: Skipping node announcement on '0.0.0.0'.
2023-11-02 18:40:46  INFO coordinator::cli: Adding node announcement within local network 192.168.178.56. Do not use for production!
2023-11-02 18:40:46  INFO ln_dlc_node::on_chain_wallet: Creating the wallet network=Regtest
2023-11-02 18:40:47  INFO ln_dlc_node::node::channel_manager: Found channel manager data on disk. Recovering from stored state
2023-11-02 18:40:47  INFO lightning::ln::channelmanager: Successfully loaded channel 190c731887bfd7644d16ee22da3e486327d99cbae0488ad0e6cf2f2be55d1c5b at update_id 0 against monitor at update id 0
2023-11-02 18:40:47  INFO lightning::ln::channelmanager: Successfully loaded channel 84fa01f9fae2999302f145746559758b11c27a36467c5539ce49c2dd17013e40 at update_id 13 against monitor at update id 13
2023-11-02 18:40:47  INFO lightning::ln::channelmanager: Successfully loaded channel 1e41104c8dd12cfc1c396ddf95dfa20a516a6e0f9eebdbf156ab3df2b2f4331d at update_id 5 against monitor at update id 5
2023-11-02 18:40:47 ERROR coordinator: Aborting after panic in task info=panicked at 'We already checked for monitor presence when loading channels', /Users/bonomat/.cargo/git/checkouts/rust-lightning-bee3233422fa4387/b13daab/lightning/src/ln/channelmanager.rs:8879:22
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2: std::backtrace::Backtrace::create
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/backtrace.rs:332:13
   3: coordinator::main::{{closure}}::{{closure}}
             at ./coordinator/src/bin/coordinator.rs:59:29
   4: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/alloc/src/boxed.rs:1987:9
   5: std::panicking::rust_panic_with_hook
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:695:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:582:13
   7: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:150:18
   8: rust_begin_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:578:5
   9: core::panicking::panic_fmt
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:67:14
  10: core::panicking::panic_display
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:150:5
  11: core::panicking::panic_str
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/panicking.rs:134:5
  12: core::option::expect_failed
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:2025:5
  13: core::option::Option<T>::expect
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/option.rs:913:21
  14: lightning::ln::channelmanager::<impl lightning::util::ser::ReadableArgs<lightning::ln::channelmanager::ChannelManagerReadArgs<M,T,ES,NS,SP,F,R,L>> for (bitcoin::hash_types::newtypes::BlockHash,lightning::ln::channelmanager::ChannelManager<M,T,ES,NS,SP,F,R,L>)>::read
             at /Users/bonomat/.cargo/git/checkouts/rust-lightning-bee3233422fa4387/b13daab/lightning/src/ln/channelmanager.rs:8878:19
  15: ln_dlc_node::node::channel_manager::build
             at ./crates/ln-dlc-node/src/node/channel_manager.rs:98:27
  16: ln_dlc_node::node::Node<S>::new
             at ./crates/ln-dlc-node/src/node/mod.rs:340:31
  17: coordinator::main::{{closure}}
             at ./coordinator/src/bin/coordinator.rs:106:25
  18: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.37/src/instrument.rs:272:9
  19: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/park.rs:283:63
  20: tokio::runtime::coop::with_budget
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/coop.rs:107:5
  21: tokio::runtime::coop::budget
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/coop.rs:73:5
  22: tokio::runtime::park::CachedParkThread::block_on
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/park.rs:283:31
  23: tokio::runtime::context::BlockingRegionGuard::block_on
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/context.rs:315:13
  24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/scheduler/multi_thread/mod.rs:66:9
  25: tokio::runtime::runtime::Runtime::block_on
             at /Users/bonomat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.28.2/src/runtime/runtime.rs:304:45
  26: coordinator::main
             at ./coordinator/src/bin/coordinator.rs:320:5
  27: core::ops::function::FnOnce::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/sys_common/backtrace.rs:134:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/core/src/ops/function.rs:287:13
  31: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  32: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  33: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:485:40
  36: std::panicking::try
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panicking.rs:449:19
  37: std::panic::catch_unwind
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/panic.rs:140:14
  38: std::rt::lang_start_internal
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:148:20
  39: std::rt::lang_start
             at /rustc/90c541806f23a127002de5b4038be731ba1458ca/library/std/src/rt.rs:165:17
  40: _main

[1]    90478 abort      cargo run --bin coordinator

error comes from here:

https://github.com/p2pderivatives/rust-lightning/blob/b13daab906b04efb50a2e467c0299ed6ea81b058/lightning/src/ln/channelmanager.rs#L8878-L8880

luckysori · 2023-11-02T20:01:35Z

I did a full wipe of all cached dependencies, data and docker and now I get a new error 😅

Did you check out the right commit? That's a problem that was solved with p2pderivatives/rust-lightning@a57281b for me.

bonomat · 2023-11-03T07:34:25Z

Mhm, thanks... I suspect I became the unsuspecting victim of cosmic radiation, which decided to play a little prank by flipping a few bits. This cosmic shenanigan led to the birth of a version control monster! This sneaky creature was a master at hide and seek and thwarted all my attempts to reset the git branch against the remote. But fear not, for after a good old-fashioned restart of my machine, I finally managed to outwit and defeat the cosmic trickster, and now we're on the path to world peace, one git commit at a time!

luckysori · 2023-11-03T07:44:46Z

Mhm, thanks... I suspect I became the unsuspecting victim of cosmic radiation, which decided to play a little prank by flipping a few bits. This cosmic shenanigan led to the birth of a version control monster! This sneaky creature was a master at hide and seek and thwarted all my attempts to reset the git branch against the remote. But fear not, for after a good old-fashioned restart of my machine, I finally managed to outwit and defeat the cosmic trickster, and now we're on the path to world peace, one git commit at a time!

Hahaha, I knew we would get there eventually!

Only 15 seconds was not always enough. Giving it a minute seems to always work locally.

luckysori

Nice.

holzeis

Works for me!

Some notes: - We've had to increase the payment amounts in the `multi_hop_payment` and `just_in_time_channel_with_multiple_payments` tests, as otherwise we would run into `route not found` errors. It's probably worth investigating why this is the case. - We have decided to eagerly send `rust-dlc` messages because: (1) after the upgrade they were not being automatically sent as fast as they used to and (2) we can speed up the protocols this way anyway. Co-authored-by: Philipp Hoenisch <philipp@hoenisch.at>

The same `forwarding_fee_proportional_millionths` field already exists in `LnDlcNodeSettings`.

bonomat requested review from holzeis and luckysori October 13, 2023 14:31

This comment was marked as resolved.

Sign in to view

luckysori assigned bonomat Oct 15, 2023

This comment was marked as resolved.

Sign in to view

holzeis approved these changes Oct 16, 2023

View reviewed changes

crates/ln-dlc-node/src/node/mod.rs Outdated Show resolved Hide resolved

...le/native/native.xcodeproj/xcuserdata/bonomat.xcuserdatad/xcschemes/xcschememanagement.plist Outdated Show resolved Hide resolved

luckysori reviewed Oct 18, 2023

View reviewed changes

bonomat force-pushed the chore/upgrade-to-rust-lightning-116 branch from b8b46be to c136039 Compare October 19, 2023 13:00

luckysori assigned luckysori and unassigned bonomat Oct 20, 2023

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch from 7222d84 to 949fcfa Compare October 24, 2023 05:30

luckysori changed the title ~~chore: upgrade to rust-lightning 0.0.116 and bump rust-dlc~~ Upgrade to rust-lightning 0.0.116 and bump rust-dlc Oct 24, 2023

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch 8 times, most recently from dd7a857 to 172807d Compare October 25, 2023 02:32

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch from 172807d to c0daf29 Compare November 1, 2023 14:24

luckysori requested a review from holzeis November 1, 2023 14:28

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch 2 times, most recently from aa6fe70 to 32a9226 Compare November 1, 2023 15:00

bonomat mentioned this pull request Nov 1, 2023

Collaborative revert still uses the channel_manager which it shouldn't #1546

Closed

bonomat commented Nov 1, 2023

View reviewed changes

luckysori mentioned this pull request Nov 2, 2023

Ensure that rust-dlc's ChainMonitor is loaded on start-up #1547

Closed

luckysori added 2 commits November 3, 2023 21:19

chore(ln-dlc-node): Speed up tests

bfc4035

fix: Wait enough for new faucet channel to be ready

c74ee90

Only 15 seconds was not always enough. Giving it a minute seems to always work locally.

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch from 32a9226 to 8ef9415 Compare November 3, 2023 10:23

luckysori enabled auto-merge November 3, 2023 10:33

luckysori approved these changes Nov 3, 2023

View reviewed changes

holzeis approved these changes Nov 3, 2023

View reviewed changes

luckysori and others added 2 commits November 3, 2023 22:07

chore(coordinator): Remove duplicated config parameter

247a892

The same `forwarding_fee_proportional_millionths` field already exists in `LnDlcNodeSettings`.

luckysori force-pushed the chore/upgrade-to-rust-lightning-116 branch from 8ef9415 to 247a892 Compare November 3, 2023 11:07

luckysori added this pull request to the merge queue Nov 3, 2023

Merged via the queue into main with commit f553126 Nov 3, 2023
8 checks passed

luckysori deleted the chore/upgrade-to-rust-lightning-116 branch November 3, 2023 11:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to `rust-lightning` 0.0.116 and bump `rust-dlc` #1482

Upgrade to `rust-lightning` 0.0.116 and bump `rust-dlc` #1482

bonomat commented Oct 13, 2023 •

edited by luckysori

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

holzeis left a comment

bonomat commented Oct 19, 2023

luckysori commented Nov 1, 2023 •

edited

Loading

bonomat Nov 1, 2023

bonomat Nov 1, 2023

bonomat Nov 1, 2023

bonomat Nov 1, 2023

bonomat commented Nov 1, 2023

luckysori commented Nov 2, 2023

holzeis commented Nov 2, 2023

bonomat commented Nov 2, 2023 •

edited

Loading

bonomat commented Nov 2, 2023

luckysori commented Nov 2, 2023 •

edited

Loading

bonomat commented Nov 3, 2023

luckysori commented Nov 3, 2023

luckysori left a comment

holzeis left a comment

Upgrade to rust-lightning 0.0.116 and bump rust-dlc #1482

Upgrade to rust-lightning 0.0.116 and bump rust-dlc #1482

Conversation

bonomat commented Oct 13, 2023 • edited by luckysori Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

holzeis left a comment

Choose a reason for hiding this comment

bonomat commented Oct 19, 2023

luckysori commented Nov 1, 2023 • edited Loading

bonomat Nov 1, 2023

Choose a reason for hiding this comment

bonomat Nov 1, 2023

Choose a reason for hiding this comment

bonomat Nov 1, 2023

Choose a reason for hiding this comment

bonomat Nov 1, 2023

Choose a reason for hiding this comment

bonomat commented Nov 1, 2023

luckysori commented Nov 2, 2023

holzeis commented Nov 2, 2023

bonomat commented Nov 2, 2023 • edited Loading

bonomat commented Nov 2, 2023

luckysori commented Nov 2, 2023 • edited Loading

bonomat commented Nov 3, 2023

luckysori commented Nov 3, 2023

luckysori left a comment

Choose a reason for hiding this comment

holzeis left a comment

Choose a reason for hiding this comment

Upgrade to `rust-lightning` 0.0.116 and bump `rust-dlc` #1482

Upgrade to `rust-lightning` 0.0.116 and bump `rust-dlc` #1482

bonomat commented Oct 13, 2023 •

edited by luckysori

Loading

luckysori commented Nov 1, 2023 •

edited

Loading

bonomat commented Nov 2, 2023 •

edited

Loading

luckysori commented Nov 2, 2023 •

edited

Loading