Compact blinded path handling #2961

jkczyz · 2024-03-21T21:30:42Z

Blinded paths may be specified using a more compact representation, replacing pubkeys with short channel ids as described in #2921. This PR implements this in the following manner:

Update BlindedPath and message::ForwardTlvs to support the compact representation
Add a NodeIdLookUp trait used in OnionMessenger to handle forwarding
Implement NodeIdLookUp for ChannelManager
To resolve the IntroductionNode:
- For messaging, use the NetworkGraph in DefaultMessageRouter
- For payments, use either the NetworkGraph or first hops in DefaultRouter

Does not yet include any tests as the creation methods need to be updated still.

jkczyz · 2024-03-21T21:31:36Z

Mostly looking for high-level feedback before moving on to the creation methods.

codecov-commenter · 2024-03-21T21:38:26Z

Codecov Report

Attention: Patch coverage is 68.52459% with 96 lines in your changes are missing coverage. Please review.

Project coverage is 89.37%. Comparing base (eaf76f6) to head (ba13326).
Report is 17 commits behind head on main.

Files	Patch %	Lines
lightning/src/onion_message/messenger.rs	56.17%	37 Missing and 2 partials ⚠️
lightning/src/blinded_path/mod.rs	40.38%	30 Missing and 1 partial ⚠️
lightning/src/routing/router.rs	83.15%	14 Missing and 2 partials ⚠️
lightning/src/blinded_path/message.rs	81.81%	4 Missing ⚠️
lightning/src/ln/channelmanager.rs	0.00%	3 Missing ⚠️
lightning/src/onion_message/packet.rs	66.66%	3 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2961      +/-   ##
==========================================
- Coverage   89.38%   89.37%   -0.01%     
==========================================
  Files         117      117              
  Lines       95622    96720    +1098     
  Branches    95622    96720    +1098     
==========================================
+ Hits        85468    86443     +975     
- Misses       7924     8059     +135     
+ Partials     2230     2218      -12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

TheBlueMatt

Few really quick comments, can do a more careful review later but at a high level it seems fine.

Remind me why we decided to do a separate trait from MessageRouter? I'm fine with that, just don't recall why that decision was made.

TheBlueMatt · 2024-03-21T21:47:06Z

lightning/src/blinded_path/mod.rs

+/// [`OnionMessage`]: crate::ln::msgs::OnionMessage
+pub trait NodeIdLookUp {
+	/// Returns the node if of the channel counterparty with `short_channel_id`.
+	fn next_node_id(&self, short_channel_id: u64) -> Option<PublicKey>;


Doesn't this need a Direction? :)

We could provide one, but this is only for forwarding so we know the origin (us!). Maybe it would be useful for some remote lookup though? Or if we implement this for ReadOnlyNetworkGraph (see other comment).

Ah, I totally missed the context here. Maybe lets include an "our channel's counterparty" in the docs just so people as dense as me can't miss it :)

Added "forwarding node's " as "our" can be ambiguous, IMO.

I still kinda find the trait and method docs not 100% clear that this is specific to us. Maybe add a second sentence to one of them that says like "ie usually our peers"?

I agree with Matt here
It feels like it isn't immediately clear that the forwarding node in this context means "our" node.
Maybe a small cue in the docs might help that indicates such some way.

TheBlueMatt · 2024-03-21T21:51:24Z

lightning/src/onion_message/messenger.rs

@@ -318,15 +321,21 @@ where
 	ES::Target: EntropySource,


ISTM we should implement the lookup on DefaultMessageRouter, no?

Mentioned in an earlier comment, but it would only work for announced channels. Thus, we need to a way to use the mapping in ChannelManager.

We could give DefaultMessageRouter a reference to ChannelManager, but we start getting into circular references. This is because ChannelManager has a DefaultRouter which has a DefaultMessageRouter in order to construct blinded paths for offers without adding another type parameter to ChannelManager. Any thoughts on how to re-work this are welcome, though.

lightning/src/onion_message/messenger.rs

jkczyz · 2024-03-21T22:27:33Z

Few really quick comments, can do a more careful review later but at a high level it seems fine.

Remind me why we decided to do a separate trait from MessageRouter? I'm fine with that, just don't recall why that decision was made.

The trait is only needed for forwarding onion messages, but it may need to be used for finding the next node id for an unannounced channel. For example, a user constructs a two-hop blinded path in an offer where the last hop in the blinded path uses the unannounced channel with their LSP. When forwarding an invoice request for the offer, the LSP's ChannelManager would have the mapping from the scid to the user's node id while its NetworkGraph would not.

TheBlueMatt · 2024-03-26T17:32:13Z

Should we restrict blinded paths in response_paths to require that they include already-resolved introduction points and not allow resolution in response_paths?

valentinewallace

Looks good!

valentinewallace · 2024-03-26T21:29:11Z

lightning/src/routing/router.rs

+	/// hops.
+	///
+	/// This is not exported to bindings users as lifetimes are not expressable in most languages.
+	pub source_node_id: &'a NodeId,


It seems like if we don't use a reference for source_node_id, we can avoid cloning first_channels a few times down below. I feel like I might be missing some tradeoff here though.

Hmm... I think we want to avoid putting 33-bytes in this struct, thus why we try to use references. (cc: @TheBlueMatt)

Regarding the clones, fortunately the Vec only stores references, so we aren't cloning ChannelDetails just allocating a new Vec holding their references. If the find_route took RouteParameters by value, we could do a first pass to resolve IntroductionNode::DirectedShortChannelId to IntroductionNode::NodeId. So to do that with the current interface, we'd need to clone RouteParameters. Not sure if that's any better. 🙁

Yea, we're super size-sensitive on CandiateRouteHop, sadly. Because BlindedPathCandidate is one of the smaller entries it may be that we could get away with 33 bytes here, but lets leave it as a reference for now initially.

valentinewallace · 2024-03-26T23:02:14Z

lightning/src/onion_message/messenger.rs

@@ -614,7 +619,7 @@ where
 			let our_node_id = node_signer.get_node_id(Recipient::Node)
 				.map_err(|()| SendError::GetNodeIdFailed)?;
 			if introduction_node_id == our_node_id {
-				advance_path_by_one(blinded_path, node_signer, &secp_ctx)
+				advance_path_by_one(blinded_path, node_signer, node_id_lookup, &secp_ctx)
 					.map_err(|()| SendError::BlindedPathAdvanceFailed)?;


SendError::BlindedPathAdvanceFailed docs could use an update after this. Looks like we could possibly get rid of that variant and return a SendError from advance_path but I didn't check too closely.

Not sure what you mean by get rid of the variant. What variant would we return here and possibly in advance_path_by_one?

advance_path_by_one has several different errors that it returns, and at a glance it looked like they may be covered by existing SendError variants, e.g. next_hop_pubkey failure is covered by SendError::Secp256k1. But as I said I didn't look too closely so a docs update for BlindedPathAdvanceFailed is also fine by me.

Ah, yeah, we'd need NodeSigner::ecdh to return SendError::Secp256k1. But the KeysManager implementation returns () when given a Recipient::PhantomNode.

I think any errors returned by the signer need to be a bit more general than Secp256k1 to cover the case where we have a remote signer that times out, or indeed the phantom case. One option would be to replace SendError::GetNodeIdFailed with a more generic SignerFailed variant, but that's not related to this PR at all so probably not worth it here.

jkczyz · 2024-03-27T23:38:47Z

Should we restrict blinded paths in response_paths to require that they include already-resolved introduction points and not allow resolution in response_paths?

Hmmm... I'm not sure if we can assume that. Should it really matter though? We'll hit find_path_and_enqueue_onion_message just like we do when initiating a send.

TheBlueMatt · 2024-03-28T01:55:11Z

Hmmm... I'm not sure if we can assume that. Should it really matter though? We'll hit find_path_and_enqueue_onion_message just like we do when initiating a send.

If our goal is to only allow graph lookups for user-initiated actions, rather than actions initiated by others, we shouldn't be doing lookups in response to inbound messages.

jkczyz · 2024-03-28T02:35:17Z

If our goal is to only allow graph lookups for user-initiated actions, rather than actions initiated by others, we shouldn't be doing lookups in response to inbound messages.

Don't we need to do a graph lookup anyway to determine if a direct connection is needed? And when onion messages are more ubiquitous, a more complex MessageRouter used would presumably use the graph to find a path to the introduction node, too?

TheBlueMatt · 2024-03-28T15:18:21Z

Errr, I'm lost. Sorry lol. Indeed, it doesn't matter.

TheBlueMatt · 2024-03-28T19:07:48Z

lightning/src/blinded_path/mod.rs

+		match &self.introduction_node {
+			IntroductionNode::NodeId(pubkey) => {
+				let node_id = NodeId::from_pubkey(pubkey);
+				network_graph.nodes().get_key_value(&node_id).map(|(key, _)| key)


Its super confusing that introduction_node_id could fail spuriously just because the introduction node_id isn't in the graph. Can we rename it?

Do you care if I make it a method on ReadOnlyNetworkGraph instead? I'd like to keep the name short unless you have a better alternative?

I'm not really sure that that solves it. Can we just call it publicly_reachable_introduction_node_id? Or even just public_introduction_node_id.

The name sounds good, but honestly, I got a bit confused when I first read its name as it wasn't immediately clear to me what it does.
I think we should update the function docs, to clearly explain that public in this context means "publicly reachable" node.

TheBlueMatt · 2024-03-28T19:12:05Z

lightning/src/routing/router.rs

 				return Err(LightningError{err: "Cannot generate a route to blinded paths if we are the introduction node to all of them".to_owned(), action: ErrorAction::IgnoreError});
 			}
 			for (_, blinded_path) in route_hints.iter() {
 				if blinded_path.blinded_hops.len() == 0 {
 					return Err(LightningError{err: "0-hop blinded path provided".to_owned(), action: ErrorAction::IgnoreError});
-				} else if &blinded_path.introduction_node_id == our_node_pubkey {
+				} else if blinded_path.introduction_node_id(network_graph) == Some(&our_node_id) {


Should we be caching the output here so we don't repeat graph lookups?

Sure thing!

lightning/src/routing/router.rs

TheBlueMatt · 2024-03-28T20:16:23Z

lightning/src/onion_message/messenger.rs

+			let introduction_node_id = match blinded_path.introduction_node {
+				IntroductionNode::NodeId(pubkey) => pubkey,
+				IntroductionNode::DirectedShortChannelId(..) => {
+					return Err(SendError::UnresolvedIntroductionNode);


In this case we're just using this to check if we're the introduction node, so we should be able to use the node_id_lookup, no? We'd have to check the direction flag here to decide if we're the intro but that seems doable. It seems a bit strange to fail here in a public function.

Yup, good call.

lightning/src/onion_message/messenger.rs

shaavan

Still working down through the commit, but everything looks great so far!
Just wanted to quickly write down a possible simplification I found in the code.

shaavan · 2024-04-03T13:48:13Z

lightning/src/blinded_path/mod.rs

+		let (node_one, node_two) = if node_a < node_b {
+			(node_a, node_b)
+		} else {
+			(node_b, node_a)
+		};
+		match self {
+			Direction::NodeOne => node_one,
+			Direction::NodeTwo => node_two,
+		}
+	}


I think we can simplify this function by using min, and max. Also, I think it will good idea to use them here considering the wording we have used in the Direction Enum.

pub enum Direction { /// The lesser node id when compared lexicographically in ascending order. NodeOne, /// The greater node id when compared lexicographically in ascending order. NodeTwo, }

Suggested change

let (node_one, node_two) = if node_a < node_b {

(node_a, node_b)

} else {

(node_b, node_a)

};

match self {

Direction::NodeOne => node_one,

Direction::NodeTwo => node_two,

}

}

match self {

Direction::NodeOne => core::cmp::min(node_a, node_b),

Direction::NodeTwo => core::cmp::max(node_a, node_b),

}

shaavan

The PR looks great! A few more small improvements, otherwise things look good!

shaavan · 2024-04-04T13:14:40Z

lightning/src/onion_message/messenger.rs

+		if let Destination::BlindedPath(path) = self {
+			let introduction_node = &mut path.introduction_node;
+			if let IntroductionNode::DirectedShortChannelId(direction, scid) = introduction_node {
+				let pubkey = network_graph
+					.channel(*scid)
+					.map(|c| match direction {
+						Direction::NodeOne => c.node_one,
+						Direction::NodeTwo => c.node_two,
+					})
+					.and_then(|node_id| node_id.as_pubkey().ok());
+				if let Some(pubkey) = pubkey {
+					*introduction_node = IntroductionNode::NodeId(pubkey);
+				}
+			}
+		}


Since we have already defined public_introduction_node_id in BlindedPath. We can utilize it here to keep this function clean.

Suggested change

if let Destination::BlindedPath(path) = self {

let introduction_node = &mut path.introduction_node;

if let IntroductionNode::DirectedShortChannelId(direction, scid) = introduction_node {

let pubkey = network_graph

.channel(*scid)

.map(|c| match direction {

Direction::NodeOne => c.node_one,

Direction::NodeTwo => c.node_two,

})

.and_then(|node_id| node_id.as_pubkey().ok());

if let Some(pubkey) = pubkey {

*introduction_node = IntroductionNode::NodeId(pubkey);

}

}

}

if let Destination::BlindedPath(path) = self {

if let Some(pubkey) = path

.public_introduction_node_id(network_graph)

.and_then(|node_id| node_id.as_pubkey().ok())

{

path.introduction_node = IntroductionNode::NodeId(pubkey);

}

}

On another note, router.rs defines NodeId, and IntroductionNode also has a variant NodeId. But these seem to be incompatible with each other, which doesn’t seem to me like ideal behavior.
Maybe we can think about IntroductionNode::NodeId taking NodeId as the field value, or maybe we can define an explicit conversion between them. Let me know what you think about this!

Since we have already defined public_introduction_node_id in BlindedPath. We can utilize it here to keep this function clean.

Good point, though we should only do this for the DirectedShortChannelId. If we do it for the NodeId variant then it could fail for an unannounced node even if the node is a peer.

On another note, router.rs defines NodeId, and IntroductionNode also has a variant NodeId. But these seem to be incompatible with each other, which doesn’t seem to me like ideal behavior. Maybe we can think about IntroductionNode::NodeId taking NodeId as the field value, or maybe we can define an explicit conversion between them. Let me know what you think about this!

So generally, a node id is a pubkey. NodeId is simply a convenience type used to avoid PublicKey's costly verification logic while deserializing a NetworkGraph (see #1107). That said, there should already be conversion functions on NodeId (e.g., the as_pubkey method used here).

If we do it for the NodeId variant then it could fail for an unannounced node even if the node is a peer.

Right! That makes sense!

That said, there should already be conversion functions on NodeId (e.g., the as_pubkey method used here).

Yep, that's true!
Initially, I was wondering if instead of (gossip.rs) NodeId <-> pubkey <-> IntroductionNode::NodeId, we can have a direct conversion between (gossip.rs) NodeId <-> IntroductionNode::NodeId.
But now I think about it, it would just be introducing extra complexity for not a lot of pay-off
Thanks a bunch for your thorough response!

shaavan · 2024-04-04T13:16:58Z

lightning/src/onion_message/messenger.rs

+					NextHop::ShortChannelId(scid) => match self.node_id_lookup.next_node_id(scid) {
+						Some(pubkey) => pubkey,
+						None => {
+							log_trace!(self.logger, "Dropping forwarded onion message to unresolved peer using SCID {}", scid);


nit: Maybe we can rephrase it a little.

Suggested change

log_trace!(self.logger, "Dropping forwarded onion message to unresolved peer using SCID {}", scid);

log_trace!(self.logger, "Dropping forwarded onion message: Unable to resolve next peer using SCID {}", scid);

shaavan · 2024-04-04T13:23:30Z

lightning/src/blinded_path/mod.rs

+		match &self.introduction_node {
+			IntroductionNode::NodeId(pubkey) => {
+				let node_id = NodeId::from_pubkey(pubkey);
+				network_graph.nodes().get_key_value(&node_id).map(|(key, _)| key)


The name sounds good, but honestly, I got a bit confused when I first read its name as it wasn't immediately clear to me what it does.
I think we should update the function docs, to clearly explain that public in this context means "publicly reachable" node.

TheBlueMatt · 2024-04-07T15:37:44Z

Grr, this needs a rebase.

jkczyz · 2024-04-08T21:10:53Z

Rebased

TheBlueMatt

Okay, LGTM modulo a few small nits left. Feel free to squash.

TheBlueMatt · 2024-04-10T14:58:26Z

lightning/src/blinded_path/mod.rs

+/// [`OnionMessage`]: crate::ln::msgs::OnionMessage
+pub trait NodeIdLookUp {
+	/// Returns the node if of the channel counterparty with `short_channel_id`.
+	fn next_node_id(&self, short_channel_id: u64) -> Option<PublicKey>;


I still kinda find the trait and method docs not 100% clear that this is specific to us. Maybe add a second sentence to one of them that says like "ie usually our peers"?

TheBlueMatt · 2024-04-10T15:12:38Z

lightning/src/blinded_path/mod.rs

+			Direction::NodeOne => core::cmp::min(node_a, node_b),
+			Direction::NodeTwo => core::cmp::max(node_a, node_b)


I don't believe this comparison is equivalent (and PublicKey doesn't seem to even impl Cmp is newer libsecp)

Not sure I follow here. Why isn't equivalent? Note also that PublicKey does implement Ord, which is what allows us to use comparisons.

PublicKey's Ord implementation compares libsecp's internal representation (I believe the full X/Y without serializing) and is thus different from the NodeId comparison. Newer versions of the secp256k1 crate have removed the Ord impl because its confusing.

Ah, I see. Was assuming the comment was more about min/max. I'll covert to NodeId.

lightning/src/onion_message/messenger.rs

Allow either using a node id or a short channel id when forwarding an onion message. This allows for a more compact representation of blinded paths, which is advantageous for reducing offer QR code size. Follow-up commits will implement handling the short channel id case as it requires looking up the destination node id.

Useful for applying Option::map when needing both the key and value for when needing a reference to the key with a longer lifetime.

Blinded paths use a pubkey to identify the introduction node, but it will soon allow using a directed short channel id instead. Add an introduction_node_id method to BlindedPath to facilitate lookup in the latter case.

shaavan

LGTM modulo one small nit! 🚀

shaavan · 2024-04-11T12:51:08Z

lightning/src/blinded_path/mod.rs

+/// [`OnionMessage`]: crate::ln::msgs::OnionMessage
+pub trait NodeIdLookUp {
+	/// Returns the node if of the channel counterparty with `short_channel_id`.
+	fn next_node_id(&self, short_channel_id: u64) -> Option<PublicKey>;


I agree with Matt here
It feels like it isn't immediately clear that the forwarding node in this context means "our" node.
Maybe a small cue in the docs might help that indicates such some way.

TheBlueMatt

Your turn @valentinewallace

TheBlueMatt · 2024-04-12T16:56:36Z

lightning/src/onion_message/messenger.rs

+///
+/// Returns [`SendError::UnresolvedIntroductionNode`] if:
+/// - `destination` contains a blinded path with an [`IntroductionNode::DirectedShortChannelId`],
+/// - unless it can be resolved by [`NodeIdLookUp::next_node_id`].


I guess this is true as long as the intro node is us? If its not us we'll still fail I think.

lightning/src/onion_message/messenger.rs

lightning/src/routing/router.rs

valentinewallace · 2024-04-15T15:12:03Z

lightning/src/routing/router.rs

@@ -1863,17 +1888,19 @@ where L::Target: Logger {
 			}
 		},
 		Payee::Blinded { route_hints, .. } => {
-			if route_hints.iter().all(|(_, path)| &path.introduction_node_id == our_node_pubkey) {
+			if introduction_node_id_cache.iter().all(|introduction_node_id| *introduction_node_id == Some(&our_node_id)) {


May be worth spending some time fuzzing the router changes?

Will do, was already planning on it pre-release.

Blinded paths currently contain a node id for the introduction node. However, a compact representation will allow a directed short channel id instead. Update BlindedPathCandidate and OneHopBlindedPathCandidate to store a NodeId reference from either the NetworkGraph or from the user- provided first hops. This approach avoids looking up the introduction node id on demand, which may not be resolvable. Thus, that would require returning an Option from CandidateRouteHop::source and handle None accordingly.

Allow using either a node id or a directed short channel id in blinded paths. This allows for a more compact representation of blinded paths, which is advantageous for reducing offer QR code size. Follow-up commits will implement handling the directed short channel id case in OnionMessenger as it requires resolving the introduction node in MessageRouter.

When OnionMessenger creates an OnionMessage to a Destination, the latter may contain an IntroductionNode::DirectedShortChannelId inside a BlindedPath. Resolve these in DefaultMessageRouter and handle unresolved ones in OnionMessenger.

When forwarding onion messages, the next node may be represented by a short channel id instead of a node id. Parameterize OnionMessenger with a NodeIdLookUp trait to find which node is the next hop. Implement the trait for ChannelManager for forwarding to channel counterparties. Also use this trait when advancing a blinded path one hop when the sender is the introduction node.

Add a version of the create_onion_message utility function that attempts to resolved the introduction node of the destination's BlindedPath using a ReadOnlyNetworkGraph`. Otherwise, if given a path using the compact representation, using create_onion_message would fail. Keep the current version for use internally and for external uses where the blinded path is known to be resolved.

v0.0.123 - May 08, 2024 - "BOLT12 Dust Sweeping" API Updates =========== * To reduce risk of force-closures and improve HTLC reliability the default dust exposure limit has been increased to `MaxDustHTLCExposure::FeeRateMultiplier(10_000)`. Users with existing channels might want to consider using `ChannelManager::update_channel_config` to apply the new default (lightningdevkit#3045). * `ChainMonitor::archive_fully_resolved_channel_monitors` is now provided to remove from memory `ChannelMonitor`s that have been fully resolved on-chain and are now not needed. It uses the new `Persist::archive_persisted_channel` to inform the storage layer that such a monitor should be archived (lightningdevkit#2964). * An `OutputSweeper` is now provided which will automatically sweep `SpendableOutputDescriptor`s, retrying until the sweep confirms (lightningdevkit#2825). * After initiating an outbound channel, a peer disconnection no longer results in immediate channel closure. Rather, if the peer is reconnected before the channel times out LDK will automatically retry opening it (lightningdevkit#2725). * `PaymentPurpose` now has separate variants for BOLT12 payments, which include fields from the `invoice_request` as well as the `OfferId` (lightningdevkit#2970). * `ChannelDetails` now includes a list of in-flight HTLCs (lightningdevkit#2442). * `Event::PaymentForwarded` now includes `skimmed_fee_msat` (lightningdevkit#2858). * The `hashbrown` dependency has been upgraded and the use of `ahash` as the no-std hash table hash function has been removed. As a consequence, LDK's `Hash{Map,Set}`s no longer feature several constructors when LDK is built with no-std; see the `util::hash_tables` module instead. On platforms that `getrandom` supports, setting the `possiblyrandom/getrandom` feature flag will ensure hash tables are resistant to HashDoS attacks, though the `possiblyrandom` crate should detect most common platforms (lightningdevkit#2810, lightningdevkit#2891). * `ChannelMonitor`-originated requests to the `ChannelSigner` can now fail and be retried using `ChannelMonitor::signer_unblocked` (lightningdevkit#2816). * `SpendableOutputDescriptor::to_psbt_input` now includes the `witness_script` where available as well as new proprietary data which can be used to re-derive some spending keys from the base key (lightningdevkit#2761, lightningdevkit#3004). * `OutPoint::to_channel_id` has been removed in favor of `ChannelId::v1_from_funding_outpoint` in preparation for v2 channels with a different `ChannelId` derivation scheme (lightningdevkit#2797). * `PeerManager::get_peer_node_ids` has been replaced with `list_peers` and `peer_by_node_id`, which provide more details (lightningdevkit#2905). * `Bolt11Invoice::get_payee_pub_key` is now provided (lightningdevkit#2909). * `Default[Message]Router` now take an `entropy_source` argument (lightningdevkit#2847). * `ClosureReason::HTLCsTimedOut` has been separated out from `ClosureReason::HolderForceClosed` as it is the most common case (lightningdevkit#2887). * `ClosureReason::CooperativeClosure` is now split into `{Counterparty,Locally}Initiated` variants (lightningdevkit#2863). * `Event::ChannelPending::channel_type` is now provided (lightningdevkit#2872). * `PaymentForwarded::{prev,next}_user_channel_id` are now provided (lightningdevkit#2924). * Channel init messages have been refactored towards V2 channels (lightningdevkit#2871). * `BumpTransactionEvent` now contains the channel and counterparty (lightningdevkit#2873). * `util::scid_utils` is now public, with some trivial utilities to examine short channel ids (lightningdevkit#2694). * `DirectedChannelInfo::{source,target}` are now public (lightningdevkit#2870). * Bounds in `lightning-background-processor` were simplified by using `AChannelManager` (lightningdevkit#2963). * The `Persist` impl for `KVStore` no longer requires `Sized`, allowing for the use of `dyn KVStore` as `Persist` (lightningdevkit#2883, lightningdevkit#2976). * `From<PaymentPreimage>` is now implemented for `PaymentHash` (lightningdevkit#2918). * `NodeId::from_slice` is now provided (lightningdevkit#2942). * `ChannelManager` deserialization may now fail with `DangerousValue` when LDK's persistence API was violated (lightningdevkit#2974). Bug Fixes ========= * Excess fees on counterparty commitment transactions are now included in the dust exposure calculation. This lines behavior up with some cases where transaction fees can be burnt, making them effectively dust exposure (lightningdevkit#3045). * `Future`s used as an `std::...::Future` could grow in size unbounded if it was never woken. For those not using async persistence and using the async `lightning-background-processor`, this could cause a memory leak in the `ChainMonitor` (lightningdevkit#2894). * Inbound channel requests that fail in `ChannelManager::accept_inbound_channel` would previously have stalled from the peer's perspective as no `error` message was sent (lightningdevkit#2953). * Blinded path construction has been tuned to select paths more likely to succeed, improving BOLT12 payment reliability (lightningdevkit#2911, lightningdevkit#2912). * After a reorg, `lightning-transaction-sync` could have failed to follow a transaction that LDK needed information about (lightningdevkit#2946). * `RecipientOnionFields`' `custom_tlvs` are now propagated to recipients when paying with blinded paths (lightningdevkit#2975). * `Event::ChannelClosed` is now properly generated and peers are properly notified for all channels that as a part of a batch channel open fail to be funded (lightningdevkit#3029). * In cases where user event processing is substantially delayed such that we complete multiple round-trips with our peers before a `PaymentSent` event is handled and then restart without persisting the `ChannelManager` after having persisted a `ChannelMonitor[Update]`, on startup we may have `Err`d trying to deserialize the `ChannelManager` (lightningdevkit#3021). * If a peer has relatively high latency, `PeerManager` may have failed to establish a connection (lightningdevkit#2993). * `ChannelUpdate` messages broadcasted for our own channel closures are now slightly more robust (lightningdevkit#2731). * Deserializing malformed BOLT11 invoices may have resulted in an integer overflow panic in debug builds (lightningdevkit#3032). * In exceedingly rare cases (no cases of this are known), LDK may have created an invalid serialization for a `ChannelManager` (lightningdevkit#2998). * Message processing latency handling BOLT12 payments has been reduced (lightningdevkit#2881). * Latency in processing `Event::SpendableOutputs` may be reduced (lightningdevkit#3033). Node Compatibility ================== * LDK's blinded paths were inconsistent with other implementations in several ways, which have been addressed (lightningdevkit#2856, lightningdevkit#2936, lightningdevkit#2945). * LDK's messaging blinded paths now support the latest features which some nodes may begin relying on soon (lightningdevkit#2961). * LDK's BOLT12 structs have been updated to support some last-minute changes to the spec (lightningdevkit#3017, lightningdevkit#3018). * CLN v24.02 requires the `gossip_queries` feature for all peers, however LDK by default does not set it for those not using a `P2PGossipSync` (e.g. those using RGS). This change was reverted in CLN v24.02.2 however for now LDK always sets the `gossip_queries` feature. This change is expected to be reverted in a future LDK release (lightningdevkit#2959). Security ======== 0.0.123 fixes a denial-of-service vulnerability which we believe to be reachable from untrusted input when parsing invalid BOLT11 invoices containing non-ASCII characters. * BOLT11 invoices with non-ASCII characters in the human-readable-part may cause an out-of-bounds read attempt leading to a panic (lightningdevkit#3054). Note that all BOLT11 invoices containing non-ASCII characters are invalid. In total, this release features 150 files changed, 19307 insertions, 6306 deletions in 360 commits since 0.0.121 from 17 authors, in alphabetical order: * Arik Sosman * Duncan Dean * Elias Rohrer * Evan Feenstra * Jeffrey Czyz * Keyue Bao * Matt Corallo * Orbital * Sergi Delgado Segura * Valentine Wallace * Willem Van Lint * Wilmer Paulino * benthecarman * jbesraa * olegkubrakov * optout * shaavan

jkczyz requested review from TheBlueMatt and valentinewallace March 21, 2024 21:31

TheBlueMatt reviewed Mar 21, 2024

View reviewed changes

TheBlueMatt added this to the 0.0.122 milestone Mar 26, 2024

valentinewallace reviewed Mar 26, 2024

View reviewed changes

TheBlueMatt reviewed Mar 28, 2024

View reviewed changes

jkczyz force-pushed the 2024-03-compact-blinded-paths branch from 97d71d6 to 139b3f9 Compare March 29, 2024 16:55

shaavan reviewed Apr 3, 2024

View reviewed changes

jkczyz force-pushed the 2024-03-compact-blinded-paths branch from 139b3f9 to b96d442 Compare April 4, 2024 10:20

shaavan reviewed Apr 4, 2024

View reviewed changes

jkczyz force-pushed the 2024-03-compact-blinded-paths branch from b96d442 to bd1b7ee Compare April 6, 2024 14:56

jkczyz force-pushed the 2024-03-compact-blinded-paths branch from 352ad8a to 565e456 Compare April 8, 2024 21:09

TheBlueMatt reviewed Apr 10, 2024

View reviewed changes

jkczyz added 3 commits April 10, 2024 19:11

Add IndexedMap::get_key_value

b783736

Useful for applying Option::map when needing both the key and value for when needing a reference to the key with a longer lifetime.

Add BlindedPath::introduction_node_id method

63ebacc

Blinded paths use a pubkey to identify the introduction node, but it will soon allow using a directed short channel id instead. Add an introduction_node_id method to BlindedPath to facilitate lookup in the latter case.

jkczyz force-pushed the 2024-03-compact-blinded-paths branch 2 times, most recently from af613c1 to ba13326 Compare April 11, 2024 00:23

shaavan reviewed Apr 11, 2024

View reviewed changes

TheBlueMatt previously approved these changes Apr 12, 2024

View reviewed changes

valentinewallace previously approved these changes Apr 15, 2024

View reviewed changes

jkczyz added 7 commits April 15, 2024 10:58

Resolve IntroductionNode::DirectedShortChannelId

32a5139

When OnionMessenger creates an OnionMessage to a Destination, the latter may contain an IntroductionNode::DirectedShortChannelId inside a BlindedPath. Resolve these in DefaultMessageRouter and handle unresolved ones in OnionMessenger.

Cache introduction_node_id lookup in get_route

3f90d77

Fix typos

fe114ef

jkczyz dismissed stale reviews from valentinewallace and TheBlueMatt via fe114ef April 15, 2024 16:04

jkczyz force-pushed the 2024-03-compact-blinded-paths branch from ba13326 to fe114ef Compare April 15, 2024 16:04

TheBlueMatt approved these changes Apr 15, 2024

View reviewed changes

valentinewallace approved these changes Apr 15, 2024

View reviewed changes

TheBlueMatt merged commit 59778da into lightningdevkit:main Apr 15, 2024
16 checks passed

	log_trace!(self.logger, "Dropping forwarded onion message to unresolved peer using SCID {}", scid);
	log_trace!(self.logger, "Dropping forwarded onion message: Unable to resolve next peer using SCID {}", scid);

		Direction::NodeOne => core::cmp::min(node_a, node_b),
		Direction::NodeTwo => core::cmp::max(node_a, node_b)

Compact blinded path handling #2961

Compact blinded path handling #2961

Conversation

jkczyz commented Mar 21, 2024

jkczyz commented Mar 21, 2024

codecov-commenter commented Mar 21, 2024 • edited Loading

Codecov Report

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz commented Mar 21, 2024

TheBlueMatt commented Mar 26, 2024

valentinewallace left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkczyz commented Mar 27, 2024

TheBlueMatt commented Mar 28, 2024

jkczyz commented Mar 28, 2024

TheBlueMatt commented Mar 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan Apr 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Apr 7, 2024

jkczyz commented Apr 8, 2024

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Mar 21, 2024 •

edited

Loading

shaavan Apr 11, 2024 •

edited

Loading