Introduce Padding for Payment and Message Blinded Tlvs #3177

shaavan · 2024-07-13T15:22:35Z

Description

This PR introduces padding for Payment and Message Blinded TLVs to ensure that the size of each packet in the path is uniform.

shaavan · 2024-07-13T15:23:37Z

Notes:

Padding Implementation:
- The padding field is now expanded to contain Option<Vec>. This change allows the inclusion of padding within the packet while retaining the flexibility to leave the padding as None.
Impact on Path Length:
- With the added padding, the length of each individual packet increases, which results in a reduction of the maximum path length. Consequently, the max_path_length in forward_check_failures has been adjusted from 18 to 17.

codecov-commenter · 2024-07-13T15:29:45Z

Codecov Report

Attention: Patch coverage is 98.53659% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.33%. Comparing base (85d1e5f) to head (610dd41).

Files with missing lines	Patch %	Lines
lightning/src/blinded_path/utils.rs	91.42%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3177      +/-   ##
==========================================
+ Coverage   88.30%   88.33%   +0.02%     
==========================================
  Files         149      149              
  Lines      112912   113110     +198     
  Branches   112912   113110     +198     
==========================================
+ Hits        99711    99916     +205     
+ Misses      10716    10709       -7     
  Partials     2485     2485

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lightning/src/blinded_path/message.rs

lightning/src/blinded_path/utils.rs

lightning/src/blinded_path/message.rs

shaavan · 2024-07-18T14:18:55Z

Updated from pr3177.01 to pr3177.02 (diff):
Addressed @jkczyz's comments.

Changes:

Reordered commits and separated the update to the Padding struct into a separate commit.
Updated the Padding struct to contain a usize instead of a Vec to save on heap allocation, and made the Writable implementation more efficient.
Introduced two f: commits to use iter clone instead of Vec allocation, keeping the code efficient.
Introduced a test and a test utility to verify that blinded message and payment paths are properly padded, ensuring a consistent payload size.

shaavan · 2024-07-18T14:33:16Z

Updated from pr3177.02 to pr3177.03 (diff):

Changes:

Rebase on main to fix ci.

Diff post rebase:

--- a/lightning/src/ln/offers_tests.rs
+++ b/lightning/src/ln/offers_tests.rs
@@ -1789,7 +1789,7 @@ fn test_blinded_path_padding() {
        let onion_message = charlie.onion_messenger.next_onion_message_for_peer(david_id).unwrap();
        david.onion_messenger.handle_onion_message(&charlie_id, &onion_message);
 
-       let invoice = extract_invoice(david, &onion_message);
+       let (invoice, _) = extract_invoice(david, &onion_message);
        assert_eq!(invoice, expected_invoice);

lightning/src/blinded_path/message.rs

lightning/src/ln/offers_tests.rs

shaavan · 2024-07-22T18:19:59Z

Updated from pr3177.03 to pr3177.04 (diff):
Addressed @jkczyz comments

Refactor max_length calculation to be cleaner.
Introduce wrapper struct instead of tuple to make the code clearer to understand.
Move the padding testing to onion_message/functional_tests.rs to simplify the setup, and test the code at a more appropriate place.

lightning/src/onion_message/packet.rs

lightning/src/blinded_path/payment.rs

lightning/src/onion_message/functional_tests.rs

lightning/src/ln/offers_tests.rs

shaavan · 2024-07-23T12:02:14Z

Updated from pr3177.04 to pr3177.05 (diff):

Updates:

Rebase on main.

shaavan · 2024-07-23T13:28:24Z

Updated from pr3177.05 to pr3177.06 (diff):
Addressed @jkczyz comments

Changes:

Squash commits.
Introduce a generic struct WithPadding to keep the code DRY.
Introduce a test for Blinded Payment Paths.
Use saturating sub in padding length calculation to avoid overflows.

lightning/src/blinded_path/utils.rs

lightning/src/ln/blinded_payment_tests.rs

shaavan · 2024-07-26T18:23:46Z

Updated from pr3177.06 to pr3177.07 (diff):
Addressed @jkczyz comments

Update comments.
Introduce debug_assert!() in Writeable for WithPadding.
Properly assert!() the condition in test.

shaavan · 2024-07-27T14:18:12Z

Updated from pr3177.07 to pr3177.08 (diff):
Addressed @jkczyz comments

Changes:

Introduce an f: commit with an alternative approach to WithPadding.
The new approach introduces padding as an optional field within the ForwardTlvs and the ReceiveTlvs struct.

lightning/src/blinded_path/message.rs

lightning/src/blinded_path/payment.rs

shaavan · 2024-08-01T14:04:40Z

Update: From pr3177.08 to pr3177.09 (diff):

Changes based on @jkczyz's feedback:

Fixed the imports.
Removed BlindedPaymentTlvsRef and replaced it with BlindedPaymentTlvs. The former contained an immutable reference to the underlying TLVs, but the new implementation required mutability. Therefore, I transitioned to using BlindedPaymentTlvs directly, making BlindedPaymentTlvsRef redundant.

lightning/src/onion_message/packet.rs

jkczyz · 2024-08-14T04:18:36Z

lightning/src/onion_message/messenger.rs

@@ -1772,6 +1772,7 @@ fn packet_payloads_and_keys<T: OnionMessageContents, S: secp256k1::Signing + sec
 				if let Some(ss) = prev_control_tlvs_ss.take() {
 					payloads.push((Payload::Forward(ForwardControlTlvs::Unblinded(
 						ForwardTlvs {
+							padding: None,


@TheBlueMatt One thing that I'm unsure of is whether we only need to pad the blinded path or if we should also pad the unblinded portion of the onion.

lightning/src/blinded_path/payment.rs

shaavan · 2024-08-14T09:45:16Z

Update: From pr3177.09 to pr3177.10 (diff):

Changes based on @jkczyz's feedback:

Rename pad_tlvs -> pad_to_length.

shaavan · 2024-09-27T12:25:17Z

Update: From pr3177.18 to pr3177.19 (diff):

Changes:

Rebase on main to resolve merge conflicts.

github-advanced-security · 2024-09-27T12:26:59Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

shaavan · 2024-10-01T14:29:23Z

Update:
From pr3177.19 to pr3177.20 (diff):
Addressed feedback from @TheBlueMatt and @jkczyz.

Changes:

Simplified the padding approach by removing padding as a field from {Forward, Receive} TLVs.
Introduced a WithPadding generic that handles padding for the given TLVs.

Considerations & Shortcomings:

Instead of padding to a maximum value, WithPadding pads the TLVs to a rounded value. This avoids wasting space for very small packets while providing flexibility for future TLV expansions.
The current padding round-off value is 50 bytes. I believe this should offer enough buffer to obscure the internal TLV types, but would love to get feedback on a more optimal value.
This approach doesn't safeguard against the improper use of a 1 TLV record within the TLVs themselves. We may need to add a comment to prevent this misuse.

Todo:

Currently, this approach also pads the compact blinded path. I'll address this in the next update.

shaavan · 2024-10-02T15:25:20Z

Update: From pr3177.20 to pr3177.21 (diff):
Addressed @TheBlueMatt comment

Changes:

Update code so that we would not be padding the compact blinded paths.
Introduce a test to check this condition.

shaavan · 2024-10-05T10:48:22Z

Update: From pr3177.21 to pr3177.22 (diff):

Changes:

Rebase on main to resolve merge conflicts.

shaavan · 2024-10-05T12:05:02Z

Update: From pr3177.22 to pr3177.23 (diff):

Changes:

Fixed the Padding approach.

Previous version: Packets were padded individually based on compact forward TLVs. This incorrectly led to padding ReceiveTlvs, even when they belonged to a compact blinded path.
Fix: Now, the entire blinded path remains unpadded if any short_channel_id is present. This ensures correct handling of compact paths and eliminates unnecessary padding.

TheBlueMatt

Basically LGTM, thanks!

TheBlueMatt · 2024-10-07T14:41:54Z

lightning/src/blinded_path/utils.rs

-/// Reads padding to the end, ignoring what's read.
-pub(crate) struct Padding {}
+/// Represents the padding round off size (in bytes) that is used to pad [`BlindedHop`]
+pub const PADDING_ROUND_OFF: usize = 50;


Why 50? Are common blinded path hops 50 bytes? We should at least write down how big blinded path hops generally are to make sure we're not cutting right in the middle of common sizes here.

Why 50? Are common blinded path hops 50 bytes?

Seems this would depend on the received TLVs' size (i.e., how big MessageContext or PaymentContext is for the given use case).

We should at least write down how big blinded path hops generally are to make sure we're not cutting right in the middle of common sizes here.

Could you elaborate on this concern?

My concern is if there's some feature of a blinded path which an observer wants to detect, we don't want to have a constant here that is exactly on the line between that feature and not that feature, otherwise we have gained nothing.

First off, I was able to determine the common sizes for various message and payment TLVs. You can check out the branch where I introduced the test.

After that, I ran some experiments with different padding sizes. It looks like P=50 and P=70 are both solid options. P=50 gave us 3 unique groups, and P=70 reduced it to 2 while keeping the extra padding fairly minimal.

Here's a table for comparison:

Original Sizes Rounded (P=50) Rounded (P=70)

35 50 70

70 100 70

10 50 70

28 50 70

62 100 70

96 100 140

96 100 140

96 100 140

27 50 70

42 50 70

26 50 70

26 50 70

128 150 140

53 100 70

Let me know which size you think we should go with! I'm happy to make any adjustments.

Thanks so much!

Could you label the rows by their types?

Something looks off when adding the ForwardTlvs. I'd expect each row to be divisible by P, but that's not the case for P > 0. IIUC, each additional ForwardTlvs should add 2 * P.

Hi!
Thanks so much for the nudge!

I’ve updated the table in the comment above, and with the corrected data, it looks like P=31 performs better for the Offer case, while P=29 edges out P=31 for the Refund case.

Curious to hear your thoughts on this!

Btw, I also put together a Python script to analyze the data with a grayscale visualization to better see which values work better across all cases. You can check it out here: Gist Link.

Looking forward to your thought!
Thanks!

Introduce specific paddings for Offer & Refund payee tlvs in pr3177.28
Thanks a lot for all the discussions & pointers!

Given the potential for a human-readable name in InvoiceRequestFields (or a quantity for that matter), using a size of 30 would give us 28 bytes to fit in either of those without adding another block at the cost of using only an additional 10 bytes when the path has three forwards. So I'd be inclined to use that. I don't think we need to use a different value for the refund case, though.

@TheBlueMatt How does that sound?

Sounds good to me!

TheBlueMatt · 2024-10-07T14:43:25Z

lightning/src/blinded_path/utils.rs

+
+/// Represents optional padding for encrypted payloads.
+/// Padding is used to ensure payloads have a consistent length.
+pub(crate) struct Padding {


Should we split Padding into a write-side one and a read-side one now that we only use the write-side one here and the read-side one elsewhere in the module? Also because its weird that length is only used on the write side.

That makes sense! I’m just wondering though, would introducing two separate structs for Padding add extra complexity to the code?

lightning/src/blinded_path/utils.rs

jkczyz · 2024-10-07T21:21:59Z

lightning/src/blinded_path/utils.rs

-/// Reads padding to the end, ignoring what's read.
-pub(crate) struct Padding {}
+/// Represents the padding round off size (in bytes) that is used to pad [`BlindedHop`]
+pub const PADDING_ROUND_OFF: usize = 50;


Why 50? Are common blinded path hops 50 bytes?

Seems this would depend on the received TLVs' size (i.e., how big MessageContext or PaymentContext is for the given use case).

We should at least write down how big blinded path hops generally are to make sure we're not cutting right in the middle of common sizes here.

Could you elaborate on this concern?

lightning/src/blinded_path/utils.rs

shaavan · 2024-10-10T15:32:15Z

Update: From pr3177.23 to pr3177.24 (diff):

Addressed @jkczyz comments

Changes:

Fix typo, and visibility.

shaavan · 2024-10-30T11:24:23Z

Update: From pr3177.24 to pr3177.25 (diff):
Addressed comments from @jkczyz and @TheBlueMatt.

Changes:

Introduced different padding round-off for message and payment TLVs.
Padding sizes were chosen based on the discussion in this thread.

shaavan · 2024-11-25T10:05:02Z

Update: From pr3177.25 to pr3177.26 (diff):
Addressed @jkczyz comments

Changes:

Update the Padding for Payment Tlvs to 35, according to this discussion.

1. This allows setting the length of padding at the time of writing. 2. This will be used in the following commit to allow setting the padding for blinded message paths, and blinded payment paths.

Add a generic `WithPadding` struct to handle padding for `ForwardTlvs` and `ReceiveTlvs` used in `BlindedMessagePath` and `BlindedPaymentPath`. This struct applies padding to the contained TLVs, rounding them off to a specified value. This design provides flexibility in padding TLVs of varying sizes. The `PADDING_ROUND_OFF` value is chosen to be sufficiently large to properly mask the data type of the contained TLVs.

A note of Compact Blinded Paths: Compact Blinded paths are intended to be as short as possible. So to maintain there compactness, we don't apply padding to them.

shaavan · 2024-12-25T13:46:00Z

Update: From pr3177.26 to pr3177.27 (diff):

Changes:

Rebase on main.

- This allows using optimal padding for offer & refund case.

Add test to verify blinded message and payment path padding.

shaavan · 2024-12-25T14:19:48Z

Update: From pr3177.27 to pr3177.28 (diff):
Addressed @jkczyz comments

Changes:

Introduce specific paddings for payee_tlvs based on if they belong to a Refund or an Offer.
Introduce test to check for the padding in case of Offer & Refund.

TheBlueMatt · 2025-01-16T17:39:10Z

lightning/src/blinded_path/utils.rs

+impl<T:Writeable> Writeable for WithPadding<T> {
+	fn write<W: Writer>(&self, writer: &mut W) -> Result<(), io::Error> {
+		let length = self.tlvs.serialized_length();
+		let padding_length = (length + self.round_off - 1) / self.round_off * self.round_off - length;


Should this account for the Type and Value overhead of the padding?

TheBlueMatt · 2025-01-16T17:39:37Z

lightning/src/blinded_path/utils.rs

+		let padding = Some(Padding::new(padding_length));
+
+		encode_tlv_stream!(writer, {
+			(1, padding, option),


1 seems like a weird type to use here, given its likely to conflict in many cases. Should we instead use some random-high-odd-number (or rename the struct to say something about being only for blinded paths)?

I believe this is defined in the spec, though maybe we don't need to use that type?

Right, and if we're only gonna use it for blinded paths for places where 1 is defined as padding, that's okay, but then we need to rename the struct and update the docs to indicate that :).

TheBlueMatt · 2025-01-16T17:56:18Z

lightning/src/blinded_path/utils.rs

+		Self { length }
+	}
+}
+
 impl Readable for Padding {


Let's just remove this? We currently read it in ControlTlvs in type 1, but that's dumb, its an odd type, we can just ignore it (leaving a comment in place of the read) and the TLV decoding logic will ignore its bytes.

jkczyz self-requested a review July 15, 2024 20:03

jkczyz reviewed Jul 15, 2024

View reviewed changes

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/utils.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

jkczyz reviewed Jul 17, 2024

View reviewed changes

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

shaavan force-pushed the padding branch from 51f2b7f to fd6a0e6 Compare July 18, 2024 14:18

shaavan force-pushed the padding branch from fd6a0e6 to 57e7258 Compare July 18, 2024 14:33

jkczyz reviewed Jul 19, 2024

View reviewed changes

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

jkczyz reviewed Jul 19, 2024

View reviewed changes

lightning/src/ln/offers_tests.rs Outdated Show resolved Hide resolved

shaavan force-pushed the padding branch from 57e7258 to 48384c6 Compare July 22, 2024 18:19

jkczyz reviewed Jul 22, 2024

View reviewed changes

shaavan force-pushed the padding branch from 48384c6 to 4fe5ad0 Compare July 23, 2024 12:02

shaavan force-pushed the padding branch from 4fe5ad0 to e7d370b Compare July 23, 2024 13:28

jkczyz reviewed Jul 24, 2024

View reviewed changes

shaavan force-pushed the padding branch from e7d370b to 7c479ee Compare July 26, 2024 18:23

shaavan force-pushed the padding branch from 7c479ee to 4704435 Compare July 27, 2024 14:18

jkczyz reviewed Jul 30, 2024

View reviewed changes

lightning/src/blinded_path/message.rs Outdated Show resolved Hide resolved

lightning/src/blinded_path/payment.rs Outdated Show resolved Hide resolved

shaavan force-pushed the padding branch from 4704435 to c872c44 Compare August 1, 2024 14:04

jkczyz reviewed Aug 14, 2024

View reviewed changes

shaavan force-pushed the padding branch from c872c44 to 467cb44 Compare August 14, 2024 09:45

shaavan force-pushed the padding branch from 467cb44 to a66205a Compare August 16, 2024 06:54

shaavan force-pushed the padding branch from 23888a2 to 2f4c243 Compare September 27, 2024 12:25

shaavan force-pushed the padding branch from 2f4c243 to 112c6c8 Compare October 1, 2024 14:29

shaavan force-pushed the padding branch from 112c6c8 to e307148 Compare October 2, 2024 15:25

shaavan force-pushed the padding branch from e307148 to 41ca8a6 Compare October 5, 2024 10:48

shaavan force-pushed the padding branch from 41ca8a6 to 46bcfd0 Compare October 5, 2024 12:04

TheBlueMatt reviewed Oct 7, 2024

View reviewed changes

jkczyz reviewed Oct 7, 2024

View reviewed changes

shaavan force-pushed the padding branch from 46bcfd0 to 25aea5a Compare October 10, 2024 15:32

shaavan force-pushed the padding branch from 25aea5a to c179775 Compare October 30, 2024 11:24

shaavan force-pushed the padding branch from c179775 to 699a173 Compare November 25, 2024 10:04

shaavan added 4 commits December 25, 2024 19:08

Update Padding struct to contain padding length.

9fbd9ce

1. This allows setting the length of padding at the time of writing. 2. This will be used in the following commit to allow setting the padding for blinded message paths, and blinded payment paths.

Introduce Padding for BlindedMessage Paths.

1b2ebc8

A note of Compact Blinded Paths: Compact Blinded paths are intended to be as short as possible. So to maintain there compactness, we don't apply padding to them.

Introduce Padding for Blinded Payment Paths

bdfea56

shaavan force-pushed the padding branch from 699a173 to 4819e76 Compare December 25, 2024 13:45

shaavan added 2 commits December 25, 2024 19:25

f: Use distinct padding values for Offer & Refund payee_tlvs

3353804

- This allows using optimal padding for offer & refund case.

Introduce test for Padding

610dd41

Add test to verify blinded message and payment path padding.

shaavan force-pushed the padding branch from 4819e76 to 610dd41 Compare December 25, 2024 14:19

TheBlueMatt reviewed Jan 16, 2025

View reviewed changes

Original Sizes	Rounded (P=50)	Rounded (P=70)
35	50	70
70	100	70
10	50	70
28	50	70
62	100	70
96	100	140
96	100	140
96	100	140
27	50	70
42	50	70
26	50	70
26	50	70
128	150	140
53	100	70

Original Sizes	Rounded (P=50)	Rounded (P=70)
35	50	70
70	100	70
10	50	70
28	50	70
62	100	70
96	100	140
96	100	140
96	100	140
27	50	70
42	50	70
26	50	70
26	50	70
128	150	140
53	100	70

Introduce Padding for Payment and Message Blinded Tlvs #3177

Are you sure you want to change the base?

Introduce Padding for Payment and Message Blinded Tlvs #3177

Conversation

shaavan commented Jul 13, 2024

Description

shaavan commented Jul 13, 2024

Notes:

codecov-commenter commented Jul 13, 2024 • edited by codecov bot Loading

Codecov Report

shaavan commented Jul 18, 2024

shaavan commented Jul 18, 2024

shaavan commented Jul 22, 2024

shaavan commented Jul 23, 2024

shaavan commented Jul 23, 2024

shaavan commented Jul 26, 2024

shaavan commented Jul 27, 2024

shaavan commented Aug 1, 2024

Choose a reason for hiding this comment

shaavan commented Aug 14, 2024

shaavan commented Sep 27, 2024

github-advanced-security bot commented Sep 27, 2024

shaavan commented Oct 1, 2024

shaavan commented Oct 2, 2024

shaavan commented Oct 5, 2024

shaavan commented Oct 5, 2024

Fixed the Padding approach.

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaavan commented Oct 10, 2024

shaavan commented Oct 30, 2024

shaavan commented Nov 25, 2024

shaavan commented Dec 25, 2024

shaavan commented Dec 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Jul 13, 2024 •

edited by codecov bot

Loading

shaavan Dec 23, 2024 •

edited

Loading

Original Sizes	Rounded (P=50)	Rounded (P=70)
35	50	70
70	100	70
10	50	70
28	50	70
62	100	70
96	100	140
96	100	140
96	100	140
27	50	70
42	50	70
26	50	70
26	50	70
128	150	140
53	100	70