Merge probabilistic scores from external source #3562

joostjager · 2025-01-27T08:41:35Z

Usage in LDK node: lightningdevkit/ldk-node#449

codecov · 2025-01-27T08:53:19Z

Codecov Report

Attention: Patch coverage is 89.27203% with 28 lines in your changes missing coverage. Please review.

Project coverage is 89.29%. Comparing base (b1fc7d8) to head (055177c).
Report is 35 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/routing/scoring.rs	89.27%	25 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3562      +/-   ##
==========================================
+ Coverage   88.53%   89.29%   +0.75%     
==========================================
  Files         149      149              
  Lines      114475   120978    +6503     
  Branches   114475   120978    +6503     
==========================================
+ Hits       101347   108022    +6675     
+ Misses      10634    10577      -57     
+ Partials     2494     2379     -115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joostjager · 2025-01-30T11:30:48Z

lightning/src/routing/scoring.rs

+
+	fn time_passed(&mut self, duration_since_epoch: Duration, decay_params: ProbabilisticScoringDecayParameters) {
+		self.0.retain(|_scid, liquidity| {
+			liquidity.min_liquidity_offset_msat =


Maybe move this into the ChannelLiquidity (singular) struct

If we do so, would we gain much by introducing ChannelLiquidities at all? Maybe we just use HashMap<u64, ChannelLiquidity> in the API?

I think being able to offer state-level functionality does make things a bit cleaner. Maybe I should also add a merge method on this level.

The original reason for this struct though is to be able to use ser/deser logic without a scorer.

joostjager · 2025-01-30T11:31:09Z

lightning/src/routing/scoring.rs

-	channel_liquidities: HashMap<u64, ChannelLiquidity>,
+	channel_liquidities: ChannelLiquidities,
+}
+/// ChannelLiquidities contains live and historical liquidity bounds for each channel.


Objections to moving this into its own file?

Yea, splitting scoring.rs into two modules would be nice. We generally don't put individual structs in their own module just for the sake of it but when files get too big, splitting them down the middle (if there's a clean way to do it) is always nice...there's a few files that are in desperate need of it.

For my workflow, sticking to one struct per file would work well. I do find myself navigating quite a bit in these large files, using editor features (find, find symbol, find ref) to make it easier but not perfect. I'd rather use the folder/file hierarchy and pinning of files as tabs for example. But it is personal of course.

Some type of split would be welcome either way. For this PR I could start with a liquidity_information (open to naming suggestions) module and place the new ChannelLiquidities in it. Then in a separate PR move more liquidity code (ChannelLiquidity, HistoricalLiquidityTracker, HistoricalBucketRangeTracker and tests) in there. Thoughts?

Thoughts?

I like the idea of breaking up our humongous modules and splitting more types out, be it in this PR or a follow up.
IMO, we could consider a folder structure such as:

src/routing/scoring/mod.rs (moved from src/routing/scoring.rs for backwards compat of the path) src/routing/scoring/liquidity_tracking.rs (or just liquidity.rs ?)

FWIW, another easy step towards cleaning up/smaller files would be to move the entire bucketed_history sub-module out of scoring.rs and into a dedicated bucketed_history.rs file. Although, if we do this, we could consider movng the *Liquidity* types there, too.

joostjager · 2025-01-30T11:48:38Z

lightning/src/routing/scoring.rs

+
+		// Verify that after the merge with a successful payment, the liquidity range is increased.
+		let liquidity_range = combined_scorer.scorer.estimated_channel_liquidity_range(42, &target_node_id());
+		assert_eq!(liquidity_range.unwrap(), (0, 300));


I don't think this number is too magical. Just sitting in between the previous max offset of 600 and the new offset of 0.

tnull · 2025-01-30T11:57:13Z

lightning/src/routing/scoring.rs


 impl<G: Deref<Target = NetworkGraph<L>>, L: Deref> Writeable for ProbabilisticScorer<G, L> where L::Target: Logger {
 	#[inline]
 	fn write<W: Writer>(&self, w: &mut W) -> Result<(), io::Error> {
-		write_tlv_fields!(w, {


I think we want to maintain reading/writing the TLV fields here, rather than moving them into ChannelLiquidities, given that we're more likely to add additional fields requiring persisting on the more general ProbabilisticScorer.

Is it likely that there will be more persistent state unrelated to channels? If so, wouldn't that state also be placed in ChannelLiquidities, extending it with additional fields beyond the hash map? At that point, the struct might need a more general name, but I imagine those new fields would still be part of what you'd want to export/import.

For now, the purpose of creating the struct is to allow deserialization of state from disk or network without having to construct a full probabilistic scorer, which includes non-persistent and irrelevant fields.

lightning/src/routing/scoring.rs

tnull · 2025-01-30T12:09:42Z

lightning/src/routing/scoring.rs

+
+	fn time_passed(&mut self, duration_since_epoch: Duration, decay_params: ProbabilisticScoringDecayParameters) {
+		self.0.retain(|_scid, liquidity| {
+			liquidity.min_liquidity_offset_msat =


If we do so, would we gain much by introducing ChannelLiquidities at all? Maybe we just use HashMap<u64, ChannelLiquidity> in the API?

TheBlueMatt

Sorry for the delay here.

TheBlueMatt · 2025-02-01T14:15:08Z

lightning/src/routing/scoring.rs

 					liquidity.liquidity_history.decay_buckets(elapsed_time.as_secs_f64() / half_life);
-					liquidity.offset_history_last_updated = duration_since_epoch;
+					liquidity.offset_history_last_updated += decay_params.historical_no_updates_half_life;


Not sure I get why we're calling decay_buckets in a loop. It already does buckets *= (1/2)^half_lives so we shouldn't need to call it repeatedly.

Well...actually, looking at it it is wrong, its doing buckets *= 1024 / 2048^half_lives instead of buckets *= (1024 / 2048)^half_lives, but we should fix the math instead of calling it in a loop :)

I am not sure either 😅 At some point I concluded that this was a discrete operation, probably set on the wrong foot by that if elapsed_time > decay_params.historical_no_updates_half_life statement.

I do wonder though why the buckets aren't decayed always like the live bounds and have this 1 half life waiting time? In the end, half life is just a way to express a rate, and it seems a bit strange to also use that in the way it is used in the if expression.

Good catch on the formula. Your suggestion is correct, but 1024/2048 is just 0.5 and doesn't work with integer math. Added a commit to fix it, and a unit test.

TheBlueMatt · 2025-02-01T14:31:02Z

lightning/src/routing/scoring.rs

-	channel_liquidities: HashMap<u64, ChannelLiquidity>,
+	channel_liquidities: ChannelLiquidities,
+}
+/// ChannelLiquidities contains live and historical liquidity bounds for each channel.


Yea, splitting scoring.rs into two modules would be nice. We generally don't put individual structs in their own module just for the sake of it but when files get too big, splitting them down the middle (if there's a clean way to do it) is always nice...there's a few files that are in desperate need of it.

lightning/src/routing/scoring.rs

Wrap the liquidities hash map into a struct so that decay and serialization functionality can be attached. This allows external data to be serialized into this struct and decayed to make it comparable and mergeable.

The formula for applying half lives was incorrect. Test coverage added.

joostjager force-pushed the merge-scores branch from 71a9fc0 to f726a8e Compare January 27, 2025 08:45

joostjager force-pushed the merge-scores branch 8 times, most recently from c24cc83 to 85f3fee Compare January 30, 2025 11:30

joostjager commented Jan 30, 2025

View reviewed changes

joostjager force-pushed the merge-scores branch from 85f3fee to 05bca3b Compare January 30, 2025 11:46

joostjager commented Jan 30, 2025

View reviewed changes

joostjager mentioned this pull request Jan 30, 2025

Periodical external score merge lightningdevkit/ldk-node#449

Open

tnull reviewed Jan 30, 2025

View reviewed changes

joostjager force-pushed the merge-scores branch from 05bca3b to d6caa86 Compare January 30, 2025 12:39

joostjager requested a review from tnull January 30, 2025 12:55

joostjager force-pushed the merge-scores branch 2 times, most recently from fdad047 to ced0adc Compare January 30, 2025 16:31

joostjager marked this pull request as ready for review January 31, 2025 08:53

joostjager requested a review from TheBlueMatt January 31, 2025 11:34

TheBlueMatt reviewed Feb 1, 2025

View reviewed changes

joostjager force-pushed the merge-scores branch 3 times, most recently from b459831 to 055177c Compare February 3, 2025 09:29

joostjager added 3 commits February 3, 2025 10:38

refactor hashmap to channelliquidities struct

5564c76

Wrap the liquidities hash map into a struct so that decay and serialization functionality can be attached. This allows external data to be serialized into this struct and decayed to make it comparable and mergeable.

add combined scorer

befedf6

fix historical liquidity bucket decay

6acce79

The formula for applying half lives was incorrect. Test coverage added.

joostjager requested a review from TheBlueMatt February 3, 2025 10:02

joostjager force-pushed the merge-scores branch from 055177c to 6acce79 Compare February 3, 2025 10:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge probabilistic scores from external source #3562

Merge probabilistic scores from external source #3562

joostjager commented Jan 27, 2025 •

edited

Loading

codecov bot commented Jan 27, 2025 •

edited

Loading

joostjager Jan 30, 2025

tnull Jan 30, 2025

joostjager Jan 30, 2025

joostjager Jan 30, 2025

TheBlueMatt Feb 1, 2025

joostjager Feb 3, 2025

tnull Feb 3, 2025

tnull Feb 3, 2025 •

edited

Loading

joostjager Jan 30, 2025

tnull Jan 30, 2025

joostjager Jan 30, 2025 •

edited

Loading

tnull Jan 30, 2025

TheBlueMatt left a comment

TheBlueMatt Feb 1, 2025

joostjager Feb 3, 2025

TheBlueMatt Feb 1, 2025

Merge probabilistic scores from external source #3562

Are you sure you want to change the base?

Merge probabilistic scores from external source #3562

Conversation

joostjager commented Jan 27, 2025 • edited Loading

codecov bot commented Jan 27, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnull Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joostjager commented Jan 27, 2025 •

edited

Loading

codecov bot commented Jan 27, 2025 •

edited

Loading

tnull Feb 3, 2025 •

edited

Loading

joostjager Jan 30, 2025 •

edited

Loading