Use chacha in `get_secure_random_bytes()` #1974

danielgranhao · 2023-01-20T11:38:16Z

Addresses #1958

lightning/src/chain/keysinterface.rs

tnull

Github won't let me get there, but note that rand_bytes_unique_start is still initialized but now unused:

https://github.com/danielgranhao/rust-lightning/blob/a512d3dda038aec465122aeae2de4844c4cacfca/lightning/src/chain/keysinterface.rs#L1020-L1023

danielgranhao · 2023-01-20T16:07:44Z

@tnull Thank you for your comments. Somehow I didn't notice the unused value warnings. In d3f14e7 I also removed rand_bytes_unique_start.

I'm still missing fixing the failing test ln::functional_tests::test_duplicate_payment_hash_one_failure_one_success(). I will try to look a bit more into it, but if anybody has any intuition about the problem, suggestions on what to look for would be highly appreciated!

TheBlueMatt · 2023-01-25T03:11:19Z

#1984 should fix it :)

jkczyz · 2023-01-25T15:31:30Z

lightning/src/chain/keysinterface.rs

+				let mut nonce = Vec::new();
+				nonce.append(&mut starting_time_secs.to_be_bytes().to_vec());
+				nonce.append(&mut starting_time_nanos.to_be_bytes().to_vec());
+				let chacha = Mutex::new(ChaCha20::new(seed, &nonce));


I think we can avoid three vec allocations by having a fixed array and copying the bytes into it.

Thanks! I've changed to that approach in fac7b4e.

TheBlueMatt · 2023-01-26T00:34:28Z

lightning/src/chain/keysinterface.rs

-				rand_bytes_unique_start.input(&starting_time_nanos.to_be_bytes());
-				rand_bytes_unique_start.input(seed);
+				let mut rand_bytes_unique_start = [0u8; 32];
+				rand_bytes_unique_start[..8].copy_from_slice(&starting_time_secs.to_be_bytes());


We need to hash seed in here somehow. IMO we should keep it as a hash, basically the way it was and just complete the hash rather than leaving it as an unfinished engine.

Yeah, was thinking we may just want to restore and use the original rand_bytes_unique_start as the ChaCha seed?

Yea, restore the old code but complete the hash.

Ok, if I understood you correctly, this is what you propose -> 6472af6

TheBlueMatt · 2023-01-26T00:35:44Z

lightning/src/chain/keysinterface.rs

+
+		for _ in 1..5 {
+			let keys_manager_clone = Arc::clone(&keys_manager);
+			thread::spawn(move || {


You have to join these threads at the end, no?

yep, ty! Applied in 34fa5af

TheBlueMatt · 2023-01-26T00:39:21Z

lightning/src/chain/keysinterface.rs

-	rand_bytes_child_index: AtomicUsize,
-	rand_bytes_unique_start: Sha256State,
+	rand_bytes_unique_start: [u8; 32],
+	rand_bytes_index: AtomicUsize,


I think this should be a util::atomic_counter::AtomicCounter.

Applied in 6472af6

danielgranhao · 2023-01-26T17:58:06Z

I've rebased to take in the changes from #1984. I've added d08d4f8 just to check that fixing test_bump_penalty_txn_on_revoked_htlcs will fix the remaining CI checks. I suppose it needs an improvement similar to the one on test_duplicate_payment_hash_one_failure_one_success.

TheBlueMatt · 2023-01-26T18:37:38Z

LGTM, will let tnull take a look.

tnull

Alright, repeated the benchmarks locally, which drew a pretty clear picture:

| Threads | Method         | Time (ns) |
|---------+----------------+-----------|
| 1       | Hashing        | 1,805,139 |
| 1       | ChaCha/Mutex   | 6,308     |
| 1       | ChaCha/Counter | 10,725    |
| 3       | Hashing        | 1,875,495 |
| 3       | ChaCha/Mutex   | 144,687   |
| 3       | ChaCha/Counter | 18,778    |
| 5       | Hashing        | 1,898,584 |
| 5       | ChaCha/Mutex   | 92,116    |
| 5       | ChaCha/Counter | 40,465    |
|---------+----------------+-----------|

I'd say if we're confident in the counter method, it is probably the way to go.

That said, I'd also be happy to with the Mutex variant, as it's more straight forward, the performance difference is really not that big and realistically the kind of lock contention simulated in the benchmark is really unlikely to happen in production. Note however that in this case we may want to throw in a Sha256 for good measure, instead of using the seed directly.

tnull · 2023-01-26T17:18:58Z

lightning/src/chain/keysinterface.rs

-				nonce[..8].copy_from_slice(&starting_time_secs.to_be_bytes());
-				nonce[8..12].copy_from_slice(&starting_time_nanos.to_be_bytes());
-				let chacha = Mutex::new(ChaCha20::new(seed, &nonce));
+				let mut rand_bytes_unique_start = Sha256::engine();


nit: Rather than shadowing the variable, can we maybe rename this rand_engine_unique_start?

Sure, applied in 1456770.

TheBlueMatt · 2023-01-26T18:53:15Z

I'm more comfortable using the counter method - the ChaCha in-crate rolls doesn't handle chacha counter rollover, meaning it breaks at u32::MAX * 64 bytes, which, okay, we're not gonna hit, but still, the version with the counter doesn't.

TheBlueMatt · 2023-01-26T18:53:43Z

Please also rebase/squash the fixup commit at the end into the appropriate commit.

tnull

Looks good, I think I'm ACK after the tests pass.

lightning/src/chain/keysinterface.rs

0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries" API Updates =========== * `InvoicePayer` has been removed and its features moved directly into `ChannelManager`. As such it now requires a simplified `Router` and supports `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was removed in favor of the automated retries. Invoice payment utilities in `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc). * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner` split out in anticipation of future schnorr/taproot support (lightningdevkit#1967). * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`, and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930). * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key` and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070). * A `lightning-transaction-sync` crate has been added which implements keeping LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can only be used on nodes that *never* ran a previous version of LDK. * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996). * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980). * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as associated builders have been added. Such invoices cannot yet be paid due to missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926). * A `lightning-custom-message` crate has been added to make combining multiple custom messages into one enum/handler easier (lightningdevkit#1832). * `Event::PaymentPathFailure` is now generated for failure to send an HTLC over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043). * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968). * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873). * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962). * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985). * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878) * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057). * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961). * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948). * Macros which implement serialization using TLVs or straight writing of struct fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977). Backwards Compatibility ======================= * Any inbound payments with a custom final CLTV delta will be rejected by LDK if you downgrade prior to receipt (lightningdevkit#1878). * `Event::PaymentPathFailed::network_update` will always be `None` if an 0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043). * `Event::PaymentPathFailed::all_paths_removed` will always be false if an 0.0.114-generated event is read by a prior version of LDK. Users who rely on it to determine payment retries should migrate to `Event::PaymentFailed`, in a separate release prior to upgrading to LDK 0.0.114 if downgrading is supported (lightningdevkit#2043). Performance Improvements ======================== * Channel data is now stored per-peer and channel updates across multiple peers can be operated on simultaneously (lightningdevkit#1507). * Routefinding is roughly 1.5x faster (lightningdevkit#1799). * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016). * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040). * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974). Bug Fixes ========= * Fixed a bug where a delay in processing a `PaymentSent` event longer than the time taken to persist a `ChannelMonitor` update, when occurring immediately prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048). * Fixed spurious rejections of rapid gossip sync data when the graph has been updated by other means between gossip syncs (lightningdevkit#2046). * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos` is set (lightningdevkit#1935). * Resolved an issue where the `ChannelManager::get_persistable_update_future` future would fail to wake until a second notification occurs (lightningdevkit#2064). * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037). * Fixed a deadlock on some platforms at least when using async `ChannelMonitor` updating (lightningdevkit#2006). * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964). * In some cases when payment sending fails on our local channel retries no longer take the same path and thus never succeed (lightningdevkit#2014). * Retries for spontaneous payments have been fixed (lightningdevkit#2002). * Return an `Err` if `lightning-persister` fails to read the directory listing rather than panicing (lightningdevkit#1943). * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035) Security ======== 0.0.114 fixes several denial-of-service vulnerabilities which are reachable from untrusted input from channel counterparties or in deployments accepting inbound connections or channels. It also fixes a denial-of-service vulnerability in rare cases in the route finding logic. * The number of pending un-funded channels as well as peers without funded channels is now limited to avoid denial of service (lightningdevkit#1988). * A second `channel_ready` message received immediately after the first could lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf support in LDK 0.0.107. * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount being sent (including previous-hop fees) is equal to a channel's capacity while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with historical data tracking in LDK 0.0.112. In total, this release features 130 files changed, 21457 insertions, 10113 deletions in 343 commits from 18 authors, in alphabetical order: * Alec Chen * Allan Douglas R. de Oliveira * Andrei * Arik Sosman * Daniel Granhão * Duncan Dean * Elias Rohrer * Jeffrey Czyz * John Cantrell * Kurtsley * Matt Corallo * Max Fang * Omer Yacine * Valentine Wallace * Viktor Tigerström * Wilmer Paulino * benthecarman * jurvis

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from c896c0b to a512d3d Compare January 20, 2023 12:13

danielgranhao mentioned this pull request Jan 20, 2023

Consier speeding up get_secure_random_bytes #1958

Closed

tnull reviewed Jan 20, 2023

View reviewed changes

lightning/src/chain/keysinterface.rs Outdated Show resolved Hide resolved

lightning/src/chain/keysinterface.rs Outdated Show resolved Hide resolved

tnull reviewed Jan 20, 2023

View reviewed changes

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from a512d3d to d3f14e7 Compare January 20, 2023 15:58

jkczyz reviewed Jan 25, 2023

View reviewed changes

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch 3 times, most recently from d943b28 to 17860ac Compare January 25, 2023 23:03

TheBlueMatt reviewed Jan 26, 2023

View reviewed changes

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from 17860ac to 6472af6 Compare January 26, 2023 11:10

Add get_secure_random_bytes() benchmark

eedaf90

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from 6472af6 to d08d4f8 Compare January 26, 2023 17:53

tnull reviewed Jan 26, 2023

View reviewed changes

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from d08d4f8 to 1456770 Compare January 26, 2023 19:04

danielgranhao marked this pull request as ready for review January 26, 2023 19:06

tnull reviewed Jan 26, 2023

View reviewed changes

TheBlueMatt previously approved these changes Jan 26, 2023

View reviewed changes

lightning/src/chain/keysinterface.rs Show resolved Hide resolved

Use Chacha20 in get_secure_random_bytes()

f19821d

danielgranhao dismissed TheBlueMatt’s stale review via f19821d January 26, 2023 20:58

danielgranhao force-pushed the speed-up-secure-random-byte-gen branch from 1456770 to f19821d Compare January 26, 2023 20:58

TheBlueMatt approved these changes Jan 26, 2023

View reviewed changes

tnull approved these changes Jan 26, 2023

View reviewed changes

TheBlueMatt merged commit d4de913 into lightningdevkit:main Jan 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use chacha in `get_secure_random_bytes()` #1974

Use chacha in `get_secure_random_bytes()` #1974

danielgranhao commented Jan 20, 2023 •

edited

Loading

tnull left a comment

danielgranhao commented Jan 20, 2023

TheBlueMatt commented Jan 25, 2023

jkczyz Jan 25, 2023

danielgranhao Jan 25, 2023

TheBlueMatt Jan 26, 2023

tnull Jan 26, 2023

TheBlueMatt Jan 26, 2023

danielgranhao Jan 26, 2023

TheBlueMatt Jan 26, 2023

danielgranhao Jan 26, 2023

TheBlueMatt Jan 26, 2023

danielgranhao Jan 26, 2023

danielgranhao commented Jan 26, 2023

TheBlueMatt commented Jan 26, 2023

tnull left a comment

tnull Jan 26, 2023

danielgranhao Jan 26, 2023

TheBlueMatt commented Jan 26, 2023

TheBlueMatt commented Jan 26, 2023

tnull left a comment

Use chacha in get_secure_random_bytes() #1974

Use chacha in get_secure_random_bytes() #1974

Conversation

danielgranhao commented Jan 20, 2023 • edited Loading

tnull left a comment

Choose a reason for hiding this comment

danielgranhao commented Jan 20, 2023

TheBlueMatt commented Jan 25, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielgranhao commented Jan 26, 2023

TheBlueMatt commented Jan 26, 2023

tnull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheBlueMatt commented Jan 26, 2023

TheBlueMatt commented Jan 26, 2023

tnull left a comment

Choose a reason for hiding this comment

Use chacha in `get_secure_random_bytes()` #1974

Use chacha in `get_secure_random_bytes()` #1974

danielgranhao commented Jan 20, 2023 •

edited

Loading