Skip to content

Conversation

rustyrussell
Copy link
Contributor

Depends on #8556

Now we do major surgery on autogenerate-rpc-examples.py. Finally we run it to produce the canned blocks and wallet for future runs, and re-enable it in CI.

@rustyrussell rustyrussell added this to the v25.12 milestone Sep 19, 2025
```
DEBUG   lightningd: Got depth change 2->3 for e9e31956f77c3844ee2e6e4607dbfebdee95a9aa549668a7a429b8246a6a29de
**BROKEN** lightningd: FATAL SIGNAL 6 (version v25.09-20-g003ba4a)
**BROKEN** lightningd: backtrace: common/daemon.c:41 (send_backtrace) 0x619bef20e274
**BROKEN** lightningd: backtrace: common/daemon.c:78 (crashdump) 0x619bef20e408
**BROKEN** lightningd: backtrace: ./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c:0 ((null)) 0x7a1ccf24532f
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:44 (__pthread_kill_implementation) 0x7a1ccf29eb2c
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:78 (__pthread_kill_internal) 0x7a1ccf29eb2c
**BROKEN** lightningd: backtrace: ./nptl/pthread_kill.c:89 (__GI___pthread_kill) 0x7a1ccf29eb2c
**BROKEN** lightningd: backtrace: ../sysdeps/posix/raise.c:26 (__GI_raise) 0x7a1ccf24527d
**BROKEN** lightningd: backtrace: ./stdlib/abort.c:79 (__GI_abort) 0x7a1ccf2288fe
**BROKEN** lightningd: backtrace: ./assert/assert.c:96 (__assert_fail_base) 0x7a1ccf22881a
**BROKEN** lightningd: backtrace: ./assert/assert.c:105 (__assert_fail) 0x7a1ccf23b516
**BROKEN** lightningd: backtrace: lightningd/peer_control.c:2202 (funding_depth_cb) 0x619bef1ac497
**BROKEN** lightningd: backtrace: lightningd/watch.c:223 (txw_fire) 0x619bef1cfcbf
**BROKEN** lightningd: backtrace: lightningd/watch.c:292 (watch_topology_changed) 0x619bef1cffa4
**BROKEN** lightningd: backtrace: lightningd/chaintopology.c:829 (updates_complete) 0x619bef144a8c
**BROKEN** lightningd: backtrace: lightningd/chaintopology.c:1047 (get_new_block) 0x619bef14561e
```

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…w one via splice.

This happens if the channel is *not* announcable yet.  Then we hit the assertion
in funding_depth_cb that the txid is the same as the current funding.txid.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-EXPERIMENTAL: fixed crash when we splice a channel which hasn't been announced yet.
I got a NULL deref on `infcopy->remote_funding = *inflight->funding->splice_remote_funding`
at once point in testing, so this should prevent that from happening,
yet still allow us to catch it in CI if it happens again.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…nge.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We watch if they are to do with a channel, or have outputs going to us, but otherwise
we didn't, so we never updated the blockheight in the db.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: JSON-RPC: `listtransactions` now correctly updates `blockheight` for txs created by `sendpsbt` which have no change outputs.
It now simply renames tal names, so it's harmless to do even if we're
not going to do memleak detection.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…eak coverage for any typed hash table.

You can now simply add per-tal-object helpers for memleak, but our older pattern required
calling memleak functions explicitly during memleak handling.  Hash tables in particular need
to be dynamically allocated (we override the allocators using htable_set_allocator and assume
this), so it makes sense to have a helper macro that does all three.

This eliminates a huge amount of code.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…ames.

Incorporate a time: this covers the restart case as well.  And make it time_mono(),
which doesn't get overridden when we override normal wall time.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
I noticed, because it pulled in randomness routines.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…tchannelmoves.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…lock.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was changing all the time when I tried to make
autogenerate-rpc-examples.py reproducible.  Turns out it was being
corrupted (it does suspicious things with pointers); rather than try
to diagnose it, I simply rewrote the code to create it only when we
need it.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…LN_DEV_ENTROPY_SEED

Only in developer mode, ofc.

Notes:
1. We have to move the initialization before the lightningd main trace_start,
   since that uses pseudorand().
2. To make the results stable, we need to use per-caller values to randbytes().
   Otherwise external timing changes the call order.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This allows us to override it for deterministic results.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…tion for timeouts.

This is immune to things like clock changes, and has the convenient side-effect that
it will *not* be overridden when we override time for developer purposes.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Copy link
Collaborator

@ShahanaFarooqui ShahanaFarooqui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rustyrussell The changes in ./tests/autogenerate-rpc-examples.py look good at first glance, but I am unable to test them locally due to missing test data files:

./tests/data/autogenerate-bitcoin-blocks.json
./tests/data/autogenerate-bitcoind-wallet.dat

Could you please add these files or provide instructions on how to generate them for local testing?

Except for tracing, that sticks with time_now().

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…ith CLN_DEV_ENTROPY_SEED.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…istincoming.

This ensures that bolt11/bolt12 selection of routehints/blinded paths is always the same.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…itialblockdownload.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We have to add a send_and_mine_block() for cases where we want to get
a txid and then mine it (for canned blocks, we mine it then figure out
which tx it was!).

And fix up out-by-one in saving blocks.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We temporarily add a flake8 exception so it doesn't complain about all
the now-unused variables.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…hannel example.

Now order has changed, the first one is insufficient; this is more robust.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Unless REGENERATE_BLOCKCHAIN is true, in which case we make them.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes our transaction generation (nlocktime) consistent, as well as our htlcs
for payment.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This makes many of our information outputs consistent.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
1. Don't mangle bookkeeper events.
2. Use l3, not l1, for listtransactions: it's more stable.
3. Use l2, not l1, for listpeerchannels and listchannels.
3. Use l2, not l2, for listfunds.

(l1 has a lot of variation).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
And hand in CLN_NEXT_VERSION to replace the version string.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is `make update-doc-examples-newchain`, which creates the canned blocks (and,
since it's the first time, the bitcoind wallet).

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
@ShahanaFarooqui
Copy link
Collaborator

Now uv run make update-doc-examples is throwing error:

15:12:18 - ERROR - Error in setting up nodes: txid f160d78f39a5ccb96c1439333787b98f1a37c1bb3765b34c74047c8895f4e397 not found
15:12:18 - ERROR - txid f160d78f39a5ccb96c1439333787b98f1a37c1bb3765b34c74047c8895f4e397 not found
Traceback (most recent call last):
  File "/home/shahana/workspace/lightning/tests/autogenerate-rpc-examples.py", line 1443, in test_generate_examples
    l1, l2, l3, l4, l5, l6, c12, c23, c25 = setup_test_nodes(node_factory, bitcoind, regenerate_blockchain)
                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shahana/workspace/lightning/tests/autogenerate-rpc-examples.py", line 331, in setup_test_nodes
    c12, c12res = l1.fundchannel(l2, FUND_CHANNEL_AMOUNT_SAT)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shahana/workspace/lightning/contrib/pyln-testing/pyln/testing/utils.py", line 1194, in fundchannel
    assert False, f"txid {res['txid']} not found"
           ^^^^^
AssertionError: txid f160d78f39a5ccb96c1439333787b98f1a37c1bb3765b34c74047c8895f4e397 not found

And uv run make update-doc-examples-newchain is throwing:

15:10:44 - ERROR - Error in generating transactions examples: RPC call failed: method: fetchbip353, payload: {'address': 'send.some@satsto.me'}, error: {'code': -32700, 'data': None, 'message': 'failed to fetch payment instructions: HrnResolutionError("Multiple TXT records existed for the HRN, which is invalid")'}
15:10:44 - ERROR - RPC call failed: method: fetchbip353, payload: {'address': 'send.some@satsto.me'}, error: {'code': -32700, 'data': None, 'message': 'failed to fetch payment instructions: HrnResolutionError("Multiple TXT records existed for the HRN, which is invalid")'}
Traceback (most recent call last):
  File "/home/shahana/workspace/lightning/tests/autogenerate-rpc-examples.py", line 1444, in test_generate_examples
    c23_2, c23res2, c34_2, inv_l11, inv_l21, inv_l22, inv_l31, inv_l32, inv_l34 = generate_transactions_examples(l1, l2, l3, l4, l5, c25, bitcoind)
                                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shahana/workspace/lightning/tests/autogenerate-rpc-examples.py", line 463, in generate_transactions_examples
    update_example(node=l1, method='fetchbip353', params={'address': 'send.some@satsto.me'}, description=['Example of fetching BIP-353 payment details.'])
  File "/home/shahana/workspace/lightning/tests/autogenerate-rpc-examples.py", line 260, in update_example
    response = node.rpc.call(method, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shahana/workspace/lightning/contrib/pyln-testing/pyln/testing/utils.py", line 811, in call
    res = LightningRpc.call(self, method, payload, cmdprefix, filter)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shahana/workspace/lightning/contrib/pyln-client/pyln/client/lightning.py", line 428, in call
    raise RpcError(method, payload, resp['error'])
pyln.client.lightning.RpcError: RPC call failed: method: fetchbip353, payload: {'address': 'send.some@satsto.me'}, error: {'code': -32700, 'data': None, 'message': 'failed to fetch payment instructions: HrnResolutionError("Multiple TXT records existed for the HRN, which is invalid")'}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants