Speed large account migration #8587
Open
+38
−5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Show the work we're doing (at debug level) and every 10 seconds print progress (at INFO level):x ``` lightningd-1 2025-10-08T05:13:07.973Z INFO lightningd: Creating database lightningd-1 2025-10-08T05:13:10.987Z DEBUG lightningd: Transferring 6166 chain_events lightningd-1 2025-10-08T05:13:11.780Z DEBUG lightningd: Transferring 1660043 channel_events ``` It's the inserting channel_events which takes a long time, slowing down exponentially: ``` lightningd-1 2025-10-08T05:13:18.034Z INFO lightningd: Inserted 26690/1660043 channel_events lightningd-1 2025-10-08T05:13:28.034Z INFO lightningd: Inserted 47086/1660043 channel_events lightningd-1 2025-10-08T05:13:38.035Z INFO lightningd: Inserted 61699/1660043 channel_events lightningd-1 2025-10-08T05:13:48.035Z INFO lightningd: Inserted 73743/1660043 channel_events lightningd-1 2025-10-08T05:13:58.035Z INFO lightningd: Inserted 83244/1660043 channel_events ... lightningd-1 2025-10-08T05:35:18.286Z INFO lightningd: Inserted 466720/1660043 channel_events lightningd-1 2025-10-08T05:35:29.074Z INFO lightningd: Inserted 468437/1660043 channel_events lightningd-1 2025-10-08T05:35:39.079Z INFO lightningd: Inserted 470130/1660043 channel_events lightningd-1 2025-10-08T05:35:49.081Z INFO lightningd: Inserted 471871/1660043 channel_events ``` Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…n migrations. Before db is complete, ld->wallet->db is NULL. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Testing a large db shows Postgres slowing down exponentially as it inserts the channel_events. Rather than updating the index in the db every time, do it at the end, for spectacular speedup: ``` lightningd-1 2025-10-08T05:39:44.333Z INFO lightningd: Creating database lightningd-1 2025-10-08T05:39:47.581Z DEBUG lightningd: Transferring 6166 chain_events lightningd-1 2025-10-08T05:39:48.455Z DEBUG lightningd: Transferring 1660043 channel_events lightningd-1 2025-10-08T05:39:54.390Z INFO lightningd: Inserted 103100/1660043 channel_events lightningd-1 2025-10-08T05:40:04.390Z INFO lightningd: Inserted 283280/1660043 channel_events lightningd-1 2025-10-08T05:40:14.390Z INFO lightningd: Inserted 464065/1660043 channel_events lightningd-1 2025-10-08T05:40:24.390Z INFO lightningd: Inserted 629559/1660043 channel_events lightningd-1 2025-10-08T05:40:34.390Z INFO lightningd: Inserted 800659/1660043 channel_events lightningd-1 2025-10-08T05:40:44.390Z INFO lightningd: Inserted 975433/1660043 channel_events lightningd-1 2025-10-08T05:40:54.390Z INFO lightningd: Inserted 1134719/1660043 channel_events lightningd-1 2025-10-08T05:41:04.390Z INFO lightningd: Inserted 1290549/1660043 channel_events lightningd-1 2025-10-08T05:41:14.390Z INFO lightningd: Inserted 1443304/1660043 channel_events lightningd-1 2025-10-08T05:41:24.390Z INFO lightningd: Inserted 1590013/1660043 channel_events lightningd-1 2025-10-08T05:41:29.148Z INFO lightningd: bookkeeper migration complete: migrated 6166 chainmoves, 1660043 channelmoves, 132481 descriptions ``` Now we complete the entire migration in 1 minute 45 seconds. Thanks to @Michael1101 for reporting this. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au> Changelog-Fixed: db: migration from v25.09 on a reasonable size account database could take almost infinite time.
michael1011
approved these changes
Oct 8, 2025
There was a problem hiding this comment.
Migrated our old bkpr SQLite to PostgreSQL in 8 minutes and 10 seconds on my desktop machine with PostgreSQL default settings in a docker container. That's perfectly fine for a one-time migration
The progress indication is very much appreciated btw. Gives me confidence it is doing what it is supposed to and not stuck
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Boltz failed migration, aborting after 3 hours (!). While sqlite3 migrates their db in 25 seconds, Postgres gets exponentially slower. Fortunately, the fix is to be slightly less naive, and we migrate in 1 minute 44 seconds.