Optimize sqlite-to-postgres migration speed for large tables via temporary constraint suspension #1824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Kobzol
reviewed
Feb 26, 2024
eth3lbert
force-pushed
the
speedup-sqlite2pg
branch
from
aba755d to
5049846
Compare
|
typo fixed and feedback addressed. Disabled table triggers
Copied 8879 rows from artifact table in 31.013791ms (286292 rows/second)
Copied 6909 rows from artifact_collection_duration table in 21.010667ms (328833 rows/second)
Copied 74 rows from benchmark table in 599.708µs (123393 rows/second)
Copied 11240494 rows from collection table in 19.367619292s (580376 rows/second)
Copied 308525 rows from collector_progress table in 1.909368375s (161585 rows/second)
Copied 2550 rows from error table in 1.433280209s (1779 rows/second)
Copied 9398 rows from pstat_series table in 56.046292ms (167683 rows/second)
Copied 87437248 rows from pstat table in 189.191683833s (462162 rows/second)
Copied 4332 rows from pull_request_build table in 27.5315ms (157347 rows/second)
Copied 3677103 rows from raw_self_profile table in 12.662463167s (290394 rows/second)
Copied 1175371 rows from rustc_compilation table in 3.671286584s (320152 rows/second)
Enabled table triggers |
eth3lbert
force-pushed
the
speedup-sqlite2pg
branch
from
5049846 to
bef64cc
Compare
…orary constraint suspension
eth3lbert
force-pushed
the
speedup-sqlite2pg
branch
from
bef64cc to
ef67816
Compare
Kobzol
approved these changes
Feb 27, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces an optimization to improve the migration speed from SQLite to PostgreSQL for large tables. It temporarily disables constraints during the migration process by dropping constraints before copying data and adding them back afterwards, allowing for faster data transfer.
1289s234.54s1289.337736334s(67816 rows/second)6.66825msCopied 87437248 rows from pstat table in
147.218779125s(593927 rows/second)Add constraints for table pstat in
87.332639042s46s14.05s46.025009083s(79894 rows/second)1.851375msCopied 3677103 rows from raw_self_profile table in
7.032711416s(522857 rows/second)Add constraints for table raw_self_profile in
7.020468166sNote: This approach prioritizes speed over data integrity. However, since we are migrating from an existing database, the risk of compromising data integrity is likely minimal in this specific case.