Move some common database import subroutines to Script::Utils
#3200
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
admin/MBImport.pl and admin/replication/ImportReplicationChanges contained very similar implementations of `ImportTable`, so it would be ideal to share them. I'd also like to use the same functionality in a future commit (to load dbmirror2 packets into temporary tables). The implementations in these two files did diverge slightly. For one, MBImport.pl's allowed fixing broken UTF-8 byte sequences. I'm not sure how necessary that is in 2024, or what the historical reasons for adding it were, but I kept the functionality behind a flag in `copy_table_from_file`. MBImport.pl's also supported the flags `$delete_first` (to empty the table before importing) and `$fProgress` (to control whether progress is shown). I've basically kept all of MBImport.pl's code, with these features behind `%opts` flags. I kept the definitions of `ImportTable`, but they now call `copy_table_from_file` internally. I couldn't replace all of the `ImportTable` calls with direct calls to `copy_table_from_file`, because `ImportTable` also updates statistics local to each file and has a different return value.
reosarevok
reviewed
Mar 12, 2024
There was a problem hiding this comment.
This seems sensible to me, but I'd prefer if we had a test for the extra flags if they wouldn't be too complex. Although it does kinda feel maybe we can just drop the utf8 flag, but maybe we should check with @mayhem first if he remembers the reasons. The comment mentioning automodsaccepted which I have never even seen suggests that this is ancient and the extra options might be unneeded.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We have some identical or very similar-looking subroutines duplicated between admin/MBImport.pl and admin/replication/ImportReplicationChanges:
empty(to tell if a table is empty) andImportTable(to import a table from a file). The two implementations of the latter function differ slightly.For the incremental dumps (
MusicBrainz::Script::Role::IncrementalDump), I'd like to reuse theImportTablefunction to import a dbmirror2 packet into some temporary tables (to parse it).We currently parse the old packets by hand; I'm surprised it works at all (it probably doesn't in some cases) because I don't believe it fully or properly implement's parsing of PostgreSQL's
COPYtextformat. dbmirror2 packets contain JSON, which may contain JSON escape sequences underneathCOPY'stextformat escape sequences, and the mixing of these can be tricky to parse. It's much, much easier to let PostgreSQL do the parsing!Solution
This just adds two shared subroutines,
is_table_emptyandcopy_table_from_filetoScript::Utils, and modifies admin/MBImport.pl and admin/replication/ImportReplicationChanges to use them.Testing
We have existing automated tests that make heavy use of these scripts.
I did not test the
--fix-broken-utf8or--ignore-errorsflags specifically.