CHANGELOG.rst

changelog

2.0.19 - 25 March 2023

• Merge pull request #144, mysql-replication support for PyMySQL>0.10.0 was introduced in v0.22
• add support for fillfactor when running init_replica
• improve logging on discarded rows
• add distinct on group concat when collecting foreign keys
• use mysql-replication>=0.31, fix for crash when replicating from MariaDB

2.0.18 - 31 March 2022

• Support the ON DELETE and ON UPDATE clause when creating the foreign keys in PostgreSQL
• change logic for index and foreign key names by managing only duplicates within same schema
• use mysql-replication<0.27 as new versions crash when receiving queries
• add copy_schema method for copying only the schema without data (EXPERIMENTAL)
• change type for identifiers in replica schema to varchar(64)

2.0.17 - 30 January 2022

• Remove argparse from the requirements
• Add the collect for unique constraints when keep_existing_schema is Yes
• Fix wrong order in copy data/create indices when keep_existing_schema is No
• Remove check for log_bin we are replicating from Aurora MySQL
• Manage different the different behaviour in pyyaml to allow pg_chameleon to be installed as rpm in centos 7

2.0.16 - 23 September 2020

• Fix for issue #126 init_replica failure with tables on transactional engine and invalid data

2.0.15 - 20 September 2020

• Support for reduced lock if MySQL engine is transactional, thanks to @rascalDan
• setup.py now requires python-mysql-replication to version 0.22 which adds support for PyMySQL >=0.10.0
• removed PyMySQL requirement <0.10.0 from setup.py
• prevent pg_chameleon to run as root

2.0.14 - 26 July 2020

• Add support for spatial data types (requires postgis installed on the target database)
• When keep_existing_schema is set to yes now drops and recreates indices, and constraints during the init_replica process
• Fix for issue #115 thanks to @porshkevich
• setup.py now forces PyMySQL to version <0.10.0 because it breaks the python-mysql-replication library (issue #117)

2.0.13 - 05 July 2020

• EXPERIMENTAL support for Point datatype - @jovankricka-everon
• Add keep_existing_schema in MySQL source type to keep the existing scema in place instead of rebuilding it from the mysql source
• Change tabs to spaces in code

2.0.12 - 11 Dec 2019

• Fixes for issue #96 thanks to @daniel-qcode
• Change for configuration and SQL files location
• Package can build now as source and wheel
• The minimum python requirements now is 3.5

2.0.11 - 25 Oct 2019

• Fix wrong formatting for yaml example files. @rebtoor
• Make start_replica run in foreground when log_file == stdout . @clifff
• Travis seems to break down constantly, Disable the CI until a fix is found. Evaluate to use a different CI.
• Add the add loader to yaml.load as required by the new PyYAML version.

2.0.10 - 01 Sep 2018

• Fix regression in new replay function with PostgreSQL 10
• Convert to string the dictionary entries pulled from a json field
• Let enable_replica to disable any leftover maintenance flag
• Add capture in CHANGE for tables in the form schema.table

2.0.9 - 19 Aug 2018

• Fix wrong check for the next auto maintenance run if the maintenance wasn't run before
• Improve the replay function's speed
• Remove blocking from the GTID operational mode

2.0.8 - 14 Jul 2018

• Add support for skip events as requested in issue #76. Is now possible to skip events (insert,delete,update) for single tables or for entire schemas.
• EXPERIMENTAL support for the GTID. When configured on MySQL or Percona server pg_chameleon will use the GTID to auto position the replica stream. Mariadb is not supported by this change.
• ALTER TABLE RENAME is now correctly parsed and executed
• Add horrible hack to ALTER TABLE MODIFY. Previously modify with default values would parse wrongly and fail when translating to PostgreSQL dialect
• Disable erroring the source when running with --debug switch enabled
• Add cleanup for logged events when refreshing schema and syncing tables. previously spurious logged events could lead to primary key violations when syncing single tables or refreshing single schemas.

2.0.7 - 19 May 2018

• Fix for issue #71, make the multiprocess logging safe. Now each replica process logs in a separate file
• Fix the --full option to store true instead of false. Previously the option had no effect.
• Add auto_maintenance optional parameter to trigger a vacuum over the log tables after a specific timeout
• Fix for issue #75, avoid the wrong conversion to string for None keys when cleaning up malformed rows during the init replica and replica process
• Fix for issue #73, fix for wrong data type tokenisation when an alter table adds a column with options (e.g. ADD COLUMN foo DEFAULT NULL)
• Fix wrong TRUNCATE TABLE tokenisation if the statement specifies the table with the schema.

2.0.6 - 29 April 2018

• fix for issue #69 add source's optional parameter on_error_read: to allow the read process to continue in case of connection issues with the source database (e.g. MySQL in maintenance)
• remove the detach partition during the maintenance process as this proved to be a very fragile approach
• add switch --full to run a VACUUM FULL during the maintenance
• when running the maintentenance execute a VACUUM instead of a VACUUM FULL
• fix for issue #68. fallback to binlog_row_image=FULL if the parameter is missing in mysql 5.5.
• add cleanup for default value NOW() when adding a new column with ALTER TABLE
• allow enable_replica to reset the source status in the case of a catalogue version mismatch

2.0.5 - 25 March 2018

• fix wrong exclusion when running sync_tables with limit_tables set
• add run_maintenance command to perform a VACUUM FULL on the source's log tables
• add stop_all_replicas command to stop all the running sources within the target postgresql database

2.0.4 - 04 March 2018

• Fix regression added in 2.0.3 when handling MODIFY DDL
• Improved handling of dropped columns during the replica

2.0.3 - 11 February 2018

• fix regression added by commit 8c09ccb. when ALTER TABLE ADD COLUMN is in the form datatype DEFAULT (NOT) NULL the parser captures two words instead of one
• Improve the speed of the cleanup on startup deleting only for the source's log tables instead of the parent table
• fix for issue #63. change the field i_binlog_position to bigint in order to avoid an integer overflow error when the binlog is largher than 2 GB.
• change to psycopg2-binary in install_requires. This change will ensure the psycopg2 will install using the wheel package when available.
• add upgrade_catalogue_v20 for minor schema upgrades

2.0.2 - 21 January 2018

• Fix for issue #61, missing post replay cleanup for processed batches.
• add private method _swap_enums to the class pg_engine which moves the enumerated types from the loading to the destination schema.

2.0.1 - 14 January 2018

• Fix for issue #58. Improve the read replica performance by filtering the row images when limit_tables/skip_tables are set.
• Make the read_replica_stream method private.
• Fix read replica crash if in alter table a column was defined as character varying

2.0.0 - 01 January 2018

• Add option --rollbar-level to set the maximum level for the messages to be sent to rollbar. Accepted values: "critical", "error", "warning", "info". The Default is "info".
• Add command enable_replica used to reset the replica status in case of error or unespected crash
• Add script alias chameleon along with chameleon.py

2.0.0.rc1 - 24 December 2017

• Fix for issue #52, When adding a unique key the table's creation fails because of the NULLable field
• Add check for the MySQL configuration when initialising or refreshing replicated entities
• Add class rollbar_notifier for simpler message management
• Add end of init_replica,refresh_schema,sync_tables notification to rollbar
• Allow --tables disabled when syncing the tables to re synchronise all the tables excluded from the replica

2.0.0.beta1 - 10 December 2017

• fix a race condition where an unrelated DDL can cause the collected binlog rows to be added several times to the log_table
• fix regression in write ddl caused by the change of private method
• fix wrong ddl parsing when a column definition is surrounded by parentheses e.g. ALTER TABLE foo ADD COLUMN(bar varchar(30));
• error handling for wrong table names, wrong schema names, wrong source name and wrong commands
• init_replica for source pgsql now can read from an hot standby but the copy is not consistent
• init_replica for source pgsql adds "replicated tables" for better show_status display
• check if the source is registered when running commands that require a source name

2.0.0.alpha3 - 03 December 2017

• Remove limit_tables from binlogreader initialisation, as we can read from multiple schemas we should only exclude the tables not limit
• Fix wrong formatting for default value when altering a field
• Add upgrade procedure from version 1.8.2 to 2.0
• Improve error logging and table exclusion in replay function
• Add stack trace capture to the rollbar and log message when one of the replica daemon crash
• Add on_error_replay to set whether the replay process should skip the tables or exit on error
• Add init_replica support for source type pgsql (EXPERIMENTAL)

2.0.0.alpha2 - 18 November 2017

• Fix wrong position when determining the destination schema in read_replica_stream
• Fix wrong log position stored in the source's high watermark
• Fix wrong table inclusion/exclusion in read_replica_steam
• Add source parameter replay_max_rows to set the amount of rows to replay. Previously the value was set by replica_batch_size
• Fix crash when an alter table affected a table not replicated
• Fixed issue with alter table during the drop/set default for the column (thanks to psycopg2's sql.Identifier)
• add type display to source status
• Add fix for issue #33 cleanup NUL markers from the rows before trying to insert them in PostgreSQL
• Fix broken save_discarded_row
• Add more detail to show_status when specifying the source with --source
• Changed some methods to private
• ensure the match for the alter table's commands are enclosed by word boundaries
• add if exists when trying to drop the table in swap tables. previously adding a new table failed because the table wasn't there
• fix wrong drop enum type when adding a new field
• add log error for storing the errors generated during the replay
• add not functional class pgsql_source for source type pgsql
• allow type_override to be empty
• add show_status command for displaying the log error entries
• add separate logs for per source
• change log line formatting inspired by the super clean look in pgbackrest (thanks you guys)

2.0.0.alpha1 - 11 November 2017

• Python 3 only development
• Add support for reading from multiple MySQL schemas and restore them it into a target PostgreSQL database. The source and target schema names can be different.
• Conservative approach to the replica. Tables which generate errors are automatically excluded from the replica.
• Daemonised init_replica process.
• Daemonised replica process with two separated subprocess, one for the read and one for the replay.
• Soft replica initialisation. The tables are locked when needed and stored with their log coordinates. The replica damon will put the database in a consistent status gradually.
• Rollbar integration for a simpler error detection.