Skip to content

Commit

Permalink
New release (#104)
Browse files Browse the repository at this point in the history
* - Change: drop support for < 8.2
- Change: moved to enums, promoted properties
- Added: logger for more socket info
- Added: slave_uuid support
- Change: config no longer static
- Chore: typos in README/code
- Chore: replace/remove old urls from code
- Chore: changed variables to underscore
- Added: support caching_sha2_password
- Change: BinLogServerInfo static calls removed also added method getServerInfo to MySQLReplicationFactory
  • Loading branch information
krowinski authored Jan 30, 2024
1 parent 95f852a commit 9202753
Show file tree
Hide file tree
Showing 88 changed files with 2,124 additions and 2,831 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ jobs:

strategy:
matrix:
php: [ '7.3', '7.4', '8.0' ]
php: [ '8.2' ]

steps:
- name: Checkout
uses: actions/checkout@v2

- name: Start mysql service
run: |
echo -e "\n[mysqld]\nserver-id=1\nbinlog_format=row\nlog_bin=/var/log/mysql/mysql-bin.log" | sudo tee -a /etc/mysql/my.cnf
Expand Down
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ composer.phar
composer.lock
.php_cs.cache
/example/profiler.php
.idea/
.idea/
.cache/
88 changes: 0 additions & 88 deletions .travis.yml

This file was deleted.

36 changes: 0 additions & 36 deletions .travis/initializedb.sh

This file was deleted.

16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,22 @@
# Release Notes

## v8.0.0 (2024-01-29)

- Change: drop support for < 8.2
- Change: moved to enums, promoted properties
- Added: logger for more socket info
- Added: slave_uuid support (#99)
- Change: EventInfo->id is now EventInfo->serverId (#83)
- Change: config no longer static (#94)
- Chore: typos in README/code
- Chore: replace/remove old dead doc urls from code
- Chore: changed variables to underscore
- Added: support caching_sha2_password (#102)
- Change: BinLogServerInfo static calls removed also added method getServerInfo to MySQLReplicationFactory
- Change: type of bin log position is now string as it can be bigger then php can hande 2^64-1 (#84)

## v7.0.1 (2021-03-09)

- Fixed negative number handling (#80)

## v7.0.0 (2021-01-24)
Expand Down
52 changes: 36 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
php-mysql-replication
=========
[![Build Status](https://travis-ci.org/krowinski/php-mysql-replication.svg?branch=master)](https://travis-ci.org/krowinski/php-mysql-replication)
[![Latest Stable Version](https://poser.pugx.org/krowinski/php-mysql-replication/v/stable)](https://packagist.org/packages/krowinski/php-mysql-replication) [![Total Downloads](https://poser.pugx.org/krowinski/php-mysql-replication/downloads)](https://packagist.org/packages/krowinski/php-mysql-replication) [![Latest Unstable Version](https://poser.pugx.org/krowinski/php-mysql-replication/v/unstable)](https://packagist.org/packages/krowinski/php-mysql-replication)
[![SensioLabsInsight](https://insight.sensiolabs.com/projects/4a0e49d4-3802-41d3-bb32-0a8194d0fd4d/mini.png)](https://insight.sensiolabs.com/projects/4a0e49d4-3802-41d3-bb32-0a8194d0fd4d) [![License](https://poser.pugx.org/krowinski/php-mysql-replication/license)](https://packagist.org/packages/krowinski/php-mysql-replication)
[![Latest Stable Version](https://poser.pugx.org/krowinski/php-mysql-replication/v/stable)](https://packagist.org/packages/krowinski/php-mysql-replication) [![Total Downloads](https://poser.pugx.org/krowinski/php-mysql-replication/downloads)](https://packagist.org/packages/krowinski/php-mysql-replication) [![Latest Unstable Version](https://poser.pugx.org/krowinski/php-mysql-replication/v/unstable)](https://packagist.org/packages/krowinski/php-mysql-replication)
[![Scrutinizer Code Quality](https://scrutinizer-ci.com/g/krowinski/php-mysql-replication/badges/quality-score.png?b=master)](https://scrutinizer-ci.com/g/krowinski/php-mysql-replication/?branch=master)
[![Code Coverage](https://scrutinizer-ci.com/g/krowinski/php-mysql-replication/badges/coverage.png?b=master)](https://scrutinizer-ci.com/g/krowinski/php-mysql-replication/?branch=master)

Pure PHP Implementation of MySQL replication protocol. This allow you to receive event like insert, update, delete with their data and raw SQL queries.
Pure PHP Implementation of MySQL replication protocol. This allows you to receive event like insert, update, delete with their data and raw SQL queries.

Based on a great work of creators:https://github.com/noplay/python-mysql-replication and https://github.com/fengxiangyun/mysql-replication

Expand All @@ -29,11 +27,16 @@ composer install -o

Compatibility (based on integration tests)
=========
PHP

- php 8.2
- php 8.3

MYSQL
- mysql 5.5
- mysql 5.6
- mysql 5.7
- mysql 8.0 (ONLY with mysql_native_password)
- mysql 8.0 (mysql_native_password and caching_sha2_password supported)
- mariadb 5.5
- mariadb 10.0
- mariadb 10.1
Expand Down Expand Up @@ -83,7 +86,7 @@ Available options:

'mariaDbGtid' - MariaDB GTID marker(s) to start from (format 1-1-3,0-1-88)

'slaveId' - script slave id for identification (SHOW SLAVE HOSTS)
'slaveId' - script slave id for identification (default: 666) (SHOW SLAVE HOSTS)

'binLogFileName' - bin log file name to start from

Expand All @@ -103,6 +106,8 @@ Available options:

'heartbeatPeriod' - sets the interval in seconds between replication heartbeats. Whenever the master's binary log is updated with an event, the waiting period for the next heartbeat is reset. interval is a decimal value having the range 0 to 4294967 seconds and a resolution in milliseconds; the smallest nonzero value is 0.001. Heartbeats are sent by the master only if there are no unsent events in the binary log file for a period longer than interval.

'saveUuid' - sets slave uuid for identification (default: 0015d2b6-8a06-4e5e-8c07-206ef3fbd274)

Similar projects
=========
Ruby: https://github.com/y310/kodama
Expand Down Expand Up @@ -394,27 +399,42 @@ FAQ
=========

1. ### Why and when need php-mysql-replication ?
Well first of all mysql don't give you async calls. You usually need to program this in your application (by event dispaching and adding to some queue system and if your db have many point of entry like web, backend other microservices its not always cheap to add processing to all of them. But using mysql replication protocol you can lisen on write events and process then asynchronously (best combo it's to add item to some queue system like rabbitmq, redis or kafka). Also in invalidate cache, search engine replication, real time analytics and audits.

2. ### It's awsome ! but what is the catch ?
Well first of all you need to know that a lot of events may come through, like if you update 1 000 000 records in table "bar" and you need this one insert from your table "foo" Then all must be processed by script and you need to wait for your data. This is normal and this how it's work. You can speed up using [config options](https://github.com/krowinski/php-mysql-replication#configuration).
Also if script crashes you need to save from time to time position form binlog (or gtid) to start from this position when you run this script again to avoid duplicates.
Well first of all MYSQL don't give you async calls. You usually need to program this in your application (by event dispatching and adding to some queue system
and if your db have many point of entry like web, backend other microservices its not always cheap to add processing to all of them. But using mysql replication
protocol you can listen on write events and process then asynchronously (the best combo it's to add item to some queue system like rabbitmq, redis or kafka).
Also in invalidate cache, search engine replication, real time analytics and audits.

2. ### It's awesome ! but what is the catch ?

Well first of all you need to know that a lot of events may come through, like if you update 1 000 000 records in table "bar" and you need this one insert from
your table "foo" Then all must be processed by script, and you need to wait for your data. This is normal and this how it's work. You can speed up
using [config options](https://github.com/krowinski/php-mysql-replication#configuration).
Also, if script crashes you need to save from time to time position form binlog (or gtid) to start from this position when you run this script again to avoid
duplicates.

3. ### I need to process 1 000 000 records and its taking forever!!
Like I mention in 1 point use queue system like rabbitmq, redis or kafka, they will give you ability to process data in multiple scripts.

4. ### I have a problem ? you script is missing something ! I have found a bug !
Create an [issue](https://github.com/krowinski/php-mysql-replication/issues) I will try to work on it in my free time :)

5. ### How much its give overhead to mysql server ?
It work like any other mysql in slave mode and its giving same overhead.
5. ### How much its give overhead to MYSQL server ?

It work like any other MYSQL in slave mode and its giving same overhead.

6. ### Socket timeouts error
To fix this best is to increase db configurations "net_read_timeout" and "net_write_timeout" to 3600. (tx Bijimon)

To fix this best is to increase db configurations ```net_read_timeout``` and ```net_write_timeout``` to 3600. (tx Bijimon)

7. ### Partial updates fix
Set in my.conf ```binlog_row_image=full``` to fix reciving only partial updates.


Set in my.conf ```binlog_row_image=full``` to fix receiving only partial updates.

8. ### No replication events when connected to replica server
Set in my.conf ```log_slave_updates=on``` to fix this (#71)(#66)
Set in my.conf ```log_slave_updates=on``` to fix this (#71)(#66)

9. ### "Big" updates / inserts
Default MYSQL setting generates one big blob of stream this require more RAM/CPU you can change this for smaller stream using
variable ```binlog_row_event_max_size``` [https://dev.mysql.com/doc/refman/8.0/en/replication-options-binary-log.html#sysvar_binlog_row_event_max_size] to
split into smaller chunks
28 changes: 21 additions & 7 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,23 @@
],
"type": "library",
"require": {
"php": ">=7.3",
"php": ">=8.2",
"ext-bcmath": "*",
"ext-json": "*",
"ext-sockets": "*",
"doctrine/collections": "^1.3",
"doctrine/dbal": "^3.0",
"psr/simple-cache": "^1.0",
"symfony/dependency-injection": "^3.1|^4.0|^5.0",
"symfony/event-dispatcher": "^3.1|^4.0|^5.0"
"doctrine/collections": "^2.1",
"doctrine/dbal": "^3.8",
"psr/log": "^3.0",
"psr/simple-cache": "^3.0",
"symfony/dependency-injection": "^7.0",
"symfony/event-dispatcher": "^7.0"
},
"require-dev": {
"phpunit/phpunit": "^9.0"
"kubawerlos/php-cs-fixer-custom-fixers": "^3.19",
"monolog/monolog": "^3.5",
"phpstan/phpstan": "^1.10",
"phpunit/phpunit": "^10.5",
"symplify/easy-coding-standard": "^12.1"
},
"license": "MIT",
"authors": [
Expand All @@ -45,5 +50,14 @@
"minimum-stability": "stable",
"config": {
"sort-packages": true
},
"scripts": {
"cs:check": "ecs check",
"cs:fix": "ecs check --fix",
"phpstan:analyse": "phpstan analyse -cphpstan.neon",
"sa": [
"@cs:check",
"@phpstan:analyse"
]
}
}
22 changes: 22 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
services:
replication-test-mysql-percona:
container_name: replication-test-mysql-percona
hostname: replication-test-mysql-percona
image: mysql:8.0
platform: linux/amd64
command: [
'--character-set-server=utf8mb4',
'--collation-server=utf8mb4_unicode_ci',
# '--default-authentication-plugin=caching_sha2_password',
#'--default-authentication-plugin=mysql_native_password',
'--log_bin=binlog',
'--max_binlog_size=8M',
'--binlog_format=row',
'--server-id=1'
]
environment:
- MYSQL_ROOT_PASSWORD=root
- MYSQL_DATABASE=mysqlreplication_test
ports:
- "3306:3306/tcp"
restart: unless-stopped
50 changes: 50 additions & 0 deletions ecs.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
<?php

declare(strict_types=1);

use PhpCsFixer\Fixer\ArrayNotation\ReturnToYieldFromFixer;
use PhpCsFixer\Fixer\ArrayNotation\YieldFromArrayToYieldsFixer;
use PhpCsFixer\Fixer\Casing\NativeTypeDeclarationCasingFixer;
use PhpCsFixer\Fixer\FunctionNotation\PhpdocToParamTypeFixer;
use PhpCsFixer\Fixer\FunctionNotation\PhpdocToPropertyTypeFixer;
use PhpCsFixer\Fixer\FunctionNotation\PhpdocToReturnTypeFixer;
use PhpCsFixer\Fixer\Phpdoc\PhpdocAddMissingParamAnnotationFixer;
use PhpCsFixer\Fixer\Whitespace\TypeDeclarationSpacesFixer;
use PhpCsFixerCustomFixers\Fixer\NoLeadingSlashInGlobalNamespaceFixer;
use PhpCsFixerCustomFixers\Fixer\PhpdocNoSuperfluousParamFixer;
use PhpCsFixerCustomFixers\Fixer\PromotedConstructorPropertyFixer;
use Symplify\EasyCodingStandard\Config\ECSConfig;
use Symplify\EasyCodingStandard\ValueObject\Set\SetList;

return static function (ECSConfig $ecsConfig): void {
$ecsConfig->parallel();
$ecsConfig->sets([
SetList::PSR_12,
SetList::CLEAN_CODE,
SetList::STRICT,
SetList::ARRAY,
SetList::PHPUNIT,
SetList::DOCTRINE_ANNOTATIONS,
SetList::COMMENTS,
SetList::SYMPLIFY,
SetList::CONTROL_STRUCTURES,
]);

$ecsConfig->rules([
NativeTypeDeclarationCasingFixer::class,
ReturnToYieldFromFixer::class,
TypeDeclarationSpacesFixer::class,
YieldFromArrayToYieldsFixer::class,
PhpdocToPropertyTypeFixer::class,
PhpdocToParamTypeFixer::class,
PhpdocToReturnTypeFixer::class,
PromotedConstructorPropertyFixer::class,
NoLeadingSlashInGlobalNamespaceFixer::class,
PhpdocNoSuperfluousParamFixer::class,
PhpdocAddMissingParamAnnotationFixer::class,
]);

$ecsConfig->fileExtensions(['php']);
$ecsConfig->cacheDirectory('.cache/ecs');
$ecsConfig->paths([__DIR__ . '/src', __DIR__ . '/tests', __DIR__ . '/example',]);
};
Loading

0 comments on commit 9202753

Please sign in to comment.