Support data migration from Codis sharding mode instance to Pika clas…

…sic mode instance.

CHANGES.md

@@ -0,0 +1,4 @@
+# Release Notes
+1.0.0.0
+
+支持数据从codis 分片模式实例迁移到 pika classic模式实例。

LICENSE

@@ -19,3 +19,27 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
+
+---
+
+The MIT License (MIT)
+
+Copyright (c) 2019 Alibaba Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

README.md

@@ -0,0 +1,84 @@
+# codis2pika
+
+
+[![CI](https://github.com/GetuiLaboratory/codis2pika/actions/workflows/ci.yml/badge.svg?branch=v3)](https://github.com/GetuiLaboratory/codis2pika/actions/workflows/ci.yml)
+
+Codis2pika is a tool used to migrate codis data to pika. The main purpose is to support the migration of Codis sharding mode to Pika classic mode.
+
+## 感谢
+
+codis2pika 参考借鉴了阿里开源的redis-shake项目，并进行了定制化的改造。因此基本的功能特性与原工具一致，但是功能上存在差异。
+
+
+## Features
+
+Same features as the original:
+
+* 🤗 Support the use of lua custom filtering rules (this part has not been changed, so there is no actual test, but it is theoretically supported.
+* 💪 Support large instance migration.
+
+Some features of codis2pika：
+* 🌐 Support the source side as a stand-alone instance and the destination side as a stand-alone instance.
+* 🌲 Only five basic data structures of Redis are supported.
+* ✅ Testing on the Codis server based on Redis 3.2.
+* ⏰ Support long time real-time data synchronization with several seconds delay.
+* ✊ Not sensitive to the underlying storage mode of the instance.
+
+
+### Description of changes
+* Clustering is not supported: because the data is distributed differently at the bottom of the source instance (codis sharding mode) and the target instance (pika class mode), it needs to be allocated according to the actual business situation. If necessary, add a corresponding algorithm to restore the cluster write interface.
+* If Redis migration is required, it is recommended to use [RedisShark](https://github.com/alibaba/RedisShake) tool for more comprehensive functions. This project is mainly to support the migration of sharding mode instances to pika classic instances.
+
+# Document
+
+## install
+
+### Binary package
+
+Release: [https://github.com/GetuiLaboratory/codis2pika/releases](https://github.com/GetuiLaboratory/codis2pika/releases)
+
+### Compile from source
+
+After downloading the source code, run the `sh build. sh` command to compile.
+
+```shell
+sh build.sh
+```
+
+## Usage
+
+1. Edit codis2pika.toml, modify the source and target configuration items.
+2. Start codis2pika:
+
+```shell
+./bin/codis2pika codis2pika.toml
+```
+
+3. Check data synchronization status.
+
+## Configure
+
+The codis2pika configuration file refers to `codis2pika. toml`. To avoid ambiguity, it is mandatory to assign values to each configuration in the configuration file, otherwise an error will be reported.
+
+## Data filtering
+
+codis2pika supports custom filtering rules using lua scripts. codis2pika can be started with the following command:
+
+```shell
+./bin/codis2pika codis2pika.toml filter/xxx.lua
+```
+However, the lua data filtering function has not been verified. Please refer to the redis shark project if necessary.
+
+## Attention
+* Extra large keys are not supported;
+* The migrated codis needs to set the client output buffer limit to release the restriction, otherwise the link will be broken;
+* The migrated node needs to reserve memory redundancy of the same amount of migrated data;
+* Before and after data migration, it is recommended to `compact`;
+* Multi db is not supported (I don't think it is necessary, and it is not supported for the time being);
+* The expiration time of some keys may be delayed.
+
+## Visualization
+It is recommended to configure the monitoring disk in advance to have a visual grasp of the migration process.
+
+## Verify
+Alibaba open-source redis-full-check is recommended.

README_zh.md

@@ -0,0 +1,84 @@
+# codis2pika
+
+[![CI](https://github.com/GetuiLaboratory/codis2pika/actions/workflows/ci.yml/badge.svg?branch=v3)](https://github.com/GetuiLaboratory/codis2pika/actions/workflows/ci.yml)
+
+codis2pika 是一个用来做 codis 数据迁移到 pika 的工具。主要目的是为了支持codis 分片模式迁移到 pika classic模式。
+
+## 感谢
+
+codis2pika 参考借鉴了阿里开源的redis-shake项目，并进行了定制化的改造。因此基本的功能特性与原工具一致，但是功能上存在差异。
+
+
+## 特性
+
+与原版相同的特性：
+
+* 🤗 支持使用 lua 自定义过滤规则（这部分未作改动，因此没实际测试，但理论上是支持的）
+* 💪 支持大实例迁移
+
+codis2pika的一些特性：
+* 🌐 支持源端为单机实例，目的端为单机实例
+* 🌲 仅支持 Redis 5种基础数据结构
+* ✅ 测试在 Redis 3.2 版本的codis server
+* ⏰ 支持较长时间的数据实时同步，存在几秒延迟
+* ✊ 对实例的底层存储模式不敏感
+
+### 改动说明
+* 不支持集群： 由于数据在源实例（codis sharding模式）与目标实例（pika classic模式）底层分布不同，需要结合业务实际情况分配。如有需要，可以在添加对应算法，恢复集群写入接口。
+* 如果需要redis迁移的，建议还是用[RedisShark](https://github.com/alibaba/RedisShake)工具，功能更全面。本项目主要是为了支持sharding模式实例迁移到pika classic实例。
+
+# 文档
+
+## 安装
+
+### 从 Release 下载安装
+
+Release: [https://github.com/GetuiLaboratory/codis2pika/releases](https://github.com/GetuiLaboratory/codis2pika/releases)
+
+### 从源码编译
+
+下载源码后，运行 `sh build.sh` 命令编译。
+
+```shell
+sh build.sh
+```
+
+## 运行
+
+1. 编辑 codis2pika.toml，修改其中的 source 与 target 配置项
+2. 启动 codis2pika：
+
+```shell
+./bin/codis2pika codis2pika.toml
+```
+
+3. 观察数据同步情况
+
+## 配置
+
+codis2pika 配置文件参考 `codis2pika.toml`。 为避免歧义强制要求配置文件中的每一项配置均需要赋值，否则会报错。
+
+## 数据过滤
+
+codis2pika 支持使用 lua 脚本自定义过滤规则，可以实现对数据进行过滤。 搭配 lua 脚本时，codis2pika 启动命令：
+
+```shell
+./bin/codis2pika codis2pika.toml filter/xxx.lua
+```
+lua 数据过滤功能未作验证，如有需要请参考redis-shark项目
+
+## 注意事项
+* 不支持特大key;
+* 被迁移的codis需要设置client-output-buffer-limit解除限制，否则会断开链接;
+* 被迁移节点需要预留同等迁移数据量的内存冗余;
+* 在执行数据迁移前后，建议进行compact;
+* 不支持多db（感觉必要性不大，暂时没支持;
+* 部分key的过期时间可能推后。
+
+## 迁移过程可视化
+推荐提前配置监控大盘，可以对迁移过程有可视化的掌握。
+
+## 验证迁移结果
+
+推荐使用阿里开源的redis-full-check工具
+

bin/codis2pika.toml

@@ -0,0 +1,51 @@
+[source]
+type = "sync" 
+address = "192.168.12.44:6379"
+username = "" # keep empty if not using ACL
+password = "" # keep empty if no authentication is required
+tls = false
+elasticache_psync = "" # using when source is ElastiCache. ref: https://github.com/alibaba/RedisShake/issues/373
+
+[target]
+type = "standalone" # only standalone 
+# When the target is a cluster, write the address of one of the nodes.
+# codis2pika will obtain other nodes through the `cluster nodes` command.
+address = "192.168.12.43:9221"
+username = "" # keep empty if not using ACL
+password = "" # keep empty if no authentication is required
+tls = false
+
+
+[advanced]
+dir = "data"
+
+# runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
+ncpu = 4
+
+# pprof port, 0 means disable
+pprof_port = 0
+
+# log
+log_file = "codis2pika.log"
+log_level = "info" # debug, info or warn
+log_interval = 5 # in seconds
+
+# codis2pika gets key and value from rdb file, and uses RESTORE command to
+# create the key in target redis. Redis RESTORE will return a "Target key name
+# is busy" error when key already exists. You can use this configuration item
+# to change the default behavior of restore:
+# panic:   codis2pika will stop when meet "Target key name is busy" error.
+# rewrite: codis2pika will replace the key with new value.
+# ignore:  codis2pika will skip restore the key when meet "Target key name is busy" error.
+rdb_restore_command_behavior = "skip" # panic, rewrite or skip
+
+# pipeline
+pipeline_count_limit = 1024
+
+# Client query buffers accumulate new commands. They are limited to a fixed
+# amount by default. This amount is normally 1gb.
+target_redis_client_max_querybuf_len = 1024_000_000
+
+# In the Redis protocol, bulk requests, that are, elements representing single
+# strings, are normally limited to 512 mb.
+target_redis_proto_max_bulk_len = 512_000_000

build.sh

@@ -0,0 +1,36 @@
+#!/bin/bash
+
+set -e
+
+echo "[ BUILD RELEASE ]"
+BIN_DIR=$(pwd)/bin/
+rm -rf "$BIN_DIR"
+mkdir -p "$BIN_DIR"
+
+# build the current platform
+echo "try build for current platform"
+go build -v -trimpath  -gcflags '-N -l' -o "$BIN_DIR/codis2pika" "./cmd/codis2pika"
+echo "build success"
+
+for g in "linux" "darwin"; do
+  for a in "amd64" "arm64"; do
+    echo "try build GOOS=$g GOARCH=$a"
+    export GOOS=$g
+    export GOARCH=$a
+    go build -v -trimpath  -gcflags '-N -l' -o "$BIN_DIR/codis2pika-$g-$a" "./cmd/codis2pika"
+    unset GOOS
+    unset GOARCH
+    echo "build success"
+  done
+done
+
+cp codis2pika.toml "$BIN_DIR"
+
+if [ "$1" == "dist" ]; then
+  echo "[ DIST ]"
+  cd bin
+  cp -r ../filters ./
+  tar -czvf ./codis2pika.tar.gz ./codis2pika.toml  ./codis2pika-* ./filters
+  rm -rf ./filters
+  cd ..
+fi