Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

同步过程中报错 read aof file failed #915

Closed
yuchangrui opened this issue Jan 8, 2025 · 2 comments · Fixed by #917
Closed

同步过程中报错 read aof file failed #915

yuchangrui opened this issue Jan 8, 2025 · 2 comments · Fixed by #917
Labels
type: question Further information is requested

Comments

@yuchangrui
Copy link

yuchangrui commented Jan 8, 2025

问题描述(Issue Description)

请在这里简要描述你遇到的问题。

Please provide a brief description of the issue you encountered.

托管redis集群1主1从,热迁移至 托管redis1主1从
运行节点 CentOS Linux release 7.9.2009 (Core)
完整日志在附件
shake.log

环境信息(Environment)

  • RedisShake 版本(RedisShake Version):4.3.1
  • Redis 源端版本(Redis Source Version):6.2.5
  • Redis 目的端版本(Redis Destination Version):7.2.6
  • Redis 部署方式(standalone/cluster/sentinel):源端cluster,目的端standalone
  • 是否在云服务商实例上部署(Deployed on Cloud Provider):源和目标都是云服务商托管

日志信息(Logs)

{"level":"info","time":"2025-01-08T10:59:21+08:00","message":"read_count=[12419692], read_ops=[387.55], write_count=[12419692], write_ops=[387.55], src-0, syncing aof, diff=[0]"} {"level":"info","time":"2025-01-08T10:59:26+08:00","message":"read_count=[12421140], read_ops=[289.46], write_count=[12421140], write_ops=[289.46], src-0, syncing aof, diff=[519]"} {"level":"info","time":"2025-01-08T10:59:31+08:00","message":"read_count=[12422671], read_ops=[306.38], write_count=[12422671], write_ops=[306.38], src-0, syncing aof, diff=[73]"} {"level":"info","time":"2025-01-08T10:59:36+08:00","message":"read_count=[12424120], read_ops=[289.77], write_count=[12424120], write_ops=[289.77], src-0, syncing aof, diff=[0]"} {"level":"info","time":"2025-01-08T10:59:41+08:00","message":"read_count=[12425899], read_ops=[355.77], write_count=[12425899], write_ops=[355.77], src-0, syncing aof, diff=[0]"} {"level":"info","time":"2025-01-08T10:59:46+08:00","message":"read_count=[12427379], read_ops=[296.01], write_count=[12427379], write_ops=[296.01], src-0, syncing aof, diff=[0]"} {"level":"info","time":"2025-01-08T10:59:51+08:00","message":"read_count=[12428982], read_ops=[320.58], write_count=[12428982], write_ops=[320.58], src-0, syncing aof, diff=[0]"} {"level":"info","time":"2025-01-08T10:59:56+08:00","message":"read_count=[12431698], read_ops=[543.21], write_count=[12431698], write_ops=[543.21], src-0, syncing aof, diff=[144]"} {"level":"info","time":"2025-01-08T11:00:01+08:00","message":"read_count=[12434209], read_ops=[502.19], write_count=[12434209], write_ops=[502.19], src-0, syncing aof, diff=[0]"} {"level":"error","time":"2025-01-08T11:00:05+08:00","message":"[reader_172.20.0.17_6379] read aof file failed. error=[unexpected EOF]\n\t\t\tRedisShake/internal/reader/sync_standalone_reader.go:470 -> (*syncStandaloneReader).sendAOF()\n\t\t\tRedisShake/internal/reader/sync_standalone_reader.go:143 -> (*syncStandaloneReader).StartReadWithPSync.func1()\n\t\t\truntime/asm_amd64.s:1650 -> goexit()"}

如果有错误日志或其他相关日志,请在这里提供。

If there are any error logs or other relevant logs, please provide them here.

其他信息(Additional Information)

配置文件
[sync_reader]
cluster = true # Set to true if the source is a Redis cluster
address = "xx.xx.xx.xx:xx" # For clusters, specify the address of any cluster node; use the master or slave address in master-slave mode
username = "" # Keep empty if ACL is not in use
password = "" # Keep empty if no authentication is required
tls = false # Set to true to enable TLS if needed
sync_rdb = true # Set to false if RDB synchronization is not required
sync_aof = true # Set to false if AOF synchronization is not required
prefer_replica = false # Set to true to sync from a replica node
try_diskless = false # Set to true for diskless sync if the source has repl-diskless-sync=yes

[redis_writer]
cluster = false # set to true if target is a redis cluster
address = "xx.xx.xx.xx:xx" # when cluster is true, set address to one of the cluster node
username = "xx" # keep empty if not using ACL
password = "xx" # keep empty if no authentication is required
tls = true
off_reply = false # turn off the server reply

[filter]
allow_keys = []
allow_key_prefix = []
allow_key_suffix = []
allow_key_regex = []

block_keys = []
block_key_prefix = []
block_key_suffix = []
block_key_regex = []

allow_db = []
block_db = []

allow_command = []
block_command = []

allow_command_group = []
block_command_group = []

function = ""

[advanced]
dir = "data"
ncpu = 0 # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
pprof_port = 0 # pprof port, 0 means disable
status_port = 0 # status port, 0 means disable

log_file = "shake.log"
log_level = "info" # debug, info or warn
log_interval = 5 # in seconds

rdb_restore_command_behavior = "panic" # panic, rewrite or skip

pipeline_count_limit = 1024

target_redis_client_max_querybuf_len = 1073741824 # 1GB in bytes

target_redis_proto_max_bulk_len = 512_000_000

aws_psync = "" # example: aws_psync = "10.0.0.1:6379@nmfu2sl5osync,10.0.0.1:6379@xhma21xfkssync"

empty_db_before_sync = false

[module]
target_mbbloom_version = 20603

请提供任何其他相关的信息,如配置文件、错误信息或截图等。

Please provide any additional information, such as configuration files, error messages, or screenshots.

@yuchangrui yuchangrui added the type: question Further information is requested label Jan 8, 2025
@liuguoshun
Copy link

liuguoshun commented Jan 13, 2025

遇到了类似的问题

RedisShake 版本(RedisShake Version):4.3.1
Redis 源端版本(Redis Source Version):6.2.5
Redis 目的端版本(Redis Destination Version):6.2.5
Redis 部署方式(standalone/cluster/sentinel):源端cluster,目的端cluster
是否在云服务商实例上部署(Deployed on Cloud Provider):源和目标都是容器化部署

日志信息(Logs)
{"level":"info","time":"2025-01-13T17:00:47+08:00","message":"read_count=[3162330377], read_ops=[1032092.76], write_count=[3162330303], write_ops=[1032092.56], src-65, syncing rdb, size=[3.7 GiB/5.7 GiB]"}
{"level":"error","time":"2025-01-13T17:00:52+08:00","message":"[reader_xxxxx_xxxx] read aof file failed. error=[redis: invalid reply: "\x1aی\x01\n"]\n\t\t\tRedisShake/internal/reader/sync_standalone_reader.go:470 -> (*syncStandaloneReader).sendAOF()\n\t\t\tRedisShake/internal/reader/sync_standalone_reader.go:143 -> (*syncStandaloneReader).StartReadWithPSync.func1()\n\t\t\truntime/asm_amd64.s:1650 -> goexit()"}

配置文件:

[sync_reader]
cluster = true # Set to true if the source is a Redis cluster
address = "xxxx:xx " # For clusters, specify the address of any cluster node; use the master or slave address in master-slave mode
username = "" # Keep empty if ACL is not in use
password = "" # Keep empty if no authentication is required
tls = false # Set to true to enable TLS if needed
sync_rdb = true # Set to false if RDB synchronization is not required
sync_aof = true # Set to false if AOF synchronization is not required
prefer_replica = false # Set to true to sync from a replica node
try_diskless = false # Set to true for diskless sync if the source has repl-diskless-sync=yes
[redis_writer]
cluster = true # set to true if target is a redis cluster
address = "xxxx:xx" # when cluster is true, set address to one of the cluster node
username = "" # keep empty if not using ACL
password = "" # keep empty if no authentication is required
tls = false
off_reply = false # turn off the server reply
[filter]
allow_keys = []
allow_key_prefix = []
allow_key_suffix = []
allow_key_regex = []
block_keys = []
block_key_prefix = []
block_key_suffix = []
block_key_regex = []
allow_db = []
block_db = []
allow_command = []
block_command = []
allow_command_group = []
block_command_group = []
function = ""
[advanced]
dir = "data_fs1"
ncpu = 0 # runtime.GOMAXPROCS, 0 means use runtime.NumCPU() cpu cores
pprof_port = 0 # pprof port, 0 means disable
status_port = 0 # status port, 0 means disable
log_file = "shake_fs1.log"
log_level = "info" # debug, info or warn
log_interval = 5 # in seconds
rdb_restore_command_behavior = "panic" # panic, rewrite or skip
pipeline_count_limit = 1024
target_redis_client_max_querybuf_len = 1073741824 # 1GB in bytes
target_redis_proto_max_bulk_len = 512_000_000
aws_psync = "" # example: aws_psync = "10.0.0.1:6379@nmfu2sl5osync,10.0.0.1:6379@xhma21xfkssync"
empty_db_before_sync = false
[module]
target_mbbloom_version = 20603

@MrHeroJack
Copy link

UP2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants