Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

数据分散,根据时间字段报错 #1070

Open
czxin788 opened this issue Sep 12, 2024 · 0 comments
Open

数据分散,根据时间字段报错 #1070

czxin788 opened this issue Sep 12, 2024 · 0 comments

Comments

@czxin788
Copy link

Description

按时间字段,将数据分散到不同的mysql实例,报错

 tail -f dtle.log 
2024-09-12T04:42:31.680Z [INFO]  applier: doFullCopy: loop: rowCopyComplete: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  extractor: initBinlogReader: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  reader: Connecting binlog streamer: job=dtle-demo-src-dst2 file=bin.000004 pos=3297 gtid=4d7bd416-6f5a-11ef-9c56-0242ac120004:1-22
2024-09-12T04:42:31.684Z [INFO]  extractor: monitor and update job status: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  extractor: Beginning streaming: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  reader: Rotate to next binlog: job=dtle-demo-src-dst2 name=bin.000004
2024-09-12T04:42:31.685Z [ERROR] extractor: onError: job=dtle-demo-src-dst2 err="StreamEvents encountered unexpected error: 'where' predicate does not eval to bool. demo.demo_tbl create_time>='2023-01-01'" hasShutdown=false
2024-09-12T04:42:31.685Z [INFO]  extractor: extractor shutdown: job=dtle-demo-src-dst2
2024-09-12T04:42:31.687Z [INFO]  DestroyTask: id=4ea0319f-4101-d066-d05a-9bcafe20f734/src/9979b45c
2024-09-12T04:42:31.785Z [INFO]  extractor: Shutting down: job=dtle-demo-src-dst2

Steps to reproduce the issue

  1. 创建表结构
mysql> show create table demo_tbl \G
*************************** 1. row ***************************
       Table: demo_tbl
Create Table: CREATE TABLE `demo_tbl` (
  `id` int(11) NOT NULL,
  `name` varchar(12) DEFAULT NULL,
  `create_time` datetime DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)
  1. 源表数据
mysql> select * from demo_tbl;
+----+------+---------------------+
| id | name | create_time         |
+----+------+---------------------+
|  1 | a    | 2023-01-02 14:00:00 |
|  2 | b    | 2023-01-02 14:01:00 |
|  3 | c    | 2023-01-02 14:02:00 |
|  4 | d    | 2024-01-02 14:02:00 |
|  5 | e    | 2024-01-03 14:02:00 |
|  6 | e    | 2024-06-03 14:02:00 |
|  7 | e    | 2024-06-03 14:02:00 |
|  8 | e    | 2024-06-03 14:02:00 |
|  9 | e    | 2022-06-03 14:02:00 |
| 10 | e    | 2022-06-03 14:02:00 |
| 11 | e    | 2024-06-03 14:02:00 |
+----+------+---------------------+
11 rows in set (0.00 sec)
  1. dtle任务文件
    cat src_dst1.json
{
  "Job": {
    "ID": "dtle-demo-src-dst1",
    "Datacenters": ["dc1"],
    "TaskGroups": [{
        "Name": "src",
        "Tasks": [{
          "Name": "src",
          "Driver": "dtle",
          "Config": {
            "Gtid": "",
            "ReplicateDoDb": [{
              "TableSchema": "demo",
              "Tables": [{
                "TableName": "demo_tbl",
                "Where":"create_time<'2023-01-01'"
              }]
            }],
            "SrcConnectionConfig": {
              "Host": "mysql-src",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            },
            "DestConnectionConfig": {
              "Host": "mysql-dst1",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            }
          }
        }]
      }, {
        "Name": "dest",
        "Tasks": [{
          "Name": "dest",
          "Driver": "dtle",
          "Config": {
            "DestType": "mysql"
          }
        }]
    }]
  }
}

cat src_dst2.json

{
  "Job": {
    "ID": "dtle-demo-src-dst2",
    "Datacenters": ["dc1"],
    "TaskGroups": [{
        "Name": "src",
        "Tasks": [{
          "Name": "src",
          "Driver": "dtle",
          "Config": {
            "Gtid": "",
            "ReplicateDoDb": [{
              "TableSchema": "demo",
              "Tables": [{
                "TableName": "demo_tbl",
                "Where":"create_time>='2023-01-01'"
              }]
            }],
            "SrcConnectionConfig": {
              "Host": "mysql-src",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            },
            "DestConnectionConfig": {
              "Host": "mysql-dst2",
              "Port": 3306,
              "User": "root",
              "Password": "pass"
            }
          }
        }]
      }, {
        "Name": "dest",
        "Tasks": [{
          "Name": "dest",
          "Driver": "dtle",
          "Config": {
            "DestType": "mysql"
          }
        }]
    }]
  }
}

Describe the results you received

 tail -f dtle.log 
2024-09-12T04:42:31.680Z [INFO]  applier: doFullCopy: loop: rowCopyComplete: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  extractor: initBinlogReader: job=dtle-demo-src-dst2
2024-09-12T04:42:31.680Z [INFO]  reader: Connecting binlog streamer: job=dtle-demo-src-dst2 file=bin.000004 pos=3297 gtid=4d7bd416-6f5a-11ef-9c56-0242ac120004:1-22
2024-09-12T04:42:31.684Z [INFO]  extractor: monitor and update job status: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  extractor: Beginning streaming: job=dtle-demo-src-dst2
2024-09-12T04:42:31.684Z [INFO]  reader: Rotate to next binlog: job=dtle-demo-src-dst2 name=bin.000004
2024-09-12T04:42:31.685Z [ERROR] extractor: onError: job=dtle-demo-src-dst2 err="StreamEvents encountered unexpected error: 'where' predicate does not eval to bool. demo.demo_tbl create_time>='2023-01-01'" hasShutdown=false
2024-09-12T04:42:31.685Z [INFO]  extractor: extractor shutdown: job=dtle-demo-src-dst2
2024-09-12T04:42:31.687Z [INFO]  DestroyTask: id=4ea0319f-4101-d066-d05a-9bcafe20f734/src/9979b45c
2024-09-12T04:42:31.785Z [INFO]  extractor: Shutting down: job=dtle-demo-src-dst2

Describe the results you expected

Output of ./dtle version:**

dtle 4.23.04.3

(paste your output here)

Additional information

(e.g. issue happens only occasionally)

Additional details (log, config, job config etc):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant