Assertion Error #610

MintooJinnu · 2024-02-17T05:03:33Z

Version

Please specify the versions you are using. Exact version numbers are preferred.

Pymyrepl (e.g., 1.0.2):
OS (e.g., Ubuntu 18.04):
Database and version (Remove unnecessary options):
- MySQL: 10.23
- MariaDB: 5.3
binlog_row_metadata: None

Symptoms

I have a prod db, where somw insert queries are running on top of it, I am trying to replicate in snowflake. But while reading the insert event, while getting binlog_event.event.rows throwing

Assertion Error
Expected = 111. Actual = 46. Position=130. Data=176

And I am getting the assertion error for column type varchar,

Steps to Reproduce

I cannot help you in reproducing as I am testing in prod environment

I tried with many version degrading and upgrading with all the modules such as pymysql, mysql-replication, s3, etc..

The text was updated successfully, but these errors were encountered:

sean-k1 · 2024-02-17T14:16:16Z

What is your Pymyrepli version? @MintooJinnu

MintooJinnu · 2024-02-17T14:22:01Z

I tried with multiple mysql-replication @sean-k1 , tried with 0.24, 0.25, 0.26 and 1.0.0, 0.1.0, 0.45.0, 1.0.6 and 1.0.5

sean-k1 · 2024-02-17T14:35:20Z

@MintooJinnu
If you can't tell me which rows are causing the problem, there's nothing I can do to help you.

MintooJinnu · 2024-02-17T14:37:32Z

Sure let me get you the rows @sean-k1

MintooJinnu · 2024-02-17T15:26:46Z

I have a huge insert query with multiple sub queries, Query 1 is just an insert and Query 2 is a select query

Insert into table_1 columns(.....) SELECT......

Query2 fetching records:

A	B	C	D	E	F	G	H	I	J	K	L	M	N	O	P	Q	R	S	T	U	V
2020-02-12 12:00:00	10010	1	1	1	uidjjlsnckepcbieb eiuhfief cb23yyd	vfugcjecfrifwh-cninwcbjvrdeciw	1234567	2020-02-12 12:00:00	NULL	NULL	NULL	123456789951753852456321654987	4njkfneodnkemfkekdmownfkfrnfirwlo	A1.2.7849383874.273888393	192.168.0.0.1	Nokia	Windows 10/Server 2020	Google Chrome	false	NULL	0

Also attached the excel sheet for your reference
sample_data.xlsx

The Schema of table_1:
A DATETIME(3), PRIMARY KEY, int(11) UN PK
B INT(11) UN PK
C INT(11) PK
D INT(11) UN PK
E INT(11) UN
F VARCHAR(250)
G VARCHAR(100) PK
H BIGINT(20)
I DATETIME(3)
J VARCHAR(255)
K VARCHAR(50)
L VARCHAR(9)
M VARCHAR(39)
N VARCHAR(32)
O VARCHAR(28)
P VARCHAR(20)
Q VARCHAR(50)
R VARCHAR(50)
S VARCHAR(50)
T VARCHAR(11)
U VARCHAR(6)
V TINYINT(1)

This is all happening in Maria DB @sean-k1

MintooJinnu · 2024-02-18T03:52:17Z

Adding an other piece of information, which might be helpful to you is the query2(subquery) is having PARTITION BY also, please let me know if that might create some problem in reading string in pascal method @sean-k1

MintooJinnu · 2024-02-18T21:18:37Z

I observed one more thing, ie. It is reading the other column data suddenly after reading data of column F @sean-k1. In column F receiving all the data of other columns. This is happening only to this table.

sean-k1 · 2024-02-19T10:10:24Z

mysql> CREATE TABLE YourTableName (
    ->     A DATETIME(3) PRIMARY KEY,
    ->     B INT(11) UNSIGNED NOT NULL,
    ->     C INT(11) NOT NULL,
    ->     D INT(11) UNSIGNED NOT NULL,
    ->     E INT(11) UNSIGNED,
    ->     F VARCHAR(250),
    ->     G VARCHAR(100) NOT NULL,
    ->     H BIGINT(20),
    ->     I DATETIME(3),
    ->     J VARCHAR(255),
    ->     K VARCHAR(50),
    ->     L VARCHAR(9),
    ->     M VARCHAR(39),
    ->     N VARCHAR(32),
    ->     O VARCHAR(28),
    ->     P VARCHAR(20),
    ->     Q VARCHAR(50),
    ->     R VARCHAR(50),
    ->     S VARCHAR(50),
    ->     T VARCHAR(11),
    ->     U VARCHAR(6),
    ->     V TINYINT(1)
    -> );
Query OK, 0 rows affected, 6 warnings (0.01 sec)

mysql> INSERT INTO YourTableName (A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V)
    -> VALUES (
    ->     '2020-02-12 12:00:00', -- A
    ->     10010, -- B
    ->     1, -- C
    ->     1, -- D
    ->     1, -- E
    ->     'uidjjlsnckepcbieb eiuhfief cb23yyd', -- F
    ->     'vfugcjecfrifwh-cninwcbjvrdeciw', -- G
    ->     1234567, -- H
    ->     '2020-02-12 12:00:00', -- I
    ->     NULL, -- J
    ->     NULL, -- K
    ->     NULL, -- L
    ->     '123456789951753852456321654987', -- M
    ->     '4njkfneodnkemfkekdmownfkfrnfirwo', -- N
    ->     'A1.2.7849383874.273888393', -- O
    ->     '192.168.0.0.1', -- P
    ->     'Nokia', -- Q
    ->     'Windows 10/Server 2020', -- R
    ->     'Google Chrome', -- S
    ->     false, -- T
    ->     NULL, -- U
    ->     0 -- V
    -> );
Query OK, 1 row affected (0.01 sec)

your Data '4njkfneodnkemfkekdmownfkfrnfirwo', -- N (33) is larger than 32 so i erase last word
Is it right your table and insert query?
@MintooJinnu

sean-k1 · 2024-02-19T10:11:20Z

i did not find error in mysql 8.0 enviornment
@MintooJinnu


=== QueryEvent ===
Date: 2024-02-19T10:08:01
Log position: 2322
Event size: 570
Read bytes: 570
Schema: b'test'
Execution time: 0
Query: CREATE TABLE YourTableName (
    A DATETIME(3) PRIMARY KEY,
    B INT(11) UNSIGNED NOT NULL,
    C INT(11) NOT NULL,
    D INT(11) UNSIGNED NOT NULL,
    E INT(11) UNSIGNED,
    F VARCHAR(250),
    G VARCHAR(100) NOT NULL,
    H BIGINT(20),
    I DATETIME(3),
    J VARCHAR(255),
    K VARCHAR(50),
    L VARCHAR(9),
    M VARCHAR(39),
    N VARCHAR(32),
    O VARCHAR(28),
    P VARCHAR(20),
    Q VARCHAR(50),
    R VARCHAR(50),
    S VARCHAR(50),
    T VARCHAR(11),
    U VARCHAR(6),
    V TINYINT(1)
)

=== QueryEvent ===
Date: 2024-02-19T10:08:35
Log position: 2484
Event size: 60
Read bytes: 60
Schema: b'test'
Execution time: 0
Query: BEGIN

=== TableMapEvent ===
Date: 2024-02-19T10:08:35
Log position: 2655
Event size: 148
Read bytes: 148
Table id: 104
Schema: test
Table: yourtablename
Columns: 22
=== OptionalMetaData ===
unsigned_column_list: [False, True, False, True, True, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False, False]
default_charset_collation: 255
charset_collation: {}
column_charset: []
column_name_list: ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V']
set_str_value_list : []
set_enum_str_value_list : []
geometry_type_list : []
simple_primary_key_list: [0]
primary_keys_with_prefix: {}
visibility_list: [True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True, True]
charset_collation_list: [255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255]
enum_and_set_collation_list: []

=== WriteRowsEvent ===
Date: 2024-02-19T10:08:35
Log position: 2951
Event size: 273
Read bytes: 14
Table: test.yourtablename
Affected columns: 22
Changed rows: 1
Column Name Information Flag: True
Values:
--
* A : 2020-02-12 12:00:00
* B : 10010
* C : 1
* D : 1
* E : 1
* F : uidjjlsnckepcbieb eiuhfief cb23yyd
* G : vfugcjecfrifwh-cninwcbjvrdeciw
* H : 1234567
* I : 2020-02-12 12:00:00
* J : None (null)
* K : None (null)
* L : None (null)
* M : 123456789951753852456321654987
* N : 4njkfneodnkemfkekdmownfkfrnfirwo
* O : A1.2.7849383874.273888393
* P : 192.168.0.0.1
* Q : Nokia
* R : Windows 10/Server 2020
* S : Google Chrome
* T : 0
* U : None (null)
* V : 0

=== XidEvent ===
Date: 2024-02-19T10:08:35
Log position: 2982
Event size: 8
Read bytes: 8
Transaction ID: 40

MintooJinnu · 2024-02-19T13:40:37Z

There is some problem with the data @sean-k1 because I was running very huge query, and that is not reading the correct data, unpacking is the problem actually. Let me send you the exact query by morphing it.

MintooJinnu · 2024-02-19T22:08:51Z

value[name] = self.__read_string(1, column) it is throwing a value which is not in perfect way.
Basically we are reading a session id getting the value as 'ud7ehuwkhdhiejduudjjjej.hsijd.jsjsj.\x1c5468-a♤Á\x19\x00.....^P93F0

Please help me on this @sean-k1

sean-k1 · 2024-02-22T07:57:14Z

@MintooJinnu
Please Write the problematic SQL statement query

MintooJinnu · 2024-02-24T07:30:37Z

I'm able to fix the error but I need a small help @sean-k1 I have a table with column DDL declaration as

'DOB datetime(3) NOT NULL DEFUALT '0000-00-00 00:00:00.000'

Now while extracting the rows of writerows event reading column data and it is going to method _read_datetime()

Getting a large number in the value = self.packet.uint64()

How to get the datetime stamp and what exactly should be there in struct.unpack()

I tried with struct.unpack('I3s', self.read(7))
Which is working but after converting to datetime the year is getting 2038 and month and day is also not accurate.
I tried with struct.unpack('<Q', self.read(8)

It is getting very larhe

Can you please help me on this @sean-k1

dongwook-chan · 2024-04-19T13:28:30Z

@MintooJinnu
Could you please provide

table DDL (use anonymous column names)
query that's causing error
error stack strace
packet dump (if possible)

I can not reproduce the issue without these information.
Any of above does NOT reveal any of your data in prod db.
So please feel safe to share the info.

MintooJinnu mentioned this issue Feb 18, 2024

Result length not requested length #226

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Assertion Error #610

Assertion Error #610

MintooJinnu commented Feb 17, 2024

sean-k1 commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 17, 2024

sean-k1 commented Feb 17, 2024

MintooJinnu commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 18, 2024

MintooJinnu commented Feb 18, 2024 •

edited

Loading

sean-k1 commented Feb 19, 2024 •

edited

Loading

sean-k1 commented Feb 19, 2024

MintooJinnu commented Feb 19, 2024

MintooJinnu commented Feb 19, 2024 •

edited

Loading

sean-k1 commented Feb 22, 2024 •

edited

Loading

MintooJinnu commented Feb 24, 2024

dongwook-chan commented Apr 19, 2024

Assertion Error #610

Assertion Error #610

Comments

MintooJinnu commented Feb 17, 2024

Version

Symptoms

Steps to Reproduce

sean-k1 commented Feb 17, 2024 • edited Loading

MintooJinnu commented Feb 17, 2024

sean-k1 commented Feb 17, 2024

MintooJinnu commented Feb 17, 2024 • edited Loading

MintooJinnu commented Feb 17, 2024 • edited Loading

MintooJinnu commented Feb 18, 2024

MintooJinnu commented Feb 18, 2024 • edited Loading

sean-k1 commented Feb 19, 2024 • edited Loading

sean-k1 commented Feb 19, 2024

MintooJinnu commented Feb 19, 2024

MintooJinnu commented Feb 19, 2024 • edited Loading

sean-k1 commented Feb 22, 2024 • edited Loading

MintooJinnu commented Feb 24, 2024

dongwook-chan commented Apr 19, 2024

sean-k1 commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 17, 2024 •

edited

Loading

MintooJinnu commented Feb 18, 2024 •

edited

Loading

sean-k1 commented Feb 19, 2024 •

edited

Loading

MintooJinnu commented Feb 19, 2024 •

edited

Loading

sean-k1 commented Feb 22, 2024 •

edited

Loading