Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lightning: panic when de-duplication #34163

Closed
D3Hunter opened this issue Apr 21, 2022 · 8 comments · Fixed by #36805
Closed

lightning: panic when de-duplication #34163

D3Hunter opened this issue Apr 21, 2022 · 8 comments · Fixed by #36805
Assignees
@ForwardStar
Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@D3Hunter
Copy link
Contributor

Bug Report

lightning 5.4.0 panic when de-duplication when there's previous leftted data in conflict_error_v1 and the table ddl is different with previous task which also has this table

2022-04-21 23:54:58	
github.com/pingcap/tidb/br/pkg/lightning/errormanager.(*ErrorManager).ResolveAllConflictKeys.func2(0x0, 0x0)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1443 +0xb8
2022-04-21 23:54:58	
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*local).ResolveDuplicateRows.func2(0x4760048, 0xc001949c00, 0xc001b30000, 0x3e8, 0x400, 0x0, 0x200)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1500 +0x4e7
2022-04-21 23:54:58	
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*local).deleteDuplicateRows(0xc00155a300, 0x4760048, 0xc001949c00, 0xc001949b80, 0xc001b30000, 0x3e8, 0x400, 0xc001949bc0, 0x0, 0x0)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/kv/kv2sql.go:68 +0x85
2022-04-21 23:54:58	
github.com/pingcap/tidb/br/pkg/lightning/backend/kv.(*TableKVDecoder).IterRawIndexKeys(0xc001949bc0, 0x4794278, 0xc01bcd5260, 0xc000b12160, 0x15d, 0x15d, 0xc0019f3a40, 0x133e065, 0xc000daa898)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/kv/kv2sql.go:51 +0xfb
2022-04-21 23:54:58	
github.com/pingcap/tidb/br/pkg/lightning/backend/kv.(*TableKVDecoder).DecodeRawRowData(0xc001949bc0, 0x4794278, 0xc01bcd5260, 0xc000b12160, 0x15d, 0x15d, 0xc0018d3a00, 0x42e72f0, 0x134915b, 0x364b0e5, ...)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/table/tables/tables.go:996 +0x806
2022-04-21 23:54:58	
github.com/pingcap/tidb/table/tables.DecodeRawRowData(0x47ac658, 0xc143e46c00, 0xc00083c000, 0x4794278, 0xc01bcd5260, 0xc0030541a0, 0x33, 0x33, 0xc000b12160, 0x15d, ...)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/tablecodec/tablecodec.go:465 +0x6e
2022-04-21 23:54:58	
github.com/pingcap/tidb/tablecodec.DecodeRowToDatumMap(0xc000b12160, 0x15d, 0x15d, 0xc000fbf628, 0x672d4e0, 0x1ae3, 0xc000fbf518, 0x1354393)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/tablecodec/tablecodec.go:401 +0x22a
2022-04-21 23:54:58	
github.com/pingcap/tidb/tablecodec.DecodeRowWithMapNew(0xc000b12160, 0x15d, 0x15d, 0xc000fbf628, 0x672d4e0, 0xc001b83980, 0x0, 0x3bfeea0, 0xc0003acfb0)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/util/rowcodec/decoder.go:83 +0x315
2022-04-21 23:54:58	
github.com/pingcap/tidb/util/rowcodec.(*DatumMapDecoder).DecodeToDatumMap(0xc0019f3370, 0xc000b12160, 0x15d, 0x15d, 0xc001b83980, 0x18, 0x8, 0xc000cebe00)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/util/rowcodec/decoder.go:127 +0x969
2022-04-21 23:54:58	
github.com/pingcap/tidb/util/rowcodec.(*DatumMapDecoder).decodeColDatum(0xc000fbf370, 0xc000c54530, 0xc000b12278, 0x4, 0x45, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/util/codec/decimal.go:59 +0x1b2
2022-04-21 23:54:58	
github.com/pingcap/tidb/util/codec.DecodeDecimal(0xc000b1227a, 0x2, 0x43, 0xd, 0x7faf3f0a9590, 0x0, 0x1aea, 0xc000fbf0f0, 0x1354393, 0x7faf3ec37db8, ...)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/types/mydecimal.go:1404 +0x933
2022-04-21 23:54:58	
github.com/pingcap/tidb/types.(*MyDecimal).FromBin(0xc0018d13e0, 0xc000b1227a, 0x2, 0x43, 0x52, 0x55, 0xc001bd9140, 0x19509f3, 0xc001d80940)
2022-04-21 23:54:58	
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/types/mydecimal.go:1486
2022-04-21 23:54:58	
github.com/pingcap/tidb/types.DecimalBinSize(...)
2022-04-21 23:54:58	
goroutine 40146 [running]:
2022-04-21 23:54:58	

2022-04-21 23:54:58	
panic: runtime error: index out of range [-3]

root cause is this sql didn't specify taskid, so data of previous task is queried, when decoding using current ddl, panic.

tidb/br/pkg/lightning/errormanager/errormanager.go

Lines 110 to 114 in 55f3b24

selectConflictKeys = `
SELECT _tidb_rowid, raw_handle, raw_row
FROM %s.` + conflictErrorTableName + `
WHERE table_name = ? AND _tidb_rowid >= ? and _tidb_rowid < ?
ORDER BY _tidb_rowid LIMIT ?;

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

success

3. What did you see instead (Required)

panic

4. What is your TiDB version? (Required)

5.3.1
@D3Hunter D3Hunter added the type/bug The issue is confirmed as a bug. label Apr 21, 2022
@D3Hunter
Copy link
Contributor Author

/component lightning

@ti-chi-bot ti-chi-bot added the component/lightning This issue is related to Lightning of TiDB. label Apr 21, 2022
@D3Hunter
Copy link
Contributor Author

/severity major

@ti-chi-bot ti-chi-bot added severity/major may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 labels Apr 21, 2022
@D3Hunter
Copy link
Contributor Author

/affects-5.4

@D3Hunter
Copy link
Contributor Author

affects-5.4

@D3Hunter
Copy link
Contributor Author

/severity moderate

@sleepymole sleepymole added affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.0 and removed may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-6.0 severity/major labels Apr 22, 2022
@niubell
Copy link
Contributor

niubell commented Apr 22, 2022

/assign D3Hunter

@niubell
Copy link
Contributor

niubell commented Jul 26, 2022

/assign ForwardStar

@niubell
Copy link
Contributor

niubell commented Jul 26, 2022

/unassign D3Hunter

@ForwardStar ForwardStar mentioned this issue Aug 2, 2022
Closed
12 tasks
ForwardStar added a commit to ForwardStar/tidb that referenced this issue Aug 2, 2022
@ForwardStar
Fix pingcap#34163
c934e60
@ForwardStar ForwardStar mentioned this issue Aug 2, 2022
Merged
12 tasks
@D3Hunter D3Hunter added affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 and removed affects-6.0 affects-6.2 labels Aug 12, 2022
ti-chi-bot pushed a commit that referenced this issue Aug 19, 2022
@ForwardStar
lightning: Fix panic when downstream table schema has changed (#36805)
8511b9b 
close #34163
This was referenced Aug 19, 2022
Merged
Merged
Closed
ti-chi-bot pushed a commit that referenced this issue Aug 24, 2022
@ti-srebot
lightning: Fix panic when downstream table schema has changed (#36805) (
a5fd7d1 
#37233)

close #34163
@niubell niubell mentioned this issue Aug 29, 2022
Merged
12 tasks
ti-chi-bot pushed a commit that referenced this issue Sep 16, 2022
@ti-srebot
lightning: Fix panic when downstream table schema has changed (#36805) (
5795341 
#37232)

close #34163
This was referenced Oct 8, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ForwardStar ForwardStar

Labels
affects-5.4 This bug affects the 5.4.x(LTS) versions. affects-6.1 This bug affects the 6.1.x(LTS) versions. affects-6.2 component/lightning This issue is related to Lightning of TiDB. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
5 participants
@D3Hunter @niubell @sleepymole @ForwardStar @ti-chi-bot