[Bug]: lots of error "cannot commit a orphan transaction" after dn crashed 26 times and recovered at last during stability test on distributed mode

[aressu1985]

Is there an existing issue for the same bug?

• I have checked the existing issues.

Branch Name

2.0-dev

Commit ID

ec75b49

Other Environment Information

- Hardware parameters:
3*CN: 16C 64G
1*DN: 16C 64G
3*LOG: 4C 16G
2*PROXY: 3C 6G
- OS type:
- Others:

Actual Behavior

lots of error "cannot commit a orphan transaction" after dn crashed 26 times and recovered at last during stability test on distributed mode

TN was crashed by oom from [2024-12-28 16:58:14] , and crashed for 26 times, recovery at [2024-12-28 20:02:49]
but from [2024-12-28 20:10:58], there are lots of "cannot commit a orphan transaction" errors

link:
https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22Ssm%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-ec75b49-202412272153%5C%22%7D%20%7C%3D%20%60cannot%20commit%20a%20orphan%20transaction%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221735385400000%22,%22to%22:%221735389059000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

1. run a mo cluster with config in this issue
2. run tpch 10G loop test processes in one independant tenant
3. run tpcc 10 warehouse and 10 ternimals longrunnig test processes in one independant tenant, prepare mode
4. run sysbench mixed cases(insert/delete/update/select) longrunnig test processes with 75 terminals in one independant tenant,non-prepare mode
5. run another sysbench mixed cases(insert/delete/update/select) longrunnig test processe with  75 terminals in one independant tenant,non-prepare mode

Additional information

No response

[aressu1985]

go routine [2024.12.28 20:11~2024.12.28 20:13]
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2d-a088-7565-9989-d6035b941de4.gz
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2f-a700-7144-b13e-b4e794236363.gz
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2f-3200-7b0a-b500-debd7fdf9847.gz
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2e-bc95-7c7a-982c-4f580aef206b.gz
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2e-47b9-7693-90ec-54410369644a.gz
CN_38616138-6161-6338-3936-633236616530_goroutine_01940d2d-e1a2-7cab-b897-1f7c6b9b9269.gz

[zhangxu19830126]

1. 负责定时发送心跳的goroutine存活
2. 没有任何发送心跳的错误日志
3. 检测出来超时的孤儿事物

这个判断只能是10分钟没有发送心跳。等 @iamlinjunhong 加上每次发送心跳的时候，加入日志，看看间隔到底多久吧。

[iamlinjunhong]

#21077