Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory Overflow During Data Insertion with DorisWriter, Task Stuck in Blocked State #2251

Open
Yida-Zhangfb opened this issue Dec 11, 2024 · 0 comments · May be fixed by #2252
Open

Memory Overflow During Data Insertion with DorisWriter, Task Stuck in Blocked State #2251

Yida-Zhangfb opened this issue Dec 11, 2024 · 0 comments · May be fixed by #2252

Comments

@Yida-Zhangfb
Copy link

Description:

使用hdfsreader同步至doriswriter时，从日志观察此次任务出现了内存溢出的相关信息，并生成了dump文件，但是此任务并未立即停止并标记任务失败，而且hdfs切分的24个任务最终成功了23个，最后打印的日志显示一直处于阻塞状态，显示速率为0。

Screenshots:

1.Memory Usage Graph:

1

2.Blocked Task:

2
3
Yida-Zhangfb added a commit to Yida-Zhangfb/DataX that referenced this issue Dec 11, 2024
@Yida-Zhangfb
fix: resolve sync block on memory overflow in Doris
78a76f0 
- 当检测到内存溢出时，改进了进程的终止行为，避免了同步操作的长时间阻塞。
- 修复了在内存溢出情况下，Doris 数据同步进程出现阻塞的问题。
- startAsyncFlushing()为异步线程消费flushQueue的数据，如出现内存溢出catch为Exception时则不能捕获到，flushException为null，checkFlushException()则不能检测到异常;异步线程内存溢出主动退出，由于flushQueue数据不能被消费，则导致写入的主线程仍阻塞在flushQueue.put()
Closes alibaba#2251
@Yida-Zhangfb Yida-Zhangfb linked a pull request Dec 11, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@Yida-Zhangfb