Skip to content

Used memory surged when slowly remote fluentd couldn't handle flood of incoming message via forward channel #710

New issue
New issue
Closed
Closed
Used memory surged when slowly remote fluentd couldn't handle flood of incoming message via forward channel#710
Assignees
edsiper
Labels
question
@Gabriel1688

Description

@Gabriel1688
Gabriel1688
opened on Aug 9, 2018

Bug Report

When around 2.3M records per minute were sent to remote fluentd via output_forward. the memory used by fluentbit increases dramatically and never decrease even no traffic anymore. The memory shrink until the remote fluentd restart.
The fluentbit log shows output thread for each chunk never exit as long as the remote fluentd is alive. therefore the memory used by these output threads couldn't be released unless the connection with remote fluentd is broken.

Describe the bug

Create a thread for each chunk of message to be sent. it will cause the memory increase very fast in case this thread does not exit timely.

108 * The engine dispatch is responsible for:
109 *
110 * - Get records from input plugins (fixed tags and dynamic tags)
111 * - For each set of records under the same tag, create a Task. A Task set
112 * a reference to the records and routes through output instances.
113 */
114 int flb_engine_dispatch(uint64_t id, struct flb_input_instance *in, struct flb_config config)
160 / Get data from instance buffers /
161 buf = flb_input_flush(in, &size);
162 if (!buf || size == 0) {
163 if (buf) {
164 flb_free(buf);
165 }
166 return 0;
167 }
168
169 /
170 * Create an engine task, the task will hold the buffer reference
171 * and the co-routines associated to the output instance plugins
172 * that needs to handle the data.
173 */
174 task = flb_task_create(id, buf, size, in, NULL, in->tag, config);
175 if (!task) {
176 flb_free(buf);
177 return -1;
178 }
179 flb_trace("[engine dispatch] task #%i created %p", task->id, task);
180 }

To Reproduce

  • Rubular link if applicable:
    Send message(average throughout: 2M / minute) to fluentd via input forward.

  • Example log message if applicable:

// message counter got from fluentd side per minute.
2018-08-08 16:17:08.731028558 -0400 fluentd.traffic: {"count":1140652,"count_rate":18935.81}
2018-08-08 16:17:08.768209490 -0400 fluentd.traffic: {"count":688129,"count_rate":11412.02}
2018-08-08 16:17:08.768315481 -0400 fluentd.traffic: {"count":248009,"count_rate":4113.02}
2018-08-08 16:17:08.785302177 -0400 fluentd.traffic: {"count":203739,"count_rate":3377.88}

//message counter got from fluentbit side
1533844336.000160,18 (total = 24934425)
1533844345.000154,18 (total = 24934597)
1533844346.000163,18 (total = 24934615)
the above data show fluentbit has a very light incoming traffic.

//// log of fluentbit show hundreds of thread exit when remote connections are broken
^[[1m[^[[0m2018/08/09 15:11:51^[[1m]^[[0m [^[[91merror^[[0m] [out_fw] no upstream connections available
^[[1m[^[[0m2018/08/09 15:11:52^[[1m]^[[0m [^[[91merror^[[0m] [io] TCP connection failed: 192.168.246.72:24227 (Connection timed out)
^[[1m[^[[0m2018/08/09 15:11:52^[[1m]^[[0m [^[[91merror^[[0m] [out_fw] no upstream connections available
^[[1m[^[[0m2018/08/09 15:12:05^[[1m]^[[0m [^[[91merror^[[0m] [out_fw] error writing content body
^[[1m[^[[0m2018/08/09 15:12:05^[[1m]^[[0m [^[[91merror^[[0m] [out_fw] error writing content body

�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=117 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=362 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=418 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=494 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=678 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=925 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=674 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=164 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=668 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=514 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [io] TCP connection failed: 192.168.246.72:24226 (Connection refused)
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [out_fw] no upstream connections available
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [io] TCP connection failed: 192.168.246.72:24224 (Connection refused)
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [out_fw] no upstream connections available
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [io] TCP connection failed: 192.168.246.72:24224 (Connection refused)
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [out_fw] no upstream connections available
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [io] TCP connection failed: 192.168.246.72:24224 (Connection refused)
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [out_fw] no upstream connections available
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=414 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [io] TCP connection failed: 192.168.246.72:24224 (Connection refused)
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[91merror�[0m] [out_fw] no upstream connections available
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=648 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=434 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=91 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=229 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=364 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=520 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=56 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=243 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=618 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=654 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=78 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=130 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=488 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=190 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=35 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=468 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=676 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=131 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=32 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=218 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=574 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=478 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=10 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=77 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=208 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=606 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=656 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=14 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=578 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=332 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=160 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=510 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=458 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=201 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=526 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=616 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=112 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=65 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=121 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=108 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=672 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=194 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=424 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=549 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=109 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=203 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=166 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=282 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=138 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=105 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=188 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=173 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=62 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=338 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=640 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=974 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=543 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=552 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=644 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=420 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=143 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=12 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=253 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=292 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=7 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=185 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=570 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=261 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=84 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=308 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=16 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=95 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=168 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=68 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=8 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=650 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=99 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=690 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=324 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=60 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=298 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=37 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=179 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=9 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=101 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=169 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=358 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=28 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=970 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=180 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=438 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=248 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=96 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=119 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=214 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=584 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=174 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=172 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=580 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=586 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=257 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=594 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=937 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=128 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=51 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=204 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=296 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=330 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=530 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=255 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=532 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=422 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=114 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=176 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=73 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=598 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=346 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=430 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=480 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=376 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=785 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=370 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=464 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=664 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=50 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=996 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=26 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=216 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=222 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=125 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=145 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=148 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=614 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=48 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=165 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=106 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=268 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=175 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=769 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=350 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=410 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=646 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=54 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=386 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=466 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=41 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=500 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=83 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=394 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=210 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=470 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=682 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=209 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=242 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=183 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=596 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=524 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=462 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=88 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=408 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=642 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=100 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=272 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=286 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=1016 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=670 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=19 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=71 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=558 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=69 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=116 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=49 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=251 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=90 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=127 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=104 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=432 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=158 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=25 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=666 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=87 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=63 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=316 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=288 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=144 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=124 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=600 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=696 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=103 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=562 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=658 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=334 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=442 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=154 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=374 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=13 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=66 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=310 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=152 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=290 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=270 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=402 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=1002 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=602 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=6 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=184 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=566 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=300 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=440 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=274 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=89 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=27 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=620 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=694 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=454 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=707 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=280 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=240 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=259 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=61 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=536 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=545 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=129 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=81 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=118 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=33 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=149 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=528 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=38 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=85 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=662 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=392 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=232 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=608 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=568 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=42 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=284 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=396 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=30 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=306 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=496 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=698 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=126 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=64 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=80 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=225 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=4 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=59 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=133 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=360 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=186 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=202 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=227 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=340 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=132 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=398 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=122 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=195 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=354 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=326 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=622 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=111 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=20 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=153 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=692 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=534 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=79 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=564 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=412 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=476 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=998 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=428 thread_id=3 output=forward.2
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=474 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=490 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=52 thread_id=3 output=forward.0
�[1m[�[0m2018/08/09 15:12:06�[1m]�[0m [�[93m warn�[0m] [engine] Task cannot be retried: task_id=2 thread_id=3 output=forward.0
  • Steps to reproduce the problem:

Expected behavior

Screenshots

image

After restart fluentd. Tons of thread are released and memory shrink dramatically.
image

Your Environment
[Fluent Bit]
Edition Community Edition
Version 0.13.4
Built Flags JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_METRICS FLB_HAVE_SQLDB FLB_HAVE_BUFFERING FLB_HAVE_TRACE FLB_HAVE_METRICS FLB_HAVE_HTTP_SERVER FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_JEMALLOC JEMALLOC_MANGLE FLB_HAVE_LIBBACKTRACE FLB_HAVE_REGEX FLB_HAVE_C_TLS FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY

[Operating System]
Name Linux
Release 3.10.0-123.el7.x86_64
Version #1 SMP Mon Jun 30 12:09:22 UTC 2014

[Hardware]
Architecture x86_64
Processors 40

[Built Plugins]
Inputs cpu mem kmsg tail proc disk netif dummy head exec health serial stdin tcp mqtt lib forward random syslog
Filters grep stdout throttle kubernetes parser record_modifier nest modify
Outputs azure counter es exit file forward http influxdb kafka-rest nats null plot retry splunk stdout td lib flowcounter

[SERVER] Runtime configuration
Flush 5
Daemon Off
Log_Level Info

[SERVICE]
Flush 1
Daemon Off
Log_Level info
Log_File /var/log/td-agent-bit/fluentbit.log

[INPUT]
Name tcp
Listen 0.0.0.0
Port 5170
Chunk_Size 32
Buffer_Size 64
tag server1
Mem_Buf_Limit 10MB

[INPUT]
Name tcp
Listen 0.0.0.0
Port 5171
Chunk_Size 32
Buffer_Size 64
tag server2
Mem_Buf_Limit 10MB

[INPUT]
Name tcp
Listen 0.0.0.0
Port 5172
Chunk_Size 32
Buffer_Size 64
tag server3
Mem_Buf_Limit 10MB

[INPUT]
Name tcp
Listen 0.0.0.0
Port 5173
Chunk_Size 32
Buffer_Size 64
tag server4
Mem_Buf_Limit 10MB

[FILTER]
Name throttle
Match *
Rate 10000
Window 60
Interval 1s

[OUTPUT]
Name forward
Host 192.168.246.72
Port 24224
Time_as_Integer true
num_threads 4
slow_flush_log_threshold 20
retry_limit 1
Match server1

[OUTPUT]
Name forward
Host 192.168.246.72
Port 24225
Time_as_Integer true
num_threads 4
slow_flush_log_threshold 20
retry_limit 1
Match server2

[OUTPUT]
Name forward
Host 192.168.246.72
Port 24226
Time_as_Integer true
num_threads 4
slow_flush_log_threshold 20
retry_limit 1
Match server3

[OUTPUT]
Name forward
Host 192.168.246.72
Port 24227
Time_as_Integer true
num_threads 4
slow_flush_log_threshold 20
retry_limit 1
Match server4

[OUTPUT]
Name counter
Match *

-----Configure of remote fluentd.

@type forward bind 0.0.0.0 port 24224 @id input_server1 @type forward bind 0.0.0.0 port 24225 @id input_server2 @type forward bind 0.0.0.0 port 24226 @id input_server3 @type forward bind 0.0.0.0 port 24227 @id input_server4 @type copy @type null @type flowcounter unit minute aggregate all tag fluentd.traffic @type copy @type null @type flowcounter unit minute aggregate all tag fluentd.traffic @type copy @type null @type flowcounter unit minute aggregate all tag fluentd.traffic @type copy @type null @type flowcounter unit minute aggregate all tag fluentd.traffic @type stdout

Additional context

Metadata

Metadata

Assignees

Labels

question

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions