High cpu usage with slice malloc and mallocGC

[amyangfei]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   1. running a TiCDC cluster, with 2 changefeeds, one is 32 tables, the other is 105 tables.
   2. use sysbench workload in upstream, with 250tps/1500qps of each changefeed
   3. observe cpu usage

2. What did you expect to see?

   normal cpu profile

3. What did you see instead?

   From cpu profile, we found high cpu usage in the slice malloc/gc of sorter.(*heapSorter).flush

profile files:

before:
cpu.profile.zip

after:
cpu.profile.2.zip

4. Versions of the cluster

   • TiCDC version (execute cdc version):

     master@43e6752791f8f09706a4b29a54a5b2250bd9d2fa

5. workaround

   • After simple change the default slice capacity from 65536 to 16/32, the cpu usage decreases largely

• Besides, we can use a double buffer slice to save memory alloc and dealloc, but we should process with memory more careful.