Dynamic allocation for writing parallelism in flink sink

[nicochen]

Here is a case that data throughput have significant difference between day and night. A flink job has a immutable parallelism of sinks once started. Take an un-keyed table as an example, 'none' distribution policy is select which means best utilization of all parallelism, while it creates biggest number of files even through small quality of data comes.

Therefore, I‘d like to propose a new distribution policy to dynamically allocate writing parallelism in streaming writing according to number of data. In this way, we can not only relieve the pressure from small files in hdfs, but also increate the efficiency of reading and optimisation work.