[Spark load] Doris support Spark load

[wyb]

For many users who want to load data into doris for the first time, they have large amount of data, about 10G+，it is hard to support to load so large data into doris at one time using Broker load or Stream load. To resolve this problem, We proposal a new solution to load data by using spark cluster.

Spark clusters are used to preprocess data (bitmap global dict build, partition, sort, aggregation) in spark load, which can improve Doris load performance of large data volume and save the computing resources of Doris.

Spark load is mainly used for the initial migration from other systems or loading large amounts of data into Doris.

                 +
                 | 0. User create spark load job
            +----v----+
            |   FE    |---------------------------------+
            +----+----+                                 |
                 | 3. FE send push tasks                |
                 | 5. FE publish version                |
    +------------+------------+                         |
    |            |            |                         |
+---v---+    +---v---+    +---v---+                     |
|  BE   |    |  BE   |    |  BE   |                     |1. FE submit Spark ETL job
+---^---+    +---^---+    +---^---+                     |
    |4. BE push with broker   |                         |
+---+---+    +---+---+    +---+---+                     |
|Broker |    |Broker |    |Broker |                     |
+---^---+    +---^---+    +---^---+                     |
    |            |            |                         |
+---+------------+------------+---+ 2.ETL +-------------v---------------+
|               HDFS              +------->       Spark cluster         |
|                                 <-------+                             |
+---------------------------------+       +-----------------------------+

[wyb]

Design doc
#2855 [Proposal] support spark load
#2887 [Proposal] Support Spark Convert Doris Segment
#3010 Spark load interface

[imay]

1. can refer this issue in related PR and issue.
2. I will create a project "Spark Load" to track this feature.
3. You can create an issue for each part of this project.

[wangbo]

Count Distinct Module
#3319 Support Java Verision HyperLogLog(REVIEWING)
#3061 Doris Support Using Hive Table to Build Global Dict(TESTING)
#3088 Support Java version 64 bits Integers for BITMAP type(MERGED)

Spark DPP Module
#3726 [Spark Load] Rollup Tree Builder
#3728 [Spark Load] Using SparkDpp to complete some calculation in Spark Load

[wyb]

Resource manager
#3418 [Spark load] Add resource manager (Merged)

Fe schedule job execution
#3712 [Spark load][Fe 1/6] Add spark etl job config (Merged)
#3718 [Spark load][Fe 2/6] Update push task thrift interface (Merged)
#3715 [Spark load][Fe 3/6] Fe create job (Merged)
#3819 [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt (Merged)
#3716 [Spark load][Fe 5/6] Fe submit spark etl job (Merged)
#3717 [Spark load][Fe 6/6] Fe process etl and loading state job (Merged)