[Doris On ES][Feature] Support create table with wildcard or aliase index

[wuyunfeng]

#3967

Create multiple index with same mapping in ES:

PUT wildcard_1
{
   "settings": {
      "number_of_replicas": 0,
      "number_of_shards": 1
   },
   "mappings": {
      "doc": {
         "properties": {
            "k1": {
               "type": "integer"
            },
            "k2": {
               "type": "keyword"
            }
         }
      }
   }
}

PUT wildcard_2
{
   "settings": {
      "number_of_replicas": 0,
      "number_of_shards": 1
   },
   "mappings": {
      "doc": {
         "properties": {
            "k1": {
               "type": "integer"
            },
            "index": {
               "type": "keyword"
            }
         }
      }
   }
}

PUT wildcard_3
{
   "settings": {
      "number_of_replicas": 0,
      "number_of_shards": 1
   },
   "mappings": {
      "doc": {
         "properties": {
            "k1": {
               "type": "integer"
            },
            "index": {
               "type": "keyword"
            }
         }
      }
   }
}

k2 used for indicating the data from which index

Create wildcard_alias aliases contains those above three index:

POST /_aliases
{
   "actions": [
      {
         "add": {
            "indices": [
               "wildcard_3",
               "wildcard_2",
               "wildcard_1"
            ],
            "alias": "wildcard_alias"
         }
      }
   ]
}

Create ES external table within Doris:

• Support aliase matched index:

CREATE EXTERNAL TABLE `wildcard_alias` (
  `k1` int COMMENT "",
   `k2`  varchar COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://127.0.0.1:8200",
"user" = "root",
"password" = "root",
"index" = "wildcard_alias",
"type" = "doc",
"version" = "6.5.3",
"enable_docvalue_scan" = "true"
);

• Support wildcard matched index (pattern: wildcard_*):

CREATE EXTERNAL TABLE `wildcard_alias` (
  `k1` int COMMENT "",
   `k2`  varchar COMMENT ""
) ENGINE=ELASTICSEARCH
PROPERTIES (
"hosts" = "http://10.74.167.16:8200",
"user" = "root",
"password" = "root",
"index" = "wildcard_*",
"type" = "doc",
"version" = "6.5.3",
"enable_docvalue_scan" = "true"
);

And execute SQL on Doris:

 select * from wildcard_alias;
  or
 select * from wildcard;

Finally we get all document within those three index:

+------+-------------+
| k1   | k2          |
+------+-------------+
|  111 | wiladcard_1 |
| 1111 | wiladcard_1 |
| 1112 | wiladcard_1 |
|  110 | wiladcard_1 |
|  222 | wiladcard_2 |
|  223 | wiladcard_2 |
|  223 | wiladcard_2 |
|   33 | wiladcard_3 |
|   33 | wiladcard_3 |
|   33 | wiladcard_3 |
|   33 | wiladcard_3 |
|   33 | wiladcard_3 |
+------+-------------+

BTW: predicates which can push down would push down to all related indices

[BabySid]

looks good.
but i have a question: sql like 'select k1 from wildcard_alias where k1 > '2020-05-20'', DSL will run on all related indices? or just on wildcard_alias_2020_05_21, wildcard_alias_2020_05_22...

[wuyunfeng]

@blackfox1983 yes, run on all related indices. process predicate is independent with processing resolved indidces

[morningman]

LGTM