Error when updating component template when index template has "index.mode=time_series"

[juliaElastic]

While working on a Fleet feature came across an issue:

In Fleet, added index.mode=time_series setting to metrics-system.cpu Index Template, and relying on ES automation to fill the routing_path field.

This worked fine regards to generating the routing_path field in the data streams.
However, when subsequently trying to update a parent metrics-system.cpu@custom Component Template, I ran into an issue saying that the resulting template is invalid, see error below.

After discussing with @martijnvg on Slack, the conclusion was that this should be fixed in elasticsearch:
When the put component template api validates an index template that a component template it is updating is using, we should generate the routing path as well.

Also, the update should fail if there are no dimension fields in the mappings.

Error message:

{
  "name": "ResponseError",
  "meta": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "illegal_argument_exception",
            "reason": "updating component template [metrics-system.cpu@custom] results in invalid composable template [metrics-system.cpu] after templates are merged"
          }
        ],
        "type": "illegal_argument_exception",
        "reason": "updating component template [metrics-system.cpu@custom] results in invalid composable template [metrics-system.cpu] after templates are merged",
        "caused_by": {
          "type": "illegal_argument_exception",
          "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
        }
      },
      "status": 400
    },
    "statusCode": 400,
    "headers": {
      "x-opaque-id": "59e2d33e-d6c8-4ed4-8d4a-14c412f64871;kibana::management:",
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json;charset=utf-8",
      "content-length": "549"
    },
    "meta": {
      "context": null,
      "request": {
        "params": {
          "method": "PUT",
          "path": "/_component_template/metrics-system.cpu%40custom",
          "body": "{\"template\":{\"settings\":{},\"mappings\":{\"properties\":{\"dummy\":{\"type\":\"text\"}}}},\"_meta\":{\"package\":{\"name\":\"system\"},\"managed_by\":\"fleet\",\"managed\":true}}",
          "querystring": "",
          "headers": {
            "user-agent": "Kibana/8.6.0",
            "x-elastic-product-origin": "kibana",
            "authorization": "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==",
            "x-opaque-id": "59e2d33e-d6c8-4ed4-8d4a-14c412f64871;kibana::management:",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1",
            "content-type": "application/vnd.elasticsearch+json; compatible-with=8",
            "accept": "application/vnd.elasticsearch+json; compatible-with=8",
            "content-length": "154"
          }
        },
        "options": {
          "opaqueId": "59e2d33e-d6c8-4ed4-8d4a-14c412f64871;kibana::management:",
          "headers": {
            "x-elastic-product-origin": "kibana",
            "user-agent": "Kibana/8.6.0",
            "authorization": "Basic ZWxhc3RpYzpjaGFuZ2VtZQ==",
            "x-opaque-id": "59e2d33e-d6c8-4ed4-8d4a-14c412f64871",
            "x-elastic-client-meta": "es=8.4.0p,js=16.18.1,t=8.2.0,hc=16.18.1"
          }
        },
        "id": 2
      },
      "name": "elasticsearch-js",
      "connection": {
        "url": "http://localhost:9200/",
        "id": "http://localhost:9200/",
        "headers": {},
        "status": "alive"
      },
      "attempts": 0,
      "aborted": false
    },
    "warnings": null
  }
}

Originally posted by @juliaElastic in #74660 (comment)

[juliaElastic]

For reference:

Index template:

{
    "name": "metrics-system.cpu",
    "priority": 200,
    "indexPatterns": [
        "metrics-system.cpu-*"
    ],
    "template": {
        "settings": {
            "index": {
                "mode": "time_series"
            }
        },
        "mappings": {
            "_meta": {
                "package": {
                    "name": "system"
                },
                "managed_by": "fleet",
                "managed": true
            }
        }
    },
    "composedOf": [
        "metrics-system.cpu@package",
        "metrics-system.cpu@custom",
        ".fleet_globals-1",
        ".fleet_agent_id_verification-1"
    ],
    "dataStream": {
        "hidden": false,
        "allow_custom_routing": false
    },
    "_meta": {
        "package": {
            "name": "system"
        },
        "managed_by": "fleet",
        "managed": true
    },
    "_kbnMeta": {
        "type": "managed",
        "hasDatastream": true
    }
}

Component template with dimension field:

{
    "name": "metrics-system.cpu@package",
    "template": {
        "settings": {
            "index": {
                "lifecycle": {
                    "name": "metrics"
                },
                "codec": "best_compression",
                "default_pipeline": "metrics-system.cpu-1.20.5",
                "mapping": {
                    "total_fields": {
                        "limit": "10000"
                    }
                },
                "query": {
                    "default_field": [
                        "cloud.account.id",
                        "cloud.availability_zone",
                        "cloud.instance.id",
                        "cloud.instance.name",
                        "cloud.machine.type",
                        "cloud.provider",
                        "cloud.region",
                        "cloud.project.id",
                        "cloud.image.id",
                        "container.id",
                        "container.image.name",
                        "container.name",
                        "host.architecture",
                        "host.hostname",
                        "host.id",
                        "host.mac",
                        "host.name",
                        "host.os.family",
                        "host.os.kernel",
                        "host.os.name",
                        "host.os.platform",
                        "host.os.version",
                        "host.os.build",
                        "host.os.codename",
                        "host.os.full",
                        "host.type",
                        "test_dimension"
                    ]
                }
            }
        },
        "mappings": {
            "_source": {
                "mode": "stored"
            },
            "dynamic_templates": [
                {
                    "container.labels": {
                        "path_match": "container.labels.*",
                        "mapping": {
                            "type": "keyword"
                        },
                        "match_mapping_type": "string"
                    }
                }
            ],
            "properties": {
                "cloud": {
                    "properties": {
                        "availability_zone": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "image": {
                            "properties": {
                                "id": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "instance": {
                            "properties": {
                                "name": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                },
                                "id": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "provider": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "machine": {
                            "properties": {
                                "type": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "project": {
                            "properties": {
                                "id": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "region": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "account": {
                            "properties": {
                                "id": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                },
                "container": {
                    "properties": {
                        "image": {
                            "properties": {
                                "name": {
                                    "ignore_above": 1024,
                                    "type": "keyword"
                                }
                            }
                        },
                        "name": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        },
                        "id": {
                            "ignore_above": 1024,
                            "type": "keyword"
                        }
                    }
                },
                "test_dimension": {
                    "time_series_dimension": true,
                    "type": "keyword"
                },
                "@timestamp": {
                    "type": "date"
                },
                "system": {
                    "properties": {
                        "cpu": {
                            "properties": {
                                "total": {
                                    "properties": {
                                        "pct": {
                                            "time_series_metric": "gauge",
                                            "meta": {
                                                "unit": "percent",
                                                "metric_type": "gauge"
                                            },
                                            "scaling_factor": 1000,
                                            "type": "scaled_float"
                                        },
                                        "norm": {
                                            "properties": {
                                                "pct": {
                                                    "time_series_metric": "gauge",
                                                    "meta": {
                                                        "unit": "percent",
                                                        "metric_type": "gauge"
                                                    },
                                                    "scaling_factor": 1000,
                                                    "type": "scaled_float"
                                                }
                                            }
                                        }
                                    }
                                },
                                "cores": {
                                    "meta": {
                                        "metric_type": "gauge"
                                    },
                                    "type": "long"
                                },
                                "system": {
                                    "properties": {
                                        "pct": {
                                            "time_series_metric": "gauge",
                                            "meta": {
                                                "unit": "percent",
                                                "metric_type": "gauge"
                                            },
                                            "scaling_factor": 1000,
                                            "type": "scaled_float"
                                        },
                                        "ticks": {
                                            "type": "long"
                                        },
                                        "norm": {
                                            "properties": {
                                                "pct": {
                                                    "time_series_metric": "gauge",
                                                    "meta": {
                                                        "unit": "percent",
                                                        "metric_type": "gauge"
                                                    },
                                                    "scaling_factor": 1000,
                                                    "type": "scaled_float"
                                                }
                                            }
                                        }
                                    }
                                },
                                "softirq": {
                                    "properties": {
                                        "pct": {
                                            "time_series_metric": "gauge",
                                            "meta": {
                                                "unit": "percent",
                                                "metric_type": "gauge"
                                            },
                                            "scaling_factor": 1000,
                                            "type": "scaled_float"
                                        },
                                        "ticks": {
                                            "meta": {
                                                "metric_type": "counter"
                                            },
                                            "type": "long"
                                        },
                                        "norm": {
                                            "properties": {
                                                "pct": {
                                                    "time_series_metric": "gauge",
                                                    "meta": {
                                                        "unit": "percent",
                                                        "metric_type": "gauge"
                                                    },
                                                    "scaling_factor": 1000,
                                                    "type": "scaled_float"
                                                }
                                            }
                                        }
                                    }
                                },
                                "idle": {
                                    "properties": {
                                        "pct": {
                                            "time_series_metric": "gauge",
                                            "meta": {
                                                "unit": "percent",
                                                "metric_type": "gauge"
                                            },
                                            "scaling_factor": 1000,
                                            "type": "scaled_float"
                                        },
                                        "ticks": {
                                            "meta": {
                                                "metric_type": "counter"
...

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)