[index.mode=time_series] requires a non-empty [index.routing_path] when creating component template

[rpasche]

Elasticsearch Version

8.13.3

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu

Problem Description

When trying to create a component template which should basically only set the index.mode: "time_series", this results in

[index.mode=time_series] requires a non-empty [index.routing_path]

I know, that index.mode: "time_series" needs further settings (that might need to come from other component templates).

You get the same error, when you only try to create a component template, only setting index.routing_path (without specifying index.mode: "time_series" within this component template.)

This sounds like an error to me. It should be possible to set those settings or mapping separately in different component templates.

You could have several different component templates like this:

PUT _component_template/enable_timeseries_mode
{
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  }
}

# 
PUT _component_template/prometheus_dimensions
{
  "template": {
    "settings": {
      "index": {
        "routing_path": [
          "prometheus.labels.*"
        ]
      }
    },
    "mappings": {
      "dynamic_templates": [
        {
          "prometheus_dimensions": {
            "path_match": [
              "prometheus.labels.*"
            ],
            "mapping": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        }
      ]
    }
  }
}

and use both component templates in the final index-template

{
...
  "composed_of": [
    "enable_timeseries_mod",
    "prometheus_dimensions"
  ]
}

It is - somehow - related to

• Unable to create time series index without setting routing_path #96330 and
• Error when updating component template when index template has "index.mode=time_series" #91592

Steps to Reproduce

Try to create a new component template

PUT _component_template/enable_timeseries_mode
{
  "template": {
    "settings": {
      "index": {
        "mode": "time_series"
      }
    }
  }
}

results in

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "[index.mode=time_series] requires a non-empty [index.routing_path]"
  },
  "status": 400
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[rpasche]

Is this an expected behaviour?

[martijnvg]

Hey @rpasche, Apologies for the late reply here. I think this is expected behavior. A component template needs to be valid on its own. The time series index mode requires routing path to be configured. So in this case routing path should also be configured in this component template.

Typically you would set the index.mode in an index template, since then there is complete view of what settings and mappings are available (from index template itself and other component templates) to be applied to a new data stream / backing index (when rolling over a data stream). Additionally, often you don't need to configure routing path, because the routing path can be determined from the fields marked as dimensions in the mapping.

[rpasche]

Hi @martijnvg , thank you for the answer.

Typically you would set the index.mode in an index template, since then there is complete view of what settings and mappings are available

I will need to test this again, but I think, I also had issues to only set index.mode to timeseries and not setting index.routing_path (as it was again complaining, that index.routing_path is needed in timeseries mode, even if the documentation states, that routing_path should be build of all dimension stated fields from within the index-template).

[martijnvg]

I also had issues to only set index.mode to timeseries and not setting index.routing_path

If you set index.mode to time_series in an index template and there is mapping with at least one field of type keyword with time_series_dimension attribute set to true, then the index.routing_path should automatically be generated. Otherwise maybe there is a bug.

[rpasche]

Hi @martijnvg

I tried this. I somehow assume, that this might simply be caused by the dynamic template. (so there is - not yet - a real dimension field in place

PUT _component_template/dummy
{
  "template": {
    "mappings": {
      "dynamic_templates": [
        {
          "labels_as_dimensions": {
            "path_match": [
              "labels.*",
              "prometheus.labels.*"
            ],
            "mapping": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        }
      ]
    }
  }
}

PUT _index_template/dummy
{
  "index_patterns": [
    "my-test-index"
  ],
  "priority": 201,
  "template": {
    "settings": {
      "index.mode": "time_series"
    }
  },
  "composed_of": [
    "dummy"
  ]
}

The error

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_template_exception",
        "reason": "index_template [dummy] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
      }
    ],
    "type": "invalid_index_template_exception",
    "reason": "index_template [dummy] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
  },
  "status": 400
}

trying static mapping

It also does not work for me, if I have a static mapping

# dummy2
PUT _component_template/dummy2
{
  "template": {
    "mappings": {
      "properties": {
        "labels": {
          "type": "object",
          "properties": {
            "my_dimension_field": {
              "type": "keyword",
              "time_series_dimension": true
            }
          }
        }
      }
    }
  }
}

PUT _index_template/dummy2
{
  "index_patterns": [
    "my-test-index"
  ],
  "template": {
    "settings": {
      "index.mode": "time_series"
    }
  }, 
  "composed_of": [
    "dummy2"
  ]
}

I get again

{
  "error": {
    "root_cause": [
      {
        "type": "invalid_index_template_exception",
        "reason": "index_template [dummy2] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
      }
    ],
    "type": "invalid_index_template_exception",
    "reason": "index_template [dummy2] invalid, cause [Validation Failed: 1: [index.mode=time_series] requires a non-empty [index.routing_path];]"
  },
  "status": 400
}

[martijnvg]

If you add the follow snippet to your index template, then it should work:

"data_stream": {},

[rpasche]

Oh....missed that. Will try

[rpasche]

Ok. With

"data_stream": {}

it worked. Thanks.

But this means, that index.mode: "time_series" cannot only be set in a component template. It basically must be set within an index-template.

[martijnvg]

But this means, that index.mode: "time_series" cannot only be set in a component template. It basically must be set within an index-template.

Yes. Unless you configure the index.routing_path index setting as well in the component template. The logic that determines the routing path doesn't kick in for component templates, only for index templates.

Do you have a particular reason why index.mode needs to be configured in a component template? Would defining it in an index template work too?

I think we need to update the documentation and explain the relation between index.mode, index.routing_path and dimension fields in the mapping better.

[rpasche]

Do you have a particular reason why index.mode needs to be configured in a component template? Would defining it in an index template work too?

We are trying to "template" our index-templates as much as we can (also for our customers). The idea is to be able to "pick" all your "desired" component-template your index-template should be built of and go on.

And one of those "desired" features the final "index" should have is to be a TSDS. And for this, I need component-templates

• A
• B
• ....
• enable_timeseries
• ...

something like this. Of course, we can also build the index-template directly, but we started to build different component templates with "specific" settings and just noticed this "error" when trying to build a component template, that should "just" set the index.mode: "time_series".

That's basically it.

Thanks.

[martijnvg]

Thanks for sharing this.

The index.mode setting's dependency on index.routing_path setting isn't something that can be changed with how things work today. In your example, the component template that enabled time series mode, needs to include more information. Today this means you will need to include the index.routing_path setting and the mappings for the fields defined in the routing path.

Maybe this can be made easier by also automatically generating the index.routing_path setting based on the mapping the component template. But at a minimum, time series index mode needs to know a subset of the dimensions. If time series index mode is enabled in a component template, then this information needs to be present in that component template.