OpenSearch sink not rolling over the index after upgrading from 2.7 to 2.10

[dmossakowski]

Since migrating from 2.7.0 to 2.10.1 the index id stayed at the same number (000056) and is no longer rolling over. You can see below that previously indexes would be rolled around 500mb but this latest one is now at 108.5gb. Number of documents used to be around 300000 but now it's more than 73 million. Did I miss some configuration?

  | otel-v1-apm-span-000056 | green | Yes | open | 108.5gb | 54.1gb | 73591140 | 4806948 | 1 | 1
  | otel-v1-apm-span-000055 | green | Yes | open | 471mb | 235.5mb | 231704 | 24212 | 1 | 1
  | otel-v1-apm-span-000054 | green | Yes | open | 478.3mb | 239.1mb | 299849 | 20501 | 1 | 1
  | otel-v1-apm-span-000053 | green | Yes | open | 504mb | 252mb | 310190 | 26766 |  
..

Data prepper config:

raw-pipeline:
  workers: 2
  delay: "3000"
  source:
    pipeline:
      name: "otel-trace-pipeline"
  buffer:
    bounded_blocking:
      buffer_size: 10240
      batch_size: 160
  processor:
    - delete_entries:
        with_keys: ['command_args']
    - otel_traces:
    - otel_trace_group:
        hosts: ["https://opensearch-node1:9200"]
        username: admin
        password: ------
        insecure: true
  sink:
    - opensearch:
        hosts: ["https://opensearch-node1:9200"]
        index_type: trace-analytics-raw
        username: admin
        password: --------
        insecure: true

[dmossakowski]

After looking at bug: #3506 I checked the alias setting and it is set:

{
  "otel-v1-apm-span-index-template": {
    "order": 0,
    "version": 1,
    "index_patterns": [
      "otel-v1-apm-span-*"
    ],
    "settings": {
      "index": {
        "opendistro": {
          "index_state_management": {
            "rollover_alias": "otel-v1-apm-span"
          }
        }
      }
    },

The last index also has it:

GET otel-v1-apm-span-000056/_settings

{
  "otel-v1-apm-span-000056": {
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "opendistro": {
          "index_state_management": {
            "rollover_alias": "otel-v1-apm-span"
          }
        },
        "number_of_shards": "1",
        "provided_name": "otel-v1-apm-span-000056",
        "creation_date": "1718207944075",
        "number_of_replicas": "1",
        "uuid": "9oLapWNhTo-e1WCZ-Eiu6w",
        "version": {
          "created": "136347827",
          "upgraded": "136387927"
        }
..

Now I started a completely new cluster with empty indexes and I do see the roll over so the problem seems to be only on the cluster that was upgraded. What can I do there on this existing cluster to start rolling the indexes again? This same index just keeps growing.

otel-v1-apm-span-000056 | green | Yes | open | 108.5gb | 54.4gb | 73904379 | 4806948