Elasticsearch fault after 2.4.111 update, disk cleanup commands needed

[innovate-support]

Version

2.4.111

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

16

Storage for /

165G

Storage for /nsm

335G

Network Traffic Collection

span port

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I have a Standalone system running on a small network. It's been working fine for a few years now. I recently updated it to v2.4.111 and now the Elasticsearch is showing Fault and I can't Hunt for anything until it's resolved. I ran the following command and now know what the problem is, but not how to fix it. Is there a 'cleanup' command I can run, or some config I need to adjust disk percentages to prevent this from happening? This has happened to me on other systems before and I ended up just re-installing, but I don't think that's an acceptable solution. This should be easily fixable with a few commands.

[sysadmin@so-beelink ~]$ sudo so-elasticsearch-query _cluster/allocation/explain?pretty
{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index" : ".ds-logs-system.syslog-default-2025.01.22-000003",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2025-01-28T16:48:08.071Z",
"last_allocation_status" : "no"
},
"can_allocate" : "no",
"allocate_explanation" : "Elasticsearch isn't allowed to allocate this shard to any of the nodes in the cluster. Choose a node to which you expect this shard to be allocated, find this node in the node-by-node explanation, and address the reasons which prevent Elasticsearch from allocating this shard there.",
"node_allocation_decisions" : [
{
"node_id" : "ZFHF-vP6RaaVHdFnoYGZag",
"node_name" : "so-beelink",
"transport_address" : "192.168.76.55:9300",
"node_attributes" : {
"transform.config_version" : "10.0.0",
"xpack.installed" : "true",
"ml.config_version" : "12.0.0"
},
"roles" : [
"data",
"data_hot",
"ingest",
"master",
"remote_cluster_client",
"transform"
],
"node_decision" : "no",
"weight_ranking" : 1,
"deciders" : [
{
"decider" : "disk_threshold",
"decision" : "NO",
"explanation" : "the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=85%], having less than the minimum required [46.6gb] free space, actual free: [42.3gb], actual used: [86.3%]"
}
]
}
]
}

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[innovate-support]

I adjusted the 'Elasticsearch Global ILM Policies' found on the 'Grid Configuration Quick Links' page to the following and then waited an hour, and the system seems to have healed itself.
Warm Phase: 10d
Cold Phase: 20d
Delete Phase: 30d