Replace SecurityIndexManager with special client

[jaymode]

Currently, we use the SecurityIndexManager throughout our code to be "smart" and preempt certain conditions based on the last observed cluster state when interacting with the security index. However these conditions that we try to preempt can still occur so we still need to handle them within each individual response listener. The SecurityIndexManager is also responsible for ensuring that the index has the correct mappings and has been upgraded to the latest version.

In order to simplify our code, it has been suggested that we replace this with a specialized client that knows how to handle the error conditions and that we do not preempt failures. One item that would be a prerequisite would be to disallow the auto creation of the security index by a index request, see #34737.

[elasticmachine]

Pinging @elastic/es-security

[albertzaharovits]

We've uncovered another problem of this. The callers that race to create the index (prepareIndexIfNeededThenExecute) and fail with ResourceAlreadyExistsException don't wait for shards to become available, and will error out with a missing shard exception.

[Mpdreamz]

I think this one relates as well: #45250

We see this error fairly often in the .NET client CI:elastic/elasticsearch-net#4280

[tvernum]

I think the above two comments specifically refer to #46214