[test] Fix SimpleBlocksIT testConcurrentAddBlock

[arteam]

Check that requests for adding an index block in parallel are acked. We already use the same pattern in the SimpleBlocksIT#testCloseOneMissingIndexIgnoreMissing test

Resolve #122324

[elasticsearchmachine]

Pinging @elastic/es-distributed-indexing (Team:Distributed Indexing)

[henningandersen]

I am not convinced this is the right fix. If the changes we made recently cause the index block to sometimes not be added, we need to fix that. Whether it is ack'ed seems irrelevant here.

I wonder how you arrived at the conclusion that this was the necessary fix?

Where you able to reproduce the issue? And did that reproduction no longer reproduce afterwards? If you could reproduce it, then how?

[arteam]

@henningandersen The issue is reproduced very easily locally if I check whether prepareAddBlock call is acked

I've used the same pattern to for retrying of adding the block as in testCloseOneMissingIndexIgnoreMissing test.

[arteam]

Failing index block responses look something like this

Adding block response {"acknowledged":false,"shards_acknowledged":false,"indices":[{"name":"ypqeperjvj","failed_shards":[{"id":"0","failures":[{"shard":0,"index":"ypqeperjvj","status":"INTERNAL_SERVER_ERROR","reason":{"type":"illegal_state_exception","reason":"index shard [ypqeperjvj][0] has not applied block 8,MZC4iSBuSUaJw5XaURexLA,moving to block index write (api), blocks WRITE"}}]}]}]}
Adding block response {"acknowledged":false,"shards_acknowledged":false,"indices":[{"name":"ypqeperjvj","failed_shards":[{"id":"0","failures":[{"node":"CUC650bHQC-vIA3RZjxKhw","shard":0,"index":"ypqeperjvj","status":"INTERNAL_SERVER_ERROR","reason":{"type":"illegal_state_exception","reason":"index shard [ypqeperjvj][0] has not applied block 8,MZC4iSBuSUaJw5XaURexLA,moving to block index write (api), blocks WRITE"}}]}]}]}

[henningandersen]

I see, thanks. I still think I'd prefer to fix the API to actually add the block. This seems to be an internal race condition more than an unsolvable problem. I can see 2 ways to do so:

1. Wait for the first add block to be acknowledged (in MetadataIndexStateService).
2. Wait for the right cluster state version to appear on the node holding the shard.

We should probably do so for both add-index and close-index, but can start with just add-index.

[arteam]

It looks like the exception that is thrown was introduced in c416f25 as part adding for project_id-aware based cluster block checks.

Scratch that, the check existed in the original PR #58094