[tests] putpipeline 7.x ProcessClusterEventTimeoutException

[Mpdreamz]

build.bat seed:9368 integrate 7.0.0 "writable" "putpipeline"

From: #4004

ProcessClusterEventTimeoutException it'd be good if we can somehow improve the error reporting when we get this exception from the server in general.

Capture:

• hotthreads API
• get tasks API

This one is hard to diagnose and replicate. A mitigation might be to move the ingest tests out of writable cluster to make them more isolated.

[xUnit.net 00:07:44.3679353]     Tests.Ingest.PutPipeline.PutPipelineApiTests.ReturnsExpectedResponse [FAIL]
Failed   Tests.Ingest.PutPipeline.PutPipelineApiTests.ReturnsExpectedResponse
Error Message:
 Tests.Framework.EndpointTests.ResponseAssertionException : Expected response.Acknowledged to be true, but found False.
Response Under Test:
Invalid NEST response built from a unsuccessful (503) low level call on PUT: /_ingest/pipeline/pipeline-1?pretty=true&error_trace=true
# Audit trail of this API call:
 - [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:30.0527811
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: The remote server returned an error: (503) Server Unavailable.. Call: Status code 503 from: PUT /_ingest/pipeline/pipeline-1?pretty=true&error_trace=true. ServerError: Type: process_cluster_event_timeout_exception Reason: "failed to process cluster event (put-pipeline-pipeline-1) within 30s" ---> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.
   at System.Net.HttpWebRequest.GetResponse() 
   at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) in D:\a\1\s\src\Elasticsearch.Net\Connection\HttpWebRequestConnection.cs:line 59
   --- End of inner exception stack trace ---
# Request:
{"description":"My test pipeline","processors":[{"append":{"field":"state","value":["Stable","VeryActive"]}},{"convert":{"field":"numberOfCommits","target_field":"targetField","type":"string"}},{"date":{"field":"startedOn","formats":["dd/MM/yyyy hh:mm:ss"],"target_field":"timestamp","timezone":"Europe/Amsterdam"}},{"fail":{"message":"an error message"}},{"foreach":{"field":"tags","processor":{"uppercase":{"field":"_value.name"}}}},{"grok":{"field":"description","pattern_definitions":{"FAVORITE_DOG":"border collie","RGB":"RED|BLUE|GREEN"},"patterns":["my %{FAVORITE_DOG:dog} is colored %{RGB:color}"]}},{"gsub":{"field":"name","pattern":"-","replacement":"_"}},{"join":{"field":"branches","separator":","}},{"lowercase":{"field":"name"}},{"remove":{"field":["suggest"]}},{"rename":{"field":"leadDeveloper","target_field":"projectLead"}},{"set":{"field":"name","value":"foo"}},{"split":{"field":"description","separator":"."}},{"trim":{"field":"name"}},{"uppercase":{"field":"name"}},{"dot_expander":{"field":"field.withDots"}},{"script":{"source":"ctx.numberOfCommits++"}},{"urldecode":{"field":"description","ignore_missing":true}},{"attachment":{"field":"description","ignore_missing":true,"indexed_chars":100000,"properties":["title","author"]}},{"bytes":{"field":"description","ignore_missing":true}},{"dissect":{"field":"description","pattern":"%{clientip} %{ident} %{auth} [%{@timestamp}] \"%{verb} %{request} HTTP/%{httpversion}\" %{status} %{size}","ignore_missing":true,"append_separator":" "}},{"drop":{"if":"true"}},{"kv":{"field":"description","field_split":"_","ignore_missing":true,"value_split":" "}},{"kv":{"field":"description","field_split":"_","ignore_missing":true,"strip_brackets":true,"trim_key":"xyz","trim_value":"abc","value_split":" "}},{"set_security_user":{"field":"name"}},{"pipeline":{"name":"x"}}]}
# Response:
{
  "error" : {
    "root_cause" : [
      {
        "type" : "process_cluster_event_timeout_exception",
        "reason" : "failed to process cluster event (put-pipeline-pipeline-1) within 30s",
        "stack_trace" : "ProcessClusterEventTimeoutException[failed to process cluster event (put-pipeline-pipeline-1) within 30s]\r\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:134)\r\n\tat java.base/java.util.ArrayList.forEach(ArrayList.java:1540)\r\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:133)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:835)\r\n"
      }
    ],
    "type" : "process_cluster_event_timeout_exception",
    "reason" : "failed to process cluster event (put-pipeline-pipeline-1) within 30s",
    "stack_trace" : "ProcessClusterEventTimeoutException[failed to process cluster event (put-pipeline-pipeline-1) within 30s]\r\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$0(MasterService.java:134)\r\n\tat java.base/java.util.ArrayList.forEach(ArrayList.java:1540)\r\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:133)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:835)\r\n"
  },
  "status" : 503
}