Job activation delayed by 10 seconds under stable network conditions

[lenaschoenburg]

Describe the bug

Sometimes jobs are activated exactly 10 seconds after the corresponding service task is activated. In general this can happen for various reasons, for example connection issue between worker, gateway and broker. 10 seconds is also the default request timeout for clients.

We've recently started to observe this in our QA runs, where these delays cause test failures. Since the network should be in a stable condition and this has only recently become an issue, it seems likely that this is actually a regression that we need to fix.

To Reproduce

With a current snapshot version of Zeebe and a spring-zeebe 8.0.5 this was at least not trivially reproducible.

Expected behavior

Under stable network conditions and barring any other issues, there should be virtually no delay between service task activation and job activation.

Log/Stacktrace

{
  "partitionId": 1,
  "value": {
    "version": 1,
    "processDefinitionKey": 2251799813685372,
    "bpmnProcessId": "sequential-test-process",
    "processInstanceKey": 4503599627370626,
    "parentElementInstanceKey": -1,
    "flowScopeKey": 4503599627370626,
    "parentProcessInstanceKey": -1,
    "elementId": "step-0",
    "bpmnElementType": "SERVICE_TASK"
  },
  "key": 4503599627370633,
  "timestamp": 1656635590315,
  "position": 393,
  "valueType": "PROCESS_INSTANCE",
  "intent": "ELEMENT_ACTIVATED",
  "recordType": "EVENT",
  "rejectionType": "NULL_VAL",
  "rejectionReason": "",
  "brokerVersion": "8.1.0",
  "sourceRecordPosition": 390
}
{
  "index": 176,
  "term": 1,
  "entries": [
    {
      "partitionId": 1,
      "value": {
        "type": "test-job",
        "worker": "default",
        "truncated": false,
        "timeout": 10000,
        "maxJobsToActivate": 32,
        "jobKeys": [],
        "jobs": []
      },
      "key": -1,
      "timestamp": 1656635600418,
      "position": 394,
      "valueType": "JOB_BATCH",
      "intent": "ACTIVATE",
      "recordType": "COMMAND",
      "rejectionType": "NULL_VAL",
      "rejectionReason": "",
      "brokerVersion": "8.1.0",
      "sourceRecordPosition": -1
    }
  ]
}

Environment:

• OS:
• Zeebe Version:
• Configuration: