One cluster node's JVM suddenly goes down during task execution, resulting in the RecurringTask being unable to be executed

[acupofsea]

Context

• DB-Scheduler Version : 13.0.0
• Java Version : JDK1.8
• Spring Boot (check for Yes) : [√]
• Database and Version : postgresql

Description

This is a two-node cluster where JVM suddenly exits while node A is executing a RecurringTask, and the database picked, picked_by, and last_heartbeat columns have values. At this time, Node B cannot execute the task at the next execution_time. When node A comes online, it still cannot execution the task

Steps to Reproduce the bug

1. Running cluster with two nodes
2. Node A's JVM exits while executes RecurringTask but not finished. At the same time, the JVM showdown thread is also been killed.
3. Node B is running, but RecurringTask is not executed
4. Node A comes online, but RecurringTask is not executed

[kagkarlsson]

You can read about dead executions in the readme. Checkout heartbeat-tuning for faster revive of dead executions