Fish: Deal with ELECTED Applications on restart

[sparshev]

#64 OOM triggered another issue - that if something happens during Application allocation - it will stuck in ELECTED state and will be abandoned after restart. Most probably that will not happen in the cluster (because multiple nodes will look at the election process and notice when the Application was not ALLOCATED in time), but that should not happen even in one node configurations.

Expected Behaviour

We need to clean-up and mark the Application Allocation as ERROR or try to continue the allocation if it's possible.

Actual Behaviour

The application is not picked up upon node startup

Reproduce Scenario (including but not limited to)

Should be relatively easy to reproduce by killing the node right during allocation and then restarting it

Steps to Reproduce

1. Run the Fish node
2. Try to Allocate something
3. Kill the node during allocation (as hard as possible)
4. Start the node and see that Application is not picked up and in ELECTED state forever

Platform and Version

• Ubuntu 20.04.6 LTS
• Aquarium Fish v0.7.1 (231111.070935)

Logs taken while reproducing problem

• Check logs in AWS: OOM during execution #64
• How Application State looks after restart:

  {"UID":"12050150-63b1-4ce6-9a1f-3876c06bb82e","application_UID":"12050150-63b1-463f-a0c0-340be13ab1bc","created_at":"2024-05-03T15:27:34.442634343Z","description":"Elected node: <node_name>","status":"ELECTED"}