Fix/limit schedule date alert jobs

[ulferts]

Safeguards against having too many ScheduleDateAlertsNotificationsJob instances by applying GoodJob's concurrency limitations. Doing so leads to always only having max one such job executed at the same time and max one such job queued. That queued job might get automatically rescheduled by GoodJob if the currently active job is still busy.

The change also prevents multiple jobs covering the same time interval. Now, a job will only consider time zones which are between the cron_at value of the previous job + 15 minutes till the cron_at value of the currently executed job. The cron_at value is independent of when the job is actually run so it is safe against delays. It is also safe against cases where one or multiple jobs are missed to be scheduled.

There is more explanation within the comments next to the code

---

https://community.openproject.org/wp/55101

[mereghost]

Looks good. One thing that @cbliard brought to our attention is that the concurrency limits will retry forever on a exponential backoff.

Should we add a discard_on call for the concurrency checks after a couple (let's say 5) tries?

[ulferts]

Looks good. One thing that @cbliard brought to our attention is that the concurrency limits will retry forever on a exponential backoff.

Should we add a discard_on call for the concurrency checks after a couple (let's say 5) tries?

I'll issue another PR that applies the same concurrency pattern to the ScheduleReminderMailsJob and with it will add a retry_on for both by which the jobs will be retried twice with a backoff of 5 minutes and then discarded. Because after 15 minutes, the next job can be scheduled and take over.