Fix #341: Optimization: Callbacks invocation

[jnpsk]

Fix #341 by implementing the Outbox Pattern.

Instead of dealing with callback sending in the business logic, an CallbackUrlEvent is created and published for an event listener. The listener, implemented in CallbackUrlEventDispatcher.class, then processes the CallbackUrlEvent in a separate thread and transaction.

[banterCZ]

Good job. Please update also the documentation. I have added several comments to consider.

[romanstrobl]

Good job. I have several concerns, especially related to performance and data migration which should be handled or answered.

[banterCZ]

Great, almost done. Let's polish it a little bit.

[banterCZ]

Looks good to me. @korbelm @zcgandcomp May we do performance testing from this branch before merging?

[zcgandcomp]

Looks OK

[romanstrobl]

I suggest we run load tests on this branch before merging. Otherwise I approve the changes.

[jnpsk]

Coming with a significant changes due to inability to use the used locking mechanism on MSSQL.

Now the approach works as follows:

• States of a callback event are:
  • INSTANT -> {PENDING (on full Executor’s queue), PROCESSING (when dispatched), COMPLETED (on Callback’s maxAttempts <= 0)}
  • PENDING -> {PROCESSING (when dispatched), COMPLETED (on Callback’s maxAttempts <= 0)}
  • PROCESSING -> {FAILED, COMPLETED}
  • FAILED -> {PROCESSING (on retry)}
  • COMPLETED -> {}

On an activation or operation change a new CallbackUrlEventEntity (callback event) is created and initialised to state INSTANT during
the main business transaction TrB. Only after successful commit of the TrB, the callback event is enqueued to be dispatched by via a ThreadPoolExecutor. If the ThreadPoolExecutor is busy, it rejects to dispatch the callback event and so the callback event will be transitioned to the state PENDING in database. And so, after the queueing, the callback event is either stored in database in state PENDING, or is stored in database in state INSTANT and is, at the same time submitted, to ThreadPoolExecutor.

• ThreadPoolExecutor and its queue is thread safe and so the callback event is processed once. There is no other consumer for INSTANT events.
• PENDING events are processed by a scheduled job with distributed locks, and so only one job processes PENDING events in the same time. Similarly for FAILED events.

[banterCZ]

In general, looks good. I have added several ideas to consider. Did it pass the performance tests? Let's wait for other reviews.

[banterCZ]

When I see Throwable, I am curious whether we are supposed to handle even Error. I suggest to avoid that.

[banterCZ]

Let's run the Coverity scan after merging this. Last time, we overlooked something.

[romanstrobl]

I found several minor additional issues.