RFC: Persist state across Sage executions

[AndrewDryga]

We can add a persistence adapter for Sage. The main goal of implementing it is to let users recover after node failures, making it possible to get better compensation guarantees. As I see it we need:

1. Ability to use adapters (probably Ecto and RabbitMQ) to persist execution stages;
2. Callbacks/Behaviour to define loader that resumes persistent compensations that are not completed.
3. We need to make sure that compensations are processed "exactly once", or "at least once" (we need to take a tradeoff here).

This mechanism should be prone to network errors, eg. we should persist a state before executing a transaction, so that even if the node is restarted, compensation (even trough without knowing anything about the effect) would still start, potentially on another node.

[michalmuskala]

The simplest solution could be to use the :disk_log module - it powers mnesia transactions.

[AndrewDryga]

@michalmuskala I've read through :disk_log and it looks like it's not the best fit for this problem:

1. There is no way to clean up old logs, we don't want them to grow infinitely and it's safe to remove effects after Sage execution or compensation is completed. Otherwise, we would have lots of log entries which are never used.

2. Running it in a distributed mode is not that trivial, even for this module it's recommended to have a wrapper process on each node that sync local disk logs.

We need something with fast writes and deletes by sacrificing reads, like Log-Structured Merge Trees. But I'm not aware of good implementation which would not bring too much maintenance burden.

Any ideas?

[AndrewDryga]

Implementation can work something like this:

1. We add a persistence adapter to the Sage interface
2. Before any transaction is executed we persist fact that we started the execution and current state (so if a node fails before a transaction is succeeded or async task is spawned, we would still run compensations for them).
3. After a transaction is executed, we update persistent state to include created effect.
4. Same applies for compensations, but we update state only when compensation is succeeded. We must not assume that effect is compensated just because we executed the compensation.
5. After Sage is execution is finished (either successfully or after compensating all effects), we delete persisted data
6. Sage.Executor should provide a function to run all compensations for a given Sage from a saved checkpoint
7. We need to implement simple adapter which persists Saga state to a file and restarts compensations when node is started.

Probably we should remove compensation error handler since it looks obsolete with an ability to persist and retry compensations after a critical failure.

Questions:

1. How to continue the execution on another node? Sage struct includes anonymous functions an mfa tuples, it's possible that there would be no code to handle compensations on a target node.

[michalmuskala]

I think we should keep it all local - at least in the first iteration. I don't think it's the distribution of the log (or transaction persistence) would be actually that useful.

The simple adapter could just use a dets table - it might not be the best tool for the job, but it's a no-dependencies solution that should allow validating the idea.

Lastly, saving funs might be a problem - any code change would basically make it irrecoverable (since serialisation for funs includes the md5 hash of the module). Maybe the persistence feature should only work with mfa tuples?

[AndrewDryga]

@michalmuskala Sage which relies on MFA instead of callbacks in a form &do_something/3 is so much harder to read :(. I'll think more about it.

[AndrewDryga]

If we further extend persistence with external transaction ID we would be able to trigger recovery externally by executing the same Sage with the same ID and attributes, which would provide idempotency feature described #2.

[AndrewDryga]

We can use LSM DB from NHS https://github.com/martinsumner/leveled

[keathley]

I think we should keep it all local - at least in the first iteration. I don't think it's the distribution of the log (or transaction persistence) would be actually that useful.

I think this is fine for an initial pass but I'm not sure this is a correct default to have long term. It would be problematic for our uses. But as long as persistence is behind an adapter as described everything should be fine since we can swap in our own persistence.

Thanks for this library and the thought you're putting into it 👍.

[deepankar-j]

@AndrewDryga, is there any update on this issue? Thanks!

[AndrewDryga]

Hey @deepankar-j. I'm not spending much time adding features to Sage as it fits our use cases at a time as it is now. So if anyone wants - this one is open for grabs.

[deepankar-j]

Thanks @AndrewDryga. Out of curiosity, how do you handle deployments for your system while a saga is the middle of executing.

We have a three-node cluster on which we do a rolling update. This is why I'm interested in the execution log and saga recovery. However, I imagine that updating in the middle of a saga execution might not be as big of an issue for blue-green deployments.

[chulkilee]

I think the goal is to handle process crash - not just node failure...

In my opinion this shouldn't be special to sage. For example, the whole context can be rehydrated before calling sage execution (or at the first step of sage) and the all steps should be able to handle that.

In my case we keep everything in the database, and each sage step are idempotent with the information available in database - e.g. when we know it's already done (e.g. an attr for external resource is already set ) then execution step would be no-op.