Derive

DISCLAIMER: This is an early project used internally at our company. We make no guarantees about its stability. Use at your own risk!

Derive provides the infrastructure to keep derived state up to date based on a source stream of events. This is commonly used for event sourcing or redux patterns.

For now, the state can be stored in Ecto or in memory.

The state is eventually consistent, but there are functions which allow waiting until a reducer has caught up to a certain point.

There are 3 different scenarios that Derive handles:

1. Rebuilding state from scratch
2. Incrementally keeping state up to date based on new events
3. Resuming in the event of a restart such as a server shut-down

Installation

If available in Hex, the package can be installed by adding derive to your list of dependencies in mix.exs:

def deps do
  [
    {:derive, "~> 0.1.0"}
  ]
end

Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/derive.

Basic usage

See the tests directory for now.

TODO

• Support rebuilding very large reducers
• Render progress when rebuilding large reducers
• Error handling when a commit fails
• Add tests for Ecto DB operations
• Create a Derive.EctoReducer with reduced boilerplate
• Improved logging of operations, commits, etc
• Rename version to cursor
• Integrate with turtle-api
• Allow event logs to become part of the Derive supervision tree (to minimize configuration)
• Allow awaiting events in Derive after a command is executed
• Shutdown Derive.PartitionDispatcher after a period of inactivity
• Pattern for sharing event logs between multiple Derive instances
• Make the code cursor agnostic
• Test recovery in case of an unexpected shutdown
• Log errors for partitions in the database
• Support use cases like email notifications
• Error handling for Derive.Notifier failures