Readme and Documentation Questions

[voronoipotato]

Questions

Some of these have likely been answered already in the Readme, if so consider making these broad points more obvious. I read the Readme a few times but it was frequently challenging to tease out what was some very specific point, or a more broad objective.

• What kind of project is this for, are there other easier ways to build event sourcing for F#, or is this the easiest? (as in does the compatibility layer make things easier to use)
• What is the scope of Equinox, or what features have been consciously omitted?
• Broadly speaking what are the goals of using this instead of using one particular kind of EventStore database directly? Does it just allow me to swap out "event stores" or does it do more?
• Should I reach for Equinox first if I'm doing DDD + CQRS + ES, even if I may not need to change the event store database?
• Is this layer constructed for ease of use, or does it make things more challenging toward some end?
• Do you recommend this for the average person who is building an event sourcing project with F# or should I only use Equinox if I need massive scalability?
• What are some of the other good options for F# event sourcing, is this the only fleshed out option?
• How do I get started? (development vs production)
• How do I use Equinox? (development vs production)
• Is there an API documentation? Where can I find it?
• You say I can use volatile memory for integration tests, could this also be used for learning how to get started building event sourcing programs with equinox?
• Is there a guide to building the simplest possible hello world "counter" sample, that simply counts with an add and a subtract event?

[bartelink]

Hey, thanks so much for the interest, and for spurring me on to extract #50 into https://github.com/jet/equinox/wiki/Programming-Model

I'll do type and run answers for now, but I really appreciate the fact you've taken the time to provide this feedback - it's immensely valuable to get input from someone who is approaching this repo without context, so 🙇 for being the first one to take the plunge...

Some of these have likely been answered already in the Readme, if so consider making these broad points more obvious. I read the Readme a few times but it was frequently challenging to tease out what was some very specific point, or a more broad objective.

Yeah, that's not a surprise, even if it is disappointing - lets see if we can remediate...

What kind of project is this for

While the ES store side stems from a significant API layer in Jet, where it's running the show, this is intended to be able to fit all of Jet's needs - there's a team working on this, and the intent is to support a broad array of use cases.

Personally, having used NEventStore and EventStore to varying degrees before Jet, it answers needs I've encountered in doing other systems too.

The juggling act is to to keep the abtractions count low while delivering something that works reliably and with good performance.

are there other easier ways to build event sourcing for F#, or is this the easiest?

1. there's absolutely no limit - many of F#'s features can be viewed as almost custom designed for building event-sourced solutions
2. while ease of use and approachability is a concern (see tail of Facilitate running load tests inside WebHost #55 for teasers), given the fact that this is running under load with constraints, the odds are against ti meeting anyone's definition of easy (TL;DR Simple is not Easy 😁 )

(as in does the compatibility layer make things easier to use)

While there is a compatibility layer, the common layer is more about common concerns such as optimistic concurrency retry loops than actually trying to be a compatibility layer (look in the history - EventStore and Memory Store were the only stores for a long time and the abstractions still had a purpose)

What is the scope of Equinox, or what features have been consciously omitted?

None consciously, but assessing what to put in would consider
a) broad implementability/applicablity across stores
b) likelihood of broad use of a given feature considering the complexity it brings to the table

Broadly speaking what are the goals of using this instead of using one particular kind of EventStore database directly? Does it just allow me to swap out "event stores" or does it do more?

1. unified programming model regardless of stores, raising the level of abstraction and avoiding Domain code touching the specific Store (easy to say, harder to stick to!)
2. It can be used as a way to migrate between stores

But really it depends and/or it's too early to say.

Should I reach for Equinox first if I'm doing DDD + CQRS + ES, even if I may not need to change the event store database?

have to run -- will do terse answers in a timebox and then come back and edit...

Is this layer constructed for ease of use, or does it make things more challenging toward some end?

I'd argue you end up with a good balance of simple and easy

Do you recommend this for the average person who is building an event sourcing project with F# or should I only use Equinox if I need massive scalability?

I would. But there are schools of thought that would say you should build out your own equivalent - an option thats definitely opened up by this being open source

What are some of the other good options for F# event sourcing, is this the only fleshed out option?

FsUno.Prod is the closest I know, though there is another Cosmos impl out there in the last few months - have not looked in depth at it.

How do I get started? (development vs production)

I need to build out the samples/Store/Web in the cosmos branch see #55

How do I use Equinox? (development vs production)

1. It's a library 2. yes, this needs docs

Is there an API documentation? Where can I find it?

The CI is being rigged to put this on nuget - when its there, fuget.org will help. For now, reading eventstore.fs, equinox.fs and cosmos.fs is the best approach.

You say I can use volatile memory for integration tests, could this also be used for learning how to get started building event sourcing programs with equinox?

Yes, but its not hard to stand up a local EventStore, and that gives you lots more - Equinox atm only loads and stores events.

Is there a guide to building the simplest possible hello world "counter" sample, that simply counts with an add and a subtract event?

TL;DR No but there needs to be ☹️

See #55, samples/Store on the cosmos branch and the feature/inventory-sample branch

Is there a good one you'd like to see an equivalent of ? In general, these little examples, esp if not informed by an actual domain can be problematic (how do you count meaningfully idempotently in an event-sourced fashion ? even the m-r sample would probably make Greg shudder now) - ultimately favorites is about as simple as it gets - see the cutdown version in #50 and the cosmos branch samples/store/web

I might post one (I'm leaning toward building todomvc backend - let me know if that makes sense)

Have to 🏃 now but wanted to get you a quick response - please feel free to post followups here ;)

[voronoipotato]

Thank you so much for your time and no you shouldn't be disappointed. The questions that needs answering often only become obvious after they have been asked. I would like to see the simplest example, a counter that accepts added and subtracted events. It's often used like a hello world in the MVU space.

Right now when I read the samples it's unclear to me what is required to get anything working at all, vs what is nice to have, vs what you should have in production but maybe isn't required to get it working.

[voronoipotato]

Looking at the example I tried to build a kind of event store "Hello World". Let me know if any of this makes sense to you...

Counter.fs

(*Events are things that have already happened, 
they always exist in the past, and should always be past tense verbs*)
type Event = 
    | Incremented
    | Decremented
    | Cleared of int 
(*A counter going up might clear to 0, 
but a counter going down might clear to 100. *)

type State = State of int

(*Evolve takes the present state and one event and figures out the next state*)
let evolve state event =
    match event, state with
    | Incremented , State s -> State(s + 1)
    | Decremented, State s -> State(s - 1)
    | Cleared x , _ -> State x

(*fold is just folding the evolve function over all events to get the current state
  It's equivalent to Linq's Aggregate function *)
let fold state events = Seq.fold evolve state events 

(*Commands are the things we intend to happen, though they may not*)
type Command = 
    | Increment
    | Decrement
    | Clear of int

(*Decide consumes a command and the current state to decide what events actually happened.
  This particular counter allows numbers from 0 to 100.*)
let decide command state = 
    match command with
    | Increment -> 
        if state > 100 then [] else [Incremented]
    | Decrement -> 
        if state <= 0 then [] else [Decremented]
    | Clear i -> 
        if state = i then [] else [Cleared i]

What I need to figure out now is how to hook this Domain into Equinox such that eventstore actually stores these and can generate a projection.

[bartelink]

Nice - looks good so far. You could add a Test into the CLI's Tests in Program.fs ? If you have a branch somewhere I can review (afk for about 15 mins now tho)

[bartelink]

if you look in the more-samples branch, it shows how Favorites and Saves are wired into a) the Web app and b) the CLI (NB for a real app, you wouldn't actually connect it to multiple stores; life is complex enough without that)

[bartelink]

(I'm working on something right now and will work this into the branch when I hit that organically in a few hours unless you say otherwise)

[voronoipotato]

I'll check it out but I doubt I'll get somewhere meaningful tonight. I'm probably juggling too many hobby tasks.

[voronoipotato]

My end goal is to write an (eventstore + equinox + giraffe) + (fable + elmish) sample. I've gotten the Giraffe + Fable + Elmish path more or less understood. I basically just use Giraffe with saturn to generate my routes, and I use a restful api that I consume with fable/elmish. Elmish already uses the same kind of feel of building the present state from a list of events, however it doesn't store them for later.

[bartelink]

Sounds like it'll make a very nice sample in the end. If you post a repo link (I'm trying to get gitter rigged for this project as I've enough slack tabs open). But feel free to maintain a holding pattern and wait till I get it the sample in here into a shape where it will be semi-obvious (I'm presently reorganizing Equinox.fs, triggered by your excellent questions)

[voronoipotato]

https://github.com/SAFE-Stack/SAFE-ConfPlanner

This SAFE-ConfPlanner isn't a bad sample for a full website attempt at event-store architecture if you want some inspiration. I would argue this example is at the far end of events everywhere though.

[voronoipotato]

I'll probably for my example have my Restful api modeled after "commands" which then return the present state after processing the command.

I don't have a repo link just yet, but I'll probably get one set up tonight after I get off work. This was what I slapped together during my 10% time.

[bartelink]

I'll probably for my example have my Restful api modeled after "commands" which then return the present state after processing the command.

Right; Equinox facilitates that CQ (as opposed to CQRS) pattern (yielding a render of the state post the command); it should be mentioned that that's not necessarily a panacea of course.

I don't have a repo link just yet, but I'll probably get one set up tonight after I get off work. This was what I slapped together during my 10% time.

10%, that's a new one on me ;) Looking forward to seeing/hearing

[bartelink]

BTW thanks to @michaelliao5 you'll find packages on nuget now; see https://www.fuget.org/packages/Equinox.EventStore/ etc (CI is in the works)

[bartelink]

This SAFE-ConfPlanner isn't a bad sample for a full website attempt at event-store architecture if you want some inspiration. I would argue this example is at the far end of events everywhere though.

The problem isn't inspiration/imagination - it's trying to constrain it, esp wrt this repo, which needs to maximize a) loading fast in VS b) building fast in CI.

A key concern at hand atm for the Equinox project is to load test and scale on .NET Core on both the ES and Cosmos sides. The min requirement for doing meaningful load testing is essentially having an aspnetcore selfhost to feed parallel requests through, with a UI as a basic sanity check being a nice bonus, but not something I want to add the complexity of to the repo (think implementing a TodoBackend so FE can be TodoMvc).

As alluded to in the tail of #55, a complementary thing which would be very welcome is indeed a bridge to something like ConfPlanner; Time is tight atm but I'm up for assisting if you have stuff I can look at and/or push to.

Another idea is to have a yeoman type generator be able to spin out an aspnetcore/saturn/giraffe app with an equinox.cosmos and/or equinox.eventstore backend (with the complexity of handling multiple stores only entering the picture when you actually want and need an app that's able to store in either ES or Cosmos). This lowers the bar for people with lower tenacity levels than you seem to be blessed with ;)

[voronoipotato]

Nice! Yeah for me the goal is gaining understanding and building examples to share that understanding with others. So I'm probably going to build up examples starting from very very small like the "Counter" example to something more fully fledged. I know the dotnet core template builder is pretty popular in the community.Something like Dotnet new SAFEEE -lang "f#" (Saturn Azure Fable Elmish Equinox EventStore of course)

[voronoipotato]

So you were saying that Equinox focuses more on the CQ instead of the full CQRS, but can I still project out to say a SQL Database if I wanted to (say for consumption by another app)? If yes, would it be a lot more work or only a little more work?

[bartelink]

SAFEEE -lang "f#" (Saturn Azure Fable Elmish Equinox EventStore of course)

Haha nice - Saf3 you mean ;P

So you were saying that Equinox focuses more on the CQ instead of the full CQRS

I deliberately said it facilitated it; and that such a pattern is not a panacea. But neither is CQRS; it depends.

This Equinox codebase as it stands does not have an opinion / address this question. Across Jet, we have a variety of systems that do most possible combinations of what you might ever wish to do - by necessity, Equinox would absolutely not constrain one from doing so.

There are no announcements in this space at this time, but suffice to say it exists in various forms and the need is not going anywhere ;)

but can I still project out to say a SQL Database if I wanted to (say for consumption by another app)? If yes, would it be a lot more work or only a little more work?

Yes; you absolutely can.

But how and whether you do that depends on your needs. Projections/denormalizations (especially async/eventually consistent ones) can take many forms from purely in-memory (if the nature of the queries is such that it fits), through redis and onwards to more SQL DBs etc.

Keeping the choice open is pretty important from the point of view of Core Equinox.

Having said all that, a decent starter solution, esp for a sample is to:
a) have the state maintained by your model's fold/evolve incorporate both the your decide needs
b) 🙈 add more into the state to address what's needed to be able to handle querying efficiently without actually going around loading your Store with queries
c) use the unfolds mechanism (aka snapshot) backed by an appropriate AccessStrategy.* and/or a Cache to ensure you're not going to kill your store in a read-heavy work load

Specifically:

• on Cosmos, as alluded to in the WIP doc, for instance Equinox.Cosmos's AccessStrategy supports keeping what could/should otherwise be a projection as an unfold, which means a Cosmos point read gets you a consistent version of the projected state (and if you use a cache and its a hit, you only pay one RU).
• on EventStore, you also probably use a Cache (but are not saving RUs/money, just some latency and load on your store), and (depending on whether it fits) use RollingSnapshots to avoid having to read back to the start of the stream when you get a cache miss

[bartelink]

Doing a quick scan of the conference planner... doing in-host projections (esp eventually consistent ones and/or using the Mailbox processor) has limited value from my perspective). I want my code to be stateless meaning a) I can boot it in 2 seconds b) I can kill -9 it c) I can run N of them. Equinox deals with that world very well. Eventually consistent propagation / projection / denormalization of events from the store is an orthogonal concern to me - e.g.

• you can use EventStore's various projection and subscription mechanisms
• On Cosmos, you can hang off the changefeed

A lot of wiring code falls away out of your webapp if you take the above to its logical conclusion.

Then you're left with a question of doing one of:

• deciding you're going to 🙈 and lean on Equinox unfolds and caching to read your writes consistently (any normal conference will do fine with that)
• deciding you're going whole hog; build proper CQRS with separated eventually consistent projection not inside your web host (you front end needs the smarts to deal with that)

It looks doable to remove the storage and event propagation and replace it with Equinox stuff to me.

I guess its kinda like the Fowler "start with a monolith" advice - get something working and complete (not writing to a local file and able to deal with clustering and host restarts), then deal with scaling stuff out by pulling stuff out to hang off projections where necessary.

Obviously you can't build an entire large system that needs to scale beyond the limitations of all stuff hitting your store synchronously with this approach, but going half the way and building something that follows the patterns and layout but does not achieve the goal will teach you lots (I loved playing with and thinking about possibilities with MailboxProcessor but its not a thing I have cause to use IRL), but it won't achieve a real scalable system.

[bartelink]

I've spent some time editing the https://github.com/jet/equinox/wiki based on all this - please let me know if any sections don't add value or there's anything that would really be worth covering; I'm parking that for now and getting back to working on #55. I intend to keep this issue open until we either have a good end to end Counter sample or something better when one takes this repo and the wiki together.

[bartelink]

@voronoipotato I'm going to leave this open as a marker for the fact that we should get this to the point where we have your counter example somewhere - (likely in a repo of yours) (we don't have CI rigged yet unfortunately as other work is taking precedence).

Out there idea: Perhaps we could even make it be a fully fledged Test like Favorite and SaveForLater ? Hard to know what a meaningful version of such a test would be though ;)

I hope we're slowly getting places wrt addressing some of the shortcomings you identified at the start of this too...

[bartelink]

Closing this now on the following grounds:

# install tools, templates
dotnet tool install -g equinox.tool
dotnet new -i equinox.templates

# make and stand up sample app with memory store
dotnet new equinox.web -t  # -t for todos defaults to memory store (-m)
dotnet run -p Web
start https://www.todobackend.com/client/index.html?https://localhost:5001/todos
start https://www.todobackend.com/specs/index.html?https://localhost:5001/todos

# start ES
# requires admin privilege
cinst eventstore-oss -y # where cinst is an invocation of the Chocolatey Package Installer on Windows
# run as a single-node cluster to allow connection logic to use cluster mode as for a commercial cluster
& $env:ProgramData\chocolatey\bin\EventStore.ClusterNode.exe --gossip-on-single-node --discover-via-dns 0 --ext-http-port=30778

# make and stand up sample app
dotnet new equinox.web -t -e # -t for todos, -e for eventstore
dotnet run -p Web

# run on cosmos
# TODO export 3x env vars (see readme.md)
eqx init cosmos
dotnet new equinox.web -t -c --force # -t for todos, -c for cosmos, --force to overwrite previos expansion of template app
dotnet run -p Web

[bartelink]

This is all 1.0.1 stuff, would aporeciate some feedback - will be afk for a bit but will be checking around here over the holidays dipping in and out

[bartelink]

Updated FAQ re counter sample - we have an Aggregate.fs and a TodoService.fs in the dotnet new template which don't directly include a Counter example but fall either side of that level of complexity.

@voronoipotato may I suggest making a gist out of your code? I considered including it as an option in the template, but think it's not a 100% fit (your heading comments over the functions are excellent pithy summaries which are ideal for an example, but IMHO are inappropriate for a template, i.e., if you kept seeing those same comments over all the evolve and interpret functions in a system, your eyes would glaze over and it could (arguably) distract one from adding domain-relevant comments rather than dry meta-comments about event sourcing) - I can add a link to that into the FAQ (the FAQ and the wiki in general will likely shortly move into a /docs folder in the main repo)

[bartelink]

Wiki now lives in https://github.com/jet/equinox/blob/master/DOCUMENTATION.md

[bartelink]

master now has a pretty much feature-complete preview of a CosmosDb based projection loop, which changes some of the above answers (notably that an Equinox-based conference planner app would probably now incorporate a ChangeFeedProcessor somewhere in the implementation) - most of the information in the above now lives in https://github.com/jet/equinox#faq, where there are in-date answers

[bartelink]

#96 introduces a Counter.fsx deriving from @voronoipotato's example above