Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-event Batches #48

Merged
merged 2 commits into from
Nov 29, 2018
Merged

Multi-event Batches #48

merged 2 commits into from
Nov 29, 2018

Conversation

bartelink
Copy link
Collaborator

@bartelink bartelink commented Nov 16, 2018

This PR moves the Access Strategy from it's present approach of having a special case "Index page" (which was arrived at as a spike impl of a performance improvement for the Favorites test) to a more uniform layout which achieves at least the following objectives

  • no special case documents needing to be ignored when propagating to other systems
  • both the WIP and normal pages now equally hold events, with no waste
  • reduce intrinsic contention when lots of writers are appending to the same stream by confining writes to a single document in the bulk of cases (as opposed to current scheme which always touches N+1 docs to write N events every time)
  • reduce Request Charges and roundtrips by having fewer documents in the first instance
  • allow competing writers to share the events array, allow them to naturally "catch up" by reading events not yet fed into their projections from the index page)

One key change herein is that we now have two indices (aside from p) - i is joined by n (which corresponds to the i of the next batch (which, on the -1 / Tip batch is the expectedVersion) - this does not seem to have affected insert costs

Note while some query costs have gone up, as noted, the bulk of the tests triggering this are artificially inducing the very extra roundtrips and bandwidth that this change will ultimately reduce.

Copy link
Collaborator Author

@bartelink bartelink left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed, you've probably got these sorted already OOB but just in case...

src/Equinox.Cosmos/Cosmos.fs Outdated Show resolved Hide resolved
src/Equinox.Cosmos/Cosmos.fs Outdated Show resolved Hide resolved
src/Equinox.Cosmos/Cosmos.fs Outdated Show resolved Hide resolved
src/Equinox.Cosmos/Cosmos.fs Outdated Show resolved Hide resolved
@bartelink
Copy link
Collaborator Author

@jinglegit for avoidance of doubt, I'm working on the JS atm, in line with comments above

@bartelink
Copy link
Collaborator Author

OK, I know lots more JS now; there are some NREs, presumably due to off by one errors. Also some of the tests are failing due to needing to be adjusted to fit the new representation (the test cases assume one event per item, and max batchSize events per Batch, which needs to be multiplied by the page size where one is in force etc)

@bartelink bartelink changed the title WIP: Add Cosmos event Batch support and event accumulation mechanism Equinox.Cosmos.Core Events API Nov 21, 2018
@bartelink bartelink changed the title Equinox.Cosmos.Core Events API WIP: Multi-event Batches Nov 21, 2018
@bartelink bartelink changed the base branch from cosmos to format-change November 27, 2018 12:12
@bartelink bartelink changed the title WIP: Multi-event Batches Multi-event Batches Nov 29, 2018
@bartelink bartelink merged commit 35d6b7e into format-change Nov 29, 2018
@bartelink bartelink deleted the wip branch November 29, 2018 12:19
bartelink added a commit that referenced this pull request Nov 29, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
bartelink added a commit that referenced this pull request Dec 6, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
bartelink added a commit that referenced this pull request Dec 11, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
bartelink added a commit that referenced this pull request Dec 14, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
bartelink added a commit that referenced this pull request Dec 19, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
bartelink added a commit that referenced this pull request Dec 24, 2018
* Multi-event batches; Tip now a Batch too
* Handle startIndex within multi-item batches
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant