-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Updated to clarify what a checkpoint is and how to create it #790
base: master
Are you sure you want to change the base?
Changes from all commits
8d7c968
2e1fbb6
ad9bb97
e97c7f5
d85d9b5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -8,15 +8,15 @@ Subscriptions allow you to subscribe to a stream and receive notifications about | |||||
|
||||||
You provide an event handler and an optional starting point to the subscription. The handler is called for each event from the starting point onward. | ||||||
|
||||||
If events already exist, the handler will be called for each event one by one until it reaches the end of the stream. From there, the server will notify the handler whenever a new event appears. | ||||||
If events already exist, the handler will be called for each event one by one until it reaches the end of the stream. The server will then notify the handler whenever a new event appears. | ||||||
|
||||||
:::tip | ||||||
Check the [Getting Started](getting-started.md) guide to learn how to configure and use the client SDK. | ||||||
::: | ||||||
|
||||||
## Subscribing from the start | ||||||
|
||||||
When you need to process all the events in the store, including historical events, you'd need to subscribe from the beginning. You can either subscribe to receive events from a single stream, or subscribe to `$all` if you need to process all events in the database. | ||||||
If you need to process all the events in the store, including historical events, you'll need to subscribe from the beginning. You can either subscribe to receive events from a single stream or subscribe to `$all` if you need to process all events in the database. | ||||||
|
||||||
### Subscribing to a stream | ||||||
|
||||||
|
@@ -30,33 +30,33 @@ When you subscribe to a stream with link events, for example the `$ce` category | |||||
|
||||||
### Subscribing to `$all` | ||||||
|
||||||
Subscribing to `$all` is much the same as subscribing to a single stream. The handler will be called for every event appended after the starting position. | ||||||
Subscribing to `$all` is similar to subscribing to a single stream. The handler will be called for every event appended after the starting position. | ||||||
|
||||||
@[code{subscribe-to-all}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
## Subscribing from a specific position | ||||||
|
||||||
The previous examples will subscribe to the stream from the beginning. This will end up calling the handler for every event in the stream and then wait for new events after that. | ||||||
The previous examples subscribed to the stream from the beginning. This called the handler for every event in the stream and then waited for new events. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Both the stream and $all subscriptions accept a starting position if you want to read from a specific point onward. If events already exist at the position you subscribe to, they will be read on the server side and sent to the subscription. | ||||||
Both stream and $all subscriptions accept a starting position if you want to read from a specific point onward. If events already exist at the position you subscribe to, they will be read on the server side and sent to the subscription. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
Once caught up, the sever will push any new events received on the streams to the client. There is no difference between catching up and live on the client side. | ||||||
Once caught up, the server will push any new events received on the streams to the client. There is no difference between catching up and live on the client side. | ||||||
|
||||||
::: warning | ||||||
The positions provided to the subscriptions are exclusive. You will only receive the next event after the subscribed position. | ||||||
::: | ||||||
|
||||||
### Subscribing to a stream | ||||||
|
||||||
To subscribe to a stream from a specific position, you need to provide a *stream position*. This can be `Start`, `End` or a *big int* (unsigned 64 bit integer) position. | ||||||
To subscribe to a stream from a specific position, you must provide a *stream position*. This can be `Start`, `End` or a *big int* (unsigned 64 bit integer) position. | ||||||
|
||||||
The following subscribes to the stream `some-stream` at position `20`, this means that events `21` and onward will be handled: | ||||||
|
||||||
@[code{subscribe-to-stream-from-position}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
### Subscribing to $all | ||||||
|
||||||
Subscribing to the `$all` stream is much like subscribing to a regular stream. The only difference is how you need to specify the stream position. For the `$all` stream, you have to provide a `Position` structure instead, which consists of two big integers - prepare and commit positions. The `Position` value can be `Start`, `End` or a `Position` created from a commit and prepare position. | ||||||
Subscribing to the `$all` stream is similar to subscribing to a regular stream. The only difference is how to specify the stream position. For the `$all` stream, you must provide a `Position` structure instead, consisting of two big integers - prepare and commit positions. The `Position` value can be `Start`, `End`, or a `Position` created from a commit and prepare position. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. not sure if that's 100% accurate. Please review |
||||||
|
||||||
The corresponding `$all` subscription will subscribe from the event after the one at commit position `1056` and prepare position `1056`. | ||||||
|
||||||
|
@@ -74,19 +74,19 @@ And the same works with `$all` : | |||||
|
||||||
@[code{subscribe-to-all-live}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
This won't read through the history of the stream, but will rather notify the handler when a new event appears in the respective stream. | ||||||
This will not read through the history of the stream but will notify the handler when a new event appears in the respective stream. | ||||||
|
||||||
Keep in mind that when you subscribe to a stream from a certain position, as described [above](#subscribing-from-a-specific-position), you will also get live updates after your subscription catches up (processes all the historical events). | ||||||
Keep in mind that when you subscribe to a stream from a specific position, as described [above](#subscribing-from-a-specific-position), you will also get live updates after your subscription catches up (processes all the historical events). | ||||||
|
||||||
## Resolving link-to's | ||||||
|
||||||
Link-to events point to events in other streams in EventStoreDB. These are generally created by projections such as the `$by_event_type` projection which links events of the same event type into the same stream. This makes it easier to look up all events of a certain type. | ||||||
Link-to events point to events in other streams in EventStoreDB. These are generally created by projections such as the `$by_event_type` projection which links events of the same event type into the same stream. This makes it easier to look up all events of a specific type. | ||||||
|
||||||
::: tip | ||||||
[Filtered subscriptions](subscriptions.md#server-side-filtering) make it easier and faster to subscribe to all events of a certain type or matching a prefix. | ||||||
[Filtered subscriptions](subscriptions.md#server-side-filtering) make it easier and faster to subscribe to all events of a specific type or matching a prefix. | ||||||
::: | ||||||
|
||||||
When reading a stream you can specify whether to resolve link-to's or not. By default, link-to events are not resolved. You can change this behaviour by setting the `resolveLinkTos` parameter to `true`: | ||||||
When reading a stream you can specify whether to resolve link-to's. By default, link-to events are not resolved. You can change this behaviour by setting the `resolveLinkTos` parameter to `true`: | ||||||
|
||||||
@[code{subscribe-to-stream-resolving-linktos}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
|
@@ -100,19 +100,19 @@ The possible reasons for a subscription to drop are: | |||||
|
||||||
| Reason | Why it might happen | | ||||||
|:------------------|:---------------------------------------------------------------------------------------------------------------------| | ||||||
| `Disposed` | The subscription got cancelled or disposed by the client. | | ||||||
| `Disposed` | The client canceled or disposed of the subscription. | | ||||||
| `SubscriberError` | An error occurred while handling an event in the subscription handler. | | ||||||
| `ServerError` | An error occurred on the server, and the server closed the subscription. Check the server logs for more information. | | ||||||
|
||||||
Bear in mind that a subscription can also drop because it is slow. The server tried to push all the live events to the subscription when it is in the live processing mode. If the subscription gets the reading buffer overflow and won't be able to acknowledge the buffer, it will break. | ||||||
|
||||||
### Handling subscription drops | ||||||
|
||||||
An application, which hosts the subscription, can go offline for a period of time for different reasons. It could be a crash, infrastructure failure, or a new version deployment. As you rarely would want to reprocess all the events again, you'd need to store the current position of the subscription somewhere, and then use it to restore the subscription from the point where it dropped off: | ||||||
An application, which hosts the subscription, can go offline for some time for different reasons. It could be a crash, infrastructure failure, or a new version deployment. As you rarely would want to reprocess all the events again, you'd need to store the current position of the subscription somewhere, and then use it to restore the subscription from the point where it dropped off: | ||||||
|
||||||
@[code{subscribe-to-stream-subscription-dropped}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
When subscribed to `$all` you want to keep the position of the event in the `$all` stream. As mentioned previously, the `$all` stream position consists of two big integers (prepare and commit positions), not one: | ||||||
When subscribed to `$all` you want to keep the event's position in the `$all` stream. As mentioned previously, the `$all` stream position consists of two big integers (prepare and commit positions), not one: | ||||||
|
||||||
@[code{subscribe-to-all-subscription-dropped}](@grpc:subscribing_to_stream.py;subscribing-to-streams.js;subscribing-to-streams.ts;subscribing_to_stream/SubscribingToStream.java;subscribing-to-streams/Program.cs;subscribingToStream.go;subscribing_to_stream.rs) | ||||||
|
||||||
|
@@ -126,12 +126,12 @@ The code below shows how you can provide user credentials for a subscription. Wh | |||||
|
||||||
## Server-side filtering | ||||||
|
||||||
EventStoreDB allows you to filter the events whilst you subscribe to the `$all` stream so that you only receive the events that you care about. | ||||||
EventStoreDB allows you to filter the events whilst subscribing to the `$all` stream to only receive the events you care about. | ||||||
|
||||||
You can filter by event type or stream name using either a regular expression or a prefix. Server-side filtering is currently only available on the `$all` stream. | ||||||
You can filter by event type or stream name using a regular expression or a prefix. Server-side filtering is currently only available on the `$all` stream. | ||||||
|
||||||
::: tip | ||||||
Server-side filtering introduced as a simpler alternative to projections. Before creating a projection to get the events you care about you should first consider filtering. | ||||||
Server-side filtering was introduced as a simpler alternative to projections. You should consider filtering before creating a projection to include the events you care about. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
::: | ||||||
|
||||||
A simple stream prefix filter looks like this: | ||||||
|
@@ -142,7 +142,7 @@ The filtering API is described more in-depth in the [filtering section](subscrip | |||||
|
||||||
### Filtering out system events | ||||||
|
||||||
There are a number of events in EventStoreDB called system events. These are prefixed with a `$` and under most circumstances you won't care about these. They can be filtered out by passing in a `SubscriptionFilterOptions` when subscribing to the `$all` stream. | ||||||
There are events in EventStoreDB called system events. These are prefixed with a `$` and under most circumstances you won't care about these. They can be filtered out by passing in a `SubscriptionFilterOptions` when subscribing to the `$all` stream. | ||||||
|
||||||
@[code{exclude-system}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
|
@@ -152,58 +152,79 @@ There are a number of events in EventStoreDB called system events. These are pre | |||||
|
||||||
### Filtering by event type | ||||||
|
||||||
If you only want to subscribe to events of a given type there are two options. You can either use a regular expression or a prefix. | ||||||
If you only want to subscribe to events of a given type, there are two options. You can either use a regular expression or a prefix. | ||||||
|
||||||
#### Filtering by prefix | ||||||
|
||||||
If you want to filter by prefix pass in a `SubscriptionFilterOptions` to the subscription with an `EventTypeFilter.Prefix`. | ||||||
If you want to filter by prefix, pass in a `SubscriptionFilterOptions` to the subscription with an `EventTypeFilter.Prefix`. | ||||||
|
||||||
@[code{event-type-prefix}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
This will only subscribe to events with a type that begin with `customer-`. | ||||||
|
||||||
#### Filtering by regular expression | ||||||
|
||||||
If you want to subscribe to multiple event types then it might be better to provide a regular expression. | ||||||
It might be advantageous to provide a regular expression when you want to subscribe to multiple event types. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
@[code{event-type-regex}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
This will subscribe to any event that begins with `user` or `company`. | ||||||
|
||||||
### Filtering by stream name | ||||||
|
||||||
If you only want to subscribe to a stream with a given name there are two options. You can either use a regular expression or a prefix. | ||||||
If you want to subscribe to a stream only with a given name, you have two options: you can use a regular expression or a prefix. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
#### Filtering by prefix | ||||||
|
||||||
If you want to filter by prefix pass in a `SubscriptionFilterOptions` to the subscription with an `StreamFilter.Prefix`. | ||||||
If you want to filter by prefix, pass in a `SubscriptionFilterOptions` to the subscription with an `StreamFilter.Prefix`. | ||||||
|
||||||
@[code{stream-prefix}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
This will only subscribe to all streams with a name that begin with `user-`. | ||||||
This will only subscribe to all streams with a name that begins with `user-`. | ||||||
|
||||||
#### Filtering by regular expression | ||||||
|
||||||
If you want to subscribe to multiple streams then it might be better to provide a regular expression. | ||||||
If you want to subscribe to multiple streams, providing a regular expression might be better. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
@[code{stream-regex}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
This will subscribe to any stream with a name that begins with `account` or `savings`. | ||||||
|
||||||
## Checkpointing | ||||||
|
||||||
There is one thing to consider with server-side filtering, and that is when events that match your filter are few and far between. In this scenario, you might find yourself in the situation where EventStoreDB has searched through 1 million events, and the last thing you want to happen is for the server to get to event 900k and then have your client crash. It won't have been able to take a checkpoint and upon a restart, you'd have to go back to the beginning and start again. | ||||||
When a catch-up subscription is used to process an `$all` stream containing many events, the last thing you want is for your application to crash midway, forcing you to restart from the beginning. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
||||||
In this case you can make use of an additional delegate that will be triggered every n number of events (32 by default). | ||||||
### What is a checkpoint? | ||||||
|
||||||
To make use of it set up `checkpointReached` on the `SubscriptionFilterOptions` class. | ||||||
A checkpoint is the position of an event in the `$all` stream to which your application has processed. By saving this position to a persistent store (e.g., a database), it allows your catch-up subscription to: | ||||||
- Recover from crashes by reading the checkpoint and resuming from that position | ||||||
- Avoid reprocessing all events from the start | ||||||
|
||||||
To create a checkpoint, store the event's commit or prepare position. | ||||||
|
||||||
::: warning | ||||||
If your database contains events created by the legacy TCP client using the [transaction feature](https://docs.kurrent.io/clients/tcp/dotnet/21.2/appending.html#transactions), you should store both the commit and prepare positions together as your checkpoint. | ||||||
::: | ||||||
|
||||||
### Updating checkpoints at regular intervals | ||||||
The client SDK provides a way to notify your application after processing a configurable number of events. This allows you to periodically save a checkpoint at regular intervals. | ||||||
|
||||||
@[code{checkpoint}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
|
||||||
This will be called every `n` number of events. If you want to be specific about the number of events threshold you can also pass that as a parameter. | ||||||
By default, the checkpoint notification is sent after every 32 non-system events processed from $all. | ||||||
|
||||||
### Configuring the checkpoint interval | ||||||
You can adjust the checkpoint interval to change how often the client is notified. | ||||||
|
||||||
@[code{checkpoint-with-interval}](@grpc:server_side_filtering.py;server-side-filtering.js;server-side-filtering.ts;server_side_filtering/ServerSideFiltering.java;server-side-filtering/Program.cs;serverSideFiltering.go;server_side_filtering.rs) | ||||||
::: warning | ||||||
This number will be called every `n * 32` events. | ||||||
::: | ||||||
|
||||||
By configuring this parameter, you can balance between reducing checkpoint overhead and ensuring quick recovery in case of a failure. | ||||||
|
||||||
::: info | ||||||
The checkpoint interval parameter configures the database to notify the client after `n` * 32 number of events where `n` is defined by the parameter. | ||||||
|
||||||
For example: | ||||||
- If `n` = 1, a checkpoint notification is sent every 32 events. | ||||||
- If `n` = 2, the notification is sent every 64 events. | ||||||
- If `n` = 3, it is sent every 96 events, and so on. | ||||||
::: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.