Skip to content

Document expiration & refresh updates, SQS ETL page minor fixes #2019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Document expiration & refresh updates, SQS ETL page minor fixes #2019

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
afb7d96
RDoc-3251 - fix transformation script screenshot in SQS ETL page
reebhub Apr 15, 2025
8bc50dd
Update documents Expiration and Refresh pages with modified objects a…
reebhub Apr 22, 2025
986522f
RDoc-3274 - add a Studio refresh-feature page
reebhub Apr 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 63 additions & 32 deletions ...tion/5.4/Raven.Documentation.Pages/server/extensions/expiration.dotnet.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,73 +3,104 @@

{NOTE: }

* The Expiration feature deletes expired documents, documents whose time has passed.
Documents that are set with a future expiration time will be automatically deleted.

* The Expiration feature can be turned on and off while the database is already live with data.
* Documents can be given a future expiration time in which they'll be automatically deleted.
* The Expiration feature deletes documents set for expiration, when their time has passed.
* You can enable or disable the expiration feature while the database is already live with data.

* In this page:
* [Expiration feature usages](../../server/extensions/expiration#expiration-feature-usages)
* [Configuring the expiration feature](../../server/extensions/expiration#configuring-the-expiration-feature)
* [Configure expiration settings using the client API](../../server/extensions/expiration#configure-expiration-settings-using-the-client-api)
* [Setting the document expiration time](../../server/extensions/expiration#setting-the-document-expiration-time)
* [Eventual consistency considerations](../../server/extensions/expiration#eventual-consistency-considerations)
* [More details](../../server/extensions/expiration#more-details)
{NOTE/}

---

{PANEL: Expiration feature usages}

* Use the Expiration feature when data is needed to be kept for only a temporary time.
Use the Expiration feature when data is needed only for a given time period.
E.g., for -

* Shopping cart data that is kept only for a certain time period
* Email links that need to be expired after a few hours
* A web application login session details
* Cache data from an SQL server

* Examples:
* Shopping cart data that is kept for only a specific time
* Email links that need to be expired after a few hours
* Storing a web application login session details
* When using RavenDB to hold cache data from a SQL server.
{PANEL/}

{PANEL: Configuring the expiration feature}

* By default, the expiration feature is turned off.
Documents expiration settings can be changed via Studio or the API.
It is possible to:

* The delete frequency is configurable, the default value is 60 secs.
* Enable or Disable the deletion of expired documents.
Default value: **Disable** the deletion of expired documents.
* Determine how often RavenDB would look for expired documents and delete them.
Default value: **60 seconds**
* Set the maximal number of documents that RavenDB is allowed to delete per interval.
Default value: **All expired documents**

* The Expiration feature can be turned on and off using **Studio**, see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
---

* The Expiration feature can also be configured using the **Client**:
{INFO: }
[Learn how to configure expiration settings via Studio](../../studio/database/settings/document-expiration)
{INFO/}

{CODE configuration@Server\Expiration\Expiration.cs /}
{PANEL/}
---

{PANEL: Setting the document expiration time}
### Configure expiration settings using the client API

* To set the document expiration time just add the `@expires` property to the document `@metadata` and set it to contain the appropriate expiration time.
Modify the expiration settings using the client API by setting an `ExpirationConfiguration`
object and sending it to RavenDB using a `ConfigureExpirationOperation` operation.

* Once the Expiration feature is enabled, the document will automatically be deleted at the specified time.
#### Example:

* **Note**: The date must be in **UTC** format, not local time.
{CODE configuration@Server\Expiration\Expiration.cs /}

#### `ExpirationConfiguration`

{CODE expirationConfiguration@Server\Expiration\Expiration.cs /}

* The document expiration time can be set using the following code from the client:
| Parameter | Type | Description |
| - | - | - |
| **Disabled** | `bool` | If `true`, deleting expired documents is disabled for the entire database.<BR>Default: `true` |
| **DeleteFrequencyInSec** | `long?` | Determines how often (in seconds) the expiration feature looks for expired documents and deletes them.<BR>Default: `60` |
| **MaxItemsToProcess** | `long?` | Determines the maximal number of documents the feature is allowed to delete in one run. |

{CODE expiration1@Server\Expiration\Expiration.cs /}
{PANEL/}

{PANEL: Eventual consistency considerations}
{PANEL: Setting the document expiration time}

* Once documents are expired, it can take up to the delete frequency interval (60 seconds by default) until these expired documents are actually deleted.
* To set a document expiration time, add the document's `@metadata` an
`@expires` property with the designated expiration time as a value.
Set the time in **UTC** format, not local time. E.g. -
**"@expires": "2025-04-22T08:00:00.0000000Z"**
{WARNING: }
Metadata properties starting with `@` are for internal RavenDB usage only.
Do _not_ use the metadata `@expires` property for any other purpose than
scheduling a document's expiration time for the built-in expiration feature.
{WARNING/}
* If and when the expiration feature is enabled, it will process all documents
carrying the `@expires` flag and automatically delete each document
[by its expiration time](../../server/extensions/expiration#eventual-consistency-considerations).
* To set the document expiration time from the client, use the following code:
{CODE expiration1@Server\Expiration\Expiration.cs /}

* Expired documents are _not_ filtered on load/query/indexing time, so be aware that an expired document might still be there after it has expired up to the 'delete frequency interval' timeframe.
{PANEL/}

{PANEL: More details}
{PANEL: Eventual consistency considerations}

* Internally, each document that has the `@expires` property in the metadata is tracked by the RavenDB server
even if the expiration feature is turned off.
This way, once the expiration feature is turned on we can easily delete all the expired documents.
* Internally, RavenDB tracks all documents carrying the `@expires` flag even if the
expiration feature is disabled. This way, once the expiration feature is enabled expired
documents can be processed without delay.
* Once a document expires, it may take up to the _delete frequency interval_ (60 seconds by default)
until is it actually deleted.
* Deletion may be further delayed if `MaxItemsToProcess` is set, limiting the number
of documents that RavenDB is allowed to delete each time the expiration feature is invoked.
* Expired documents are _not_ filtered out during `load`, `query`, or indexing, so be aware that
as long as an expired document hasn't been actually deleted it may still be included in the results.

* **Note**: Metadata properties starting with `@` are internal for RavenDB usage.
You should _not_ use the `@expires` property in the metadata for any other purpose other than the built-in expiration feature.
{PANEL/}

## Related Articles
Expand Down
78 changes: 38 additions & 40 deletions ...entation/5.4/Raven.Documentation.Pages/server/extensions/expiration.js.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,75 +3,73 @@

{NOTE: }

* The Expiration feature deletes expired documents, documents whose time has passed.
Documents that are set with a future expiration time will be automatically deleted.

* The Expiration feature can be turned on and off while the database is already live with data.
* Documents can be given a future expiration time in which they'll be automatically deleted.
* The Expiration feature deletes documents set for expiration, when their time has passed.
* You can enable or disable the expiration feature while the database is already live with data.

* In this page:
* [Expiration feature usages](../../server/extensions/expiration#expiration-feature-usages)
* [Configuring the expiration feature](../../server/extensions/expiration#configuring-the-expiration-feature)
* [Setting the document expiration time](../../server/extensions/expiration#setting-the-document-expiration-time)
* [Eventual consistency considerations](../../server/extensions/expiration#eventual-consistency-considerations)
* [More details](../../server/extensions/expiration#more-details)
{NOTE/}

---

{PANEL: Expiration feature usages}

* Use the Expiration feature when data is needed to be kept for only a temporary time.

* Examples:
* Shopping cart data that is kept for only a specific time
* Email links that need to be expired after a few hours
* Storing a web application login session details
* When using RavenDB to hold cache data from a SQL server.
{PANEL/}
Use the Expiration feature when data is needed only for a given time period.

{PANEL: Configuring the expiration feature}
Examples:

* By default, the expiration feature is turned off.
* Shopping cart data that is kept only for a certain time period
* Email links that need to be expired after a few hours
* A web application login session details
* Cache data from an SQL server

* The delete frequency is configurable, the default value is 60 secs.
{PANEL/}

* The Expiration feature can be turned on and off using **Studio**, see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
{PANEL: Configuring the expiration feature}

* By default, the expiration feature is turned **off**.
* The deletion frequency is configurable.
The default frequency is 60 seconds.
* The Expiration feature can be turned **on** or **off** using **Studio**,
see [Setting Document Expiration in Studio](../../studio/database/settings/document-expiration).
* The Expiration feature can also be configured using the **Client**:

{CODE:nodejs expiration_1@Server\Expiration\expiration.js /}
{CODE:nodejs expiration_1@Server\Expiration\expiration.js /}

{PANEL/}

{PANEL: Setting the document expiration time}

* To set the document expiration time just add the `@expires` property to the document `@metadata` and set it to contain the appropriate expiration time.

* Once the Expiration feature is enabled, the document will automatically be deleted at the specified time.

* **Note**: The date must be in **UTC** format, not local time.

* The document expiration time can be set using the following code from the client:

{CODE:nodejs expiration_2@Server\Expiration\expiration.js /}
* To set document expiration time add an `@expires` property to the document
`@metadata` and set the property's value to the designated expiration time.
{NOTE: }
The date must be in **UTC** format, not local time.
{NOTE/}
{WARNING: }
Metadata properties starting with `@` are internal for RavenDB usage.
Do _not_ use metadata `@expires` property for any other usage but that of the built-in expiration feature.
{WARNING/}
* Once the Expiration feature is enabled, the document will be automatically deleted at the specified time.
{INFO: }
Internally, each document that has the `@expires` property in the metadata is tracked
by the RavenDB server even if the expiration feature is turned off.
This way, once the expiration feature is turned on RavenDB can delete expired documents right away.
{INFO/}
* To set the document expiration time from the client, use the following code:
{CODE:nodejs expiration_2@Server\Expiration\expiration.js /}

{PANEL/}

{PANEL: Eventual consistency considerations}

* Once documents are expired, it can take up to the delete frequency interval (60 seconds by default) until these expired documents are actually deleted.

* Expired documents are _not_ filtered on load/query/indexing time, so be aware that an expired document might still be there after it has expired up to the 'delete frequency interval' timeframe.
{PANEL/}

{PANEL: More details}

* Internally, each document that has the `@expires` property in the metadata is tracked by the RavenDB server
even if the expiration feature is turned off.
This way, once the expiration feature is turned on we can easily delete all the expired documents.
* Once documents expire, it may take up to the _delete frequency interval_ (60 seconds by default)
until they are actually deleted.
* Expired documents are _not_ filtered out during `load`, `query`, or indexing, so be aware that an expired
document may still be included in the results after its expiration, for the period set by _delete frequency interval_.

* **Note**: Metadata properties starting with `@` are internal for RavenDB usage.
You should _not_ use the `@expires` property in the metadata for any other purpose other than the built-in expiration feature.
{PANEL/}

## Related Articles
Expand Down
Binary file modified ...tation/5.4/Raven.Documentation.Pages/server/extensions/images/StudioRefresh.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
105 changes: 61 additions & 44 deletions ...ntation/5.4/Raven.Documentation.Pages/server/extensions/refresh.dotnet.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,76 +10,93 @@

* In this page:
* [Overview](../../server/extensions/refresh#overview)
* [Syntax](../../server/extensions/refresh#syntax)
* [Examples](../../server/extensions/refresh#examples)
* [Examples](../../server/extensions/refresh#examples)
* [Syntax](../../server/extensions/refresh#syntax)
* [Configure from Studio](../../server/extensions/refresh#configure-from-studio)


{NOTE/}

---

{PANEL: Overview}

To refresh a document, add a `@refresh` flag to a document's metadata with a value of
type `DateTime` (this must be in UTC format, i.e. `DateTime.UtcNow`). This specifies
when the document should be refreshed.

This will cause the document to refresh **_only once_**! The refresh operation removes
the `@refresh` flag.
* To set a document refresh time, add the document's `@metadata` a
`@refresh` property with the designated refresh time as a value.
Set the time in **UTC** format, not local time, e.g. -
**"@refresh": "2025-04-22T08:00:00.0000000Z"**
{WARNING: }
Metadata properties starting with `@` are for internal RavenDB usage only.
Do _not_ use the metadata `@refresh` property for any other purpose than
scheduling a document's refresh time for the built-in refresh feature.
{WARNING/}

* This will cause the document to refresh **only once**.
When the refresh operation takes place, it will also remove the `@refresh` property from the document.
{INFO: The exact refresh time is determined by:}

1. The refresh time value set for the `@refresh` property.
2. The way you set the [Refresh Configuration](../../server/extensions/refresh#syntax),
including -
- The interval by which the server refreshes documents (set by default to 60 seconds).
- The way you set **maximal items to process**, potentially limiting the number
of documents that RavenDB is allowed to delete each time the refresh feature is invoked.

{INFO/}

* Refreshing a document causes its [change vector](../../server/clustering/replication/change-vector)
to increment the same way it would after any other kind of update to the document.
This triggers any features that react to document updating, including but not limited to:
- Re-indexing of the document by indexes that cover it
- [Replication](../../server/ongoing-tasks/external-replication),
[Subscriptions](../../client-api/data-subscriptions/what-are-data-subscriptions),
and [ETL](../../server/ongoing-tasks/etl/basics) triggering
- Creation of a [document revision](../../document-extensions/revisions/overview)

The exact time that the document refreshes is not determined by the value of `@refresh`
- rather, the server refreshes documents at regular intervals determined by the Refresh
Configuration. The default interval is 60 seconds.
{PANEL/}

Refreshing a document causes its [change vector](../../server/clustering/replication/change-vector)
to increment the same way it would after any other kind of update to the document.
This triggers any features that react to documents updating, including but not limited
to:
{PANEL: Examples}

* The document is re-indexed by any indexes that cover it.
* [Replication](../../server/ongoing-tasks/external-replication),
[Subscriptions](../../client-api/data-subscriptions/what-are-data-subscriptions),
and [ETL](../../server/ongoing-tasks/etl/basics) are triggered.
* A [revision](../../document-extensions/revisions/overview) of the document is created.
#### Example I

{PANEL/}
How to set refresh configuration for a database:

{PANEL: Syntax}
{CODE refresh_0@Server/Extensions/DocumentRefresh.cs /}

To activate and/or configure document refreshing, the server needs to be sent a
`RefreshConfiguration` object using a `ConfigureRefreshOperation` operation.
This activates document refreshing and sets the interval at 5 minutes.
<br/>
#### RefreshConfiguration Object
#### Example II

{CODE refresh_config@Server/Extensions/DocumentRefresh.cs /}
How to set a document to refresh 1 hour from now:

| Parameter | Type | Description |
| - | - | - |
| **Disabled** | `bool` | If set to true, document refreshing is disabled for the entire database. Default: `true` |
| **RefreshFrequencyInSec** | `long` | Determines how often the server checks for documents that need to be refreshed. Default: `60` |
<br/>
#### Studio
{CODE refresh_1@Server/Extensions/DocumentRefresh.cs /}

Alternatively, document refreshing can also be configured in the studio, under **Settings > Document Refresh**.
{PANEL/}

![NoSQL DB Server - Document Refresh](images/StudioRefresh.png "Document Refresh Settings")
{PANEL: Syntax}

{PANEL/}
To activate and/or configure document refreshing, send the server a
`RefreshConfiguration` object using the `ConfigureRefreshOperation` operation.

{PANEL: Examples}
---

#### Example I
#### `RefreshConfiguration`

How to set refresh configuration for a database:
{CODE refresh_config@Server/Extensions/DocumentRefresh.cs /}

{CODE refresh_0@Server/Extensions/DocumentRefresh.cs /}
| Parameter | Type | Description |
| - | - | - |
| **Disabled** | `bool` | If `true`, refreshing documents is disabled for the entire database.<BR>Default: `true` |
| **RefreshFrequencyInSec** | `long?` | Determines how often (in seconds) the server processes documents that need to be refreshed.<BR>Default: `60` |
| **MaxItemsToProcess** | `long?` | Determines the maximal number of documents the feature is allowed to refresh in one run. |

This activates document refreshing and sets the interval at 5 minutes.
<br/>
#### Example II
{PANEL/}

How to set a document to refresh 1 hour from now:
{PANEL: Configure from Studio}

{CODE refresh_1@Server/Extensions/DocumentRefresh.cs /}
Alternatively, document refreshing can also be configured via Studio, under **Settings > Document Refresh**.

![NoSQL DB Server - Document Refresh](images/StudioRefresh.png "Document Refresh Settings")

{PANEL/}

Expand Down
Loading
Loading