Skip to content

Commit

Permalink
Merge pull request #626 from FgForrest/41-create-and-document-backup-…
Browse files Browse the repository at this point in the history 
…-restore--vacuum-process

feat(#41): create and document backup  restore  vacuum process
  • Loading branch information
@novoj
novoj authored Jul 9, 2024
2 parents 4c8fbfb + ca83425 commit 96051e0
Show file tree
Hide file tree
Showing 409 changed files with 36,800 additions and 7,059 deletions.
3 changes: 2 additions & 1 deletion .idea/vcs.xml
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

15 changes: 8 additions & 7 deletions documentation/user/en/operate/configure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,6 @@ transaction: # [see Transaction configurati
transactionMemoryRegionCount: 256
walFileSizeBytes: 16MB
walFileCountKept: 8
maxQueueSize: 1K
flushFrequencyInMillis: 1s

cache: # [see Cache configuration](#cache-configuration)
Expand Down Expand Up @@ -441,6 +440,14 @@ This section contains configuration options for the storage layer of the databas
<p>Minimal file size threshold for compaction. If the file size is lower, the file will not be compacted even
if the share of active records is lower than the minimal share.</p>
</dd>
<dt>timeTravelEnabled</dt>
<dd>
<p>**Default:** `true`</p>
<p>When set to true, the data files are not removed immediately after compacting, but are kept on disk as long
as there is history available in the WAL log. This allows a snapshot of the database to be taken at any point
in the history covered by the WAL log. From the snapshot, the database can be restored to the exact point in
time with all the data available at that time.</p>
</dd>
</dl>

## Transaction configuration
Expand Down Expand Up @@ -481,12 +488,6 @@ This section contains configuration options for the storage layer of the databas
<p>Number of WAL files to keep. Increase this number in combination with `walFileSizeBytes` if you want to
keep longer history of changes.</p>
</dd>
<dt>maxQueueSize</dt>
<dd>
<p>**Default:** `1K`</p>
<p>Size of the catalog queue for parallel transaction. If there are more transaction than the number of free
threads in the pool, the transaction are queued. If the queue is full, the transaction is rejected.</p>
</dd>
<dt>flushFrequencyInMillis</dt>
<dd>
<p>**Default:** `1s`</p>
Expand Down
4 changes: 4 additions & 0 deletions documentation/user/en/operate/reference/jfr-events.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,8 +96,12 @@
<dl>
<dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskFinishedEvent.java</SourceClass> Background task finished</dt>
<dd>Event that is fired when a background task is finished.</dd>
<dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskRejectedEvent.java</SourceClass> Background task rejected</dt>
<dd>Event that is fired when a background task is rejected due to full queues.</dd>
<dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskStartedEvent.java</SourceClass> Background task started</dt>
<dd>Event that is fired when a background task is started.</dd>
<dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/BackgroundTaskTimedOutEvent.java</SourceClass> Background task timed out</dt>
<dd>Event that is fired when a background task was timed out and was canceled.</dd>
<dt><SourceClass>evita_engine/src/main/java/io/evitadb/core/metric/event/system/EvitaStartedEvent.java</SourceClass> Evita started</dt>
<dd>Event that is fired when evitaDB instance is started.</dd>
</dl>
Expand Down
114 changes: 38 additions & 76 deletions documentation/user/en/operate/reference/metrics.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,98 +3,48 @@
<UsedTerms>
<h4>Labels used in metrics</h4>
<dl>
<dt>taskName</dt>
<dd><strong>N/A</strong>: N/A</dd>
<dt>buildType</dt>
<dd><strong>Build type</strong>: N/A</dd>
<dt>catalogName</dt>
<dd><strong>Catalog</strong>: N/A</dd>
<dt>instanceType</dt>
<dd><strong>Instance type</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>recordType</dt>
<dd><strong>Record type</strong>: N/A</dd>
<dt>catalogName</dt>
<dd><strong>Catalog</strong>: N/A</dd>
<dt>entityType</dt>
<dd><strong>Collection</strong>: N/A</dd>
<dt>entityType</dt>
<dd><strong>Entity type</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>httpMethod</dt>
<dd><strong>HTTP method</strong>: N/A</dd>
<dt>initiator</dt>
<dd><strong>Initiator of the call (client or server)</strong>: N/A</dd>
<dt>instanceType</dt>
<dd><strong>Instance type</strong>: N/A</dd>
<dt>operationId</dt>
<dd><strong>Operation ID</strong>: N/A</dd>
<dt>operationType</dt>
<dd><strong>Operation type</strong>: N/A</dd>
<dt>responseStatus</dt>
<dd><strong>Response status</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>stage</dt>
<dd><strong>Transaction stage</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>resolution</dt>
<dd><strong>Transaction resolution</strong>: N/A</dd>
<dt>catalogName</dt>
<dd><strong>Catalog</strong>: N/A</dd>
<dt>instanceType</dt>
<dd><strong>Instance type</strong>: N/A</dd>
<dt>operationId</dt>
<dd><strong>Operation ID</strong>: N/A</dd>
<dt>operationName</dt>
<dd><strong>GraphQL operation</strong>: N/A</dd>
<dt>operationType</dt>
<dd><strong>Operation type</strong>: N/A</dd>
<dt>responseStatus</dt>
<dd><strong>Response status</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>initiator</dt>
<dd><strong>Initiator of the call (client or server)</strong>: N/A</dd>
<dt>prefetched</dt>
<dd><strong>Prefetched vs. non-prefetched query</strong>: N/A</dd>
<dt>procedureName</dt>
<dd><strong>Name of the procedure that was called</strong>: N/A</dd>
<dt>recordType</dt>
<dd><strong>Record type</strong>: N/A</dd>
<dt>resolution</dt>
<dd><strong>Transaction resolution</strong>: N/A</dd>
<dt>responseState</dt>
<dd><strong>State of the response (OK, ERROR, CANCELED)</strong>: N/A</dd>
<dt>responseStatus</dt>
<dd><strong>Response status</strong>: N/A</dd>
<dt>serviceName</dt>
<dd><strong>Name of the service that was called</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>resolution</dt>
<dd><strong>Transaction resolution</strong>: N/A</dd>
<dt>stage</dt>
<dd><strong>Transaction stage</strong>: N/A</dd>
<dt>taskName</dt>
<dd><strong>N/A</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>entityType</dt>
<dd><strong>Entity type</strong>: N/A</dd>
<dt>buildType</dt>
<dd><strong>Build type</strong>: N/A</dd>
<dt>catalogName</dt>
<dd><strong>Catalog</strong>: N/A</dd>
<dt>instanceType</dt>
<dd><strong>Instance type</strong>: N/A</dd>
<dt>entityType</dt>
<dd><strong>Entity type</strong>: N/A</dd>
<dt>fileType</dt>
<dd><strong>File type</strong>: N/A</dd>
<dt>name</dt>
<dd><strong>Logical file name</strong>: N/A</dd>
<dt>entityType</dt>
<dd><strong>Entity type</strong>: N/A</dd>
<dt>prefetched</dt>
<dd><strong>Prefetched vs. non-prefetched query</strong>: N/A</dd>
</dl>
</UsedTerms>

Expand Down Expand Up @@ -312,8 +262,12 @@
<dl>
<dt><code>io_evitadb_system_background_task_finished_total</code> (COUNTER)</dt>
<dd>Background tasks finished<br/><br/><strong>Labels:</strong> <Term>taskName</Term><br/></dd>
<dt><code>io_evitadb_system_background_task_rejected_total</code> (COUNTER)</dt>
<dd>Background tasks rejected<br/><br/><strong>Labels:</strong> <Term>taskName</Term><br/></dd>
<dt><code>io_evitadb_system_background_task_started_total</code> (COUNTER)</dt>
<dd>Background tasks started<br/><br/><strong>Labels:</strong> <Term>taskName</Term><br/></dd>
<dt><code>io_evitadb_system_background_task_timed_out_timed_out_tasks</code> (COUNTER)</dt>
<dd><strong>N/A</strong>: N/A<br/><br/><strong>Labels:</strong> <Term>taskName</Term><br/></dd>
<dt><code>io_evitadb_system_evita_started_cache_anteroom_record_limit</code> (GAUGE)</dt>
<dd><strong>Maximal number of records in cache anteroom</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_cache_reevaluation_seconds</code> (GAUGE)</dt>
Expand All @@ -324,24 +278,32 @@
<dd><strong>Minimal file size threshold to start compaction in Bytes</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_compaction_minimal_active_record_share_percent</code> (GAUGE)</dt>
<dd><strong>Minimal share of active records in the file to start compaction in %</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_max_threads</code> (GAUGE)</dt>
<dd><strong>Maximal number of background threads</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_max_threads_queue_size</code> (GAUGE)</dt>
<dd><strong>Maximal queue size for background threads</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_query_timeout_seconds</code> (GAUGE)</dt>
<dd><strong>Read only request timeout in seconds</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_read_only_handles_limit</code> (GAUGE)</dt>
<dd><strong>Maximal count of opened read-only handles</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_request_max_threads</code> (GAUGE)</dt>
<dd><strong>Maximal number of threads read only request handling</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_request_max_threads_queue_size</code> (GAUGE)</dt>
<dd><strong>Maximal queue size for read only request handling</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_service_max_threads</code> (GAUGE)</dt>
<dd><strong>Maximal number of threads for service tasks</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_service_max_threads_queue_size</code> (GAUGE)</dt>
<dd><strong>Maximal queue size for service tasks</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_session_max_inactive_age_seconds</code> (GAUGE)</dt>
<dd><strong>Maximal session inactivity age in seconds</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_short_tasks_timeout_seconds</code> (GAUGE)</dt>
<dd><strong>Short running tasks timeout in seconds</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_total</code> (COUNTER)</dt>
<dd>Evita started total</dd>
<dt><code>io_evitadb_system_evita_started_transaction_max_queue_size</code> (GAUGE)</dt>
<dd><strong>Maximal count of commited transactions in queue</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_transaction_max_threads</code> (GAUGE)</dt>
<dd><strong>Maximal number of threads for read/write requests</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_transaction_max_threads_queue_size</code> (GAUGE)</dt>
<dd><strong>Maximal queue size for read/write requests</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_transaction_memory_buffer_limit_size_bytes</code> (GAUGE)</dt>
<dd><strong>Size of off-heap memory buffer for transactions in Bytes</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_transaction_memory_regions</code> (GAUGE)</dt>
<dd><strong>Number of off-heap memory regions for transactions</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_transaction_timeout_seconds</code> (GAUGE)</dt>
<dd><strong>Read/write request timeout in seconds</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_wal_max_file_count_kept</code> (GAUGE)</dt>
<dd><strong>Maximal write-ahead log file count to keep</strong>: N/A</dd>
<dt><code>io_evitadb_system_evita_started_wal_max_file_size_bytes</code> (GAUGE)</dt>
Expand Down
19 changes: 19 additions & 0 deletions evita_api/src/main/java/io/evitadb/api/CatalogContract.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@
import io.evitadb.api.exception.EntityTypeAlreadyPresentInCatalogSchemaException;
import io.evitadb.api.exception.InvalidMutationException;
import io.evitadb.api.exception.SchemaAlteringException;
import io.evitadb.api.exception.TemporalDataNotAvailableException;
import io.evitadb.api.file.FileForFetch;
import io.evitadb.api.requestResponse.EvitaRequest;
import io.evitadb.api.requestResponse.EvitaResponse;
import io.evitadb.api.requestResponse.mutation.Mutation;
Expand All @@ -40,10 +42,13 @@
import io.evitadb.api.requestResponse.system.CatalogVersionDescriptor;
import io.evitadb.api.requestResponse.system.TimeFlow;
import io.evitadb.api.requestResponse.transaction.TransactionMutation;
import io.evitadb.api.task.ServerTask;
import io.evitadb.dataType.PaginatedList;

import javax.annotation.Nonnull;
import javax.annotation.Nullable;
import java.io.Serializable;
import java.time.OffsetDateTime;
import java.util.Map;
import java.util.Optional;
import java.util.Set;
Expand Down Expand Up @@ -310,11 +315,25 @@ boolean renameCollectionOfEntity(@Nonnull String entityType, @Nonnull String new
@Nonnull
Stream<Mutation> getReversedCommittedMutationStream(long catalogVersion);

/**
* Creates a backup of the specified catalog and returns an InputStream to read the binary data of the zip file.
*
* @param pastMoment leave null for creating backup for actual dataset, or specify past moment to create backup for
* the dataset as it was at that moment
* @param includingWAL if true, the backup will include the Write-Ahead Log (WAL) file and when the catalog is
* restored, it'll replay the WAL contents locally to bring the catalog to the current state
* @return jobId of the backup process
* @throws TemporalDataNotAvailableException when the past data is not available
*/
@Nonnull
ServerTask<?, FileForFetch> backup(@Nullable OffsetDateTime pastMoment, boolean includingWAL) throws TemporalDataNotAvailableException;

/**
* Terminates catalog instance and frees all claimed resources. Prepares catalog instance to be garbage collected.
*
* This method is idempotent and may be called multiple times. Only first call is really processed and others are
* ignored.
*/
void terminate();

}
Loading

0 comments on commit 96051e0

Please sign in to comment.