feat(#41): create and document backup restore vacuum process #626
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
novoj
commented
Jul 9, 2024
novoj
commented
- backup + restore
- new management API for tasks, files, backup and restore
- time machine implementation
Backup & restore done and working. No client implementation and missing integration test.
…tore--vacuum-process
…parallel write transaction load
Implemented and tested
…tore--vacuum-process # Conflicts: # evita_engine/src/main/java/io/evitadb/core/Evita.java # evita_engine/src/main/java/io/evitadb/store/spi/EntityCollectionPersistenceService.java # evita_external_api/evita_external_api_grpc/server/src/main/java/io/evitadb/externalApi/grpc/services/EvitaService.java # evita_external_api/evita_external_api_grpc/shared/src/main/java/io/evitadb/externalApi/grpc/generated/GrpcHealthProblem.java # evita_external_api/evita_external_api_observability/src/main/java/io/evitadb/externalApi/observability/metric/ObservabilityProbesDetector.java # evita_store/evita_store_server/src/main/java/io/evitadb/store/catalog/DefaultCatalogPersistenceService.java # evita_store/evita_store_server/src/main/java/io/evitadb/store/catalog/DefaultEntityCollectionPersistenceService.java # evita_store/evita_store_server/src/main/java/io/evitadb/store/catalog/OffsetIndexStoragePartPersistenceService.java
…tore--vacuum-process
…tore--vacuum-process
…tore--vacuum-process
Error occurred on our test system: ``` 2024-05-30T05:02:40.171480900Z SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@66e3d4d9 2024-05-30T05:02:40.171486100Z io.evitadb.exception.GenericEvitaInternalError: Formula results haven't been computed! 2024-05-30T05:02:40.171489959Z at io.evitadb.utils.Assert.isPremiseValid(Assert.java:89) 2024-05-30T05:02:40.171493839Z at io.evitadb.core.query.algebra.AbstractFormula.getCost(AbstractFormula.java:127) 2024-05-30T05:02:40.171497789Z at java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:231) 2024-05-30T05:02:40.171501369Z at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) 2024-05-30T05:02:40.171517549Z at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) 2024-05-30T05:02:40.171521309Z at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) 2024-05-30T05:02:40.171533709Z at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) 2024-05-30T05:02:40.171537469Z at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) 2024-05-30T05:02:40.171540929Z at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:498) 2024-05-30T05:02:40.171544379Z at java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:456) 2024-05-30T05:02:40.171547899Z at io.evitadb.core.query.algebra.AbstractFormula.getCostInternal(AbstractFormula.java:252) 2024-05-30T05:02:40.171551899Z at io.evitadb.core.query.algebra.base.NotFormula.getCostInternal(NotFormula.java:190) 2024-05-30T05:02:40.171555719Z at io.evitadb.core.query.algebra.AbstractFormula.initializeInternal(AbstractFormula.java:324) 2024-05-30T05:02:40.171559269Z at io.evitadb.core.query.algebra.AbstractFormula.initialize(AbstractFormula.java:87) 2024-05-30T05:02:40.171563109Z at io.evitadb.core.query.algebra.AbstractFormula.initializeInternal(AbstractFormula.java:290) 2024-05-30T05:02:40.171566459Z at io.evitadb.core.query.algebra.AbstractFormula.initialize(AbstractFormula.java:87) 2024-05-30T05:02:40.171570149Z at io.evitadb.core.query.algebra.AbstractFormula.initializeInternal(AbstractFormula.java:290) 2024-05-30T05:02:40.171573929Z at io.evitadb.core.query.algebra.AbstractFormula.initialize(AbstractFormula.java:87) 2024-05-30T05:02:40.171576869Z at io.evitadb.core.query.algebra.AbstractFormula.getCost(AbstractFormula.java:126) ``` This commit also aligns more UserFormula with AndFormula internal logic.
…tore--vacuum-process # Conflicts: # evita_engine/src/main/java/io/evitadb/core/query/algebra/base/NotFormula.java # evita_external_api/evita_external_api_grpc/client/src/main/java/io/evitadb/driver/EvitaClient.java # evita_store/evita_store_server/src/main/java/io/evitadb/store/catalog/DefaultCatalogPersistenceService.java
It seems that when the catalog file exceeds the threshold size (100MB by default) and is compressed into a new file, it's somehow corrupted. The previous file is not deleted, but partially overwritten, and the new file is also corrupted. This was discovered by analyzing file remnants where the original file was much smaller than the 100MB it should have been, and the contents were completely corrupted. It is likely that different tasks are writing to corrupted versions of the file. We need to investigate this issue and write a more complex integration test for this scenario.
…tore--vacuum-process # Conflicts: # evita_functional_tests/src/test/java/io/evitadb/api/EvitaTransactionalFunctionalTest.java # evita_store/evita_store_server/src/main/java/io/evitadb/store/catalog/DefaultCatalogPersistenceService.java
…tore--vacuum-process # Conflicts: # evita_external_api/evita_external_api_grpc/shared/src/main/java/io/evitadb/externalApi/grpc/certificate/ClientCertificateManager.java
This will open-up the path to time-traveling catalog restoration.
…tore--vacuum-process
…exceeded We set strict timeouts for requests made via HTTP endpoints, but these timeouts probably don't end the execution of the requests - they just interrupt the thread, which is not checked in the request evaluation. We need to cover this part with tests and introduce injection points that would check the thread interrupt state and terminate early if an interrupt is detected.
…exceeded Added interruptions to GraphQL library as well.
…estore--vacuum-process' into 41-create-and-document-backup--restore--vacuum-process
…exceeded Added support for asynchronous querying.
…estore--vacuum-process' into 41-create-and-document-backup--restore--vacuum-process
Refactored work with QueryContext dividing it into two - QueryPlanningContext and QueryExecutionContext. The latter is unique per (isolated for) QueryPlan execution. This lead to simplification of the QueryPlanning and avoiding issues with multiple plans execution and result comparison for debug requirements.
We decided to make backup and restore operation asynchronous.
…tore--vacuum-process
…ocument-backup--restore--vacuum-process # Conflicts: # evita_engine/src/main/java/io/evitadb/core/Catalog.java # evita_engine/src/main/java/io/evitadb/store/spi/CatalogPersistenceService.java # evita_engine/src/main/java/io/evitadb/store/spi/CatalogStoragePartPersistenceService.java # evita_functional_tests/src/test/java/io/evitadb/store/catalog/DefaultCatalogPersistenceServiceTest.java
Also removed duplicate max queue configuration for transactions.
Also corrected the backup logic, that takes heavy indexing into an account.
Also corrected the backup logic, that takes heavy indexing into an account.
Automatic re-scheduling and transaction pipeline recreation.
Automatic re-scheduling and transaction pipeline recreation.
Automatic re-scheduling and transaction pipeline recreation.
Automatic re-scheduling and transaction pipeline recreation.
Automatic re-scheduling and transaction pipeline recreation.
Automatic re-scheduling and transaction pipeline recreation.
…tore--vacuum-process
…tore--vacuum-process
Backup, restore, export file manager and scheduler with additional method. VIP.
…tore--vacuum-process # Conflicts: # documentation/user/en/operate/reference/metrics.md # evita_engine/src/main/java/io/evitadb/core/query/ReferencedEntityFetcher.java
…tore--vacuum-process # Conflicts: # documentation/user/en/operate/reference/metrics.md # evita_external_api/evita_external_api_grpc/shared/src/main/java/io/evitadb/externalApi/grpc/generated/GrpcEvitaSessionAPI.java # evita_functional_tests/src/test/java/io/evitadb/server/EvitaServerTest.java
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.