Retention for incremental backups #6029

rsmontero · 2022-12-02T13:24:05Z

Description
Backup interface provides retention (KEEP_LAST) to only keep the last N backups. This feature is not implemented for incremental chains.

This feature needs to implement a "rebase" or "commit" action in the backup drivers that takes an increment (or a set of increments) and commit the changes to the previous one. This way we can opt to rebase on the FULL backup or an arbitrary increment for performance reasons.

Use case
Control backup datastore usage without the need of doing a reset of the incremental chain

Interface Changes
None (or add a new parameter to KEEP_LAST_INCREMENTS, to be able to keep some separated FULL backups). Driver new actions required.

Additional Context
NONE

Progress Status

Code committed
Testing - QA
Documentation (Release notes - resolved issues, compatibility, known issues)

The text was updated successfully, but these errors were encountered:

nachowork90 · 2023-01-05T15:52:32Z

This idea is something as synthetic_full, merging the incr#1 in the previous full#0.

The active_full already implemented activating the reset flag.

Will be good if the reset backup chain flag were a separated function, and not inside the same backup execution.

* Introduce support to follow KEEP_LAST for incremental backups. - New increment_flatten action added for backup datastores. - increment_flatten will consolidate KEEP_LAST increments into the current first increment in the chain. - increment_flatten MUST return the new chain (inc1:source1,...) and size of the new first increment (FULL) in the chain. * Downloader logic for restore has been extracted from downloader.sh to reuse the increment flatten logic. A new command restic_downloader.rb process restic:// pseudo-urls. * Restore process uses two new attributes to customize the restore process: - NAME to be used as base name for images and VM Template - INCREMENT_ID to restore disks from a given increment (not always the last one) * Common logic has been added to BackupImage class (backup.rb) * Includes the following fixes: - Fix when increment includes blocks larger than max qemu-io request size - Fix IMAGE counter for quotas on backup images - Fix rsync restore NO_IP / NO_NIC attributes TODO: * Mimic increment_flatten logic and restore images on the backup server * Suntone restore options co-authored-by:Michal Opala <mopala@opennebula.io>

Some backup drivers may not allow delete (forget) operations while an active backup is being performed. This commits fails the one.image.delete API call in that case. A new configuration attribute to describe the related driver behavior has been added CONCURRENT_FORGET for DS_MAD_CONF. (cherry picked from commit 940c1d7)

FrederickBor · 2023-02-15T11:20:36Z

Sunstone:

Code:

(cherry picked from commit 3ba6041)

This prevents a race condition on when flatten and backup actions are triggered simultaneously. (cherry picked from commit e6c7c51)

(cherry picked from commit 694ff37)

Extending the BACKUP state may lead to unneed wait times to perform VM management operations Fixes reset of backup chains for poweroff state (cherry picked from commit 377db96)

The result of a flatten operation can be optionally sparsify to free space. This needs 'virt-sparsify' tool from libguestfs package. Note: if not present the sparsify step will be skipped. (cherry picked from commit de10e61)

* Fix rsync datastore monitoring * Add increment flatten for rsync backups * Fix/Refactor rsync restore action handler * Fix/Refactor rsync rm action handler * Add dedicated downloader code for rsync (cherry picked from commit f4eee21)

(cherry picked from commit 0528055)

When doing incremental backups it MUST reuse the current datastore ID. This commits forces this setting ignoring the API parameter in this case. (cherry picked from commit 01c490a)

…#2520) (cherry picked from commit 3937182)

* Add sparsify for rsync + merge_chain refactor * Linting (cherry picked from commit b369372)

* F #6029: Fix rsync downloader full backup handling - "short-circuit" full backups - disable sparsify for rsync downlaoder * F #6029: Fix restic downloader full backup handling - "short-circuit" full backups - disable sparsify for restic downlaoder (cherry picked from commit 53caf68)

Possible fix for "Message received: BACKUP SUCCESS 96 -" (?) (cherry picked from commit e6652bd)

* Introduce support to follow KEEP_LAST for incremental backups. - New increment_flatten action added for backup datastores. - increment_flatten will consolidate KEEP_LAST increments into the current first increment in the chain. - increment_flatten MUST return the new chain (inc1:source1,...) and size of the new first increment (FULL) in the chain. * Downloader logic for restore has been extracted from downloader.sh to reuse the increment flatten logic. A new command restic_downloader.rb process restic:// pseudo-urls. * Restore process uses two new attributes to customize the restore process: - NAME to be used as base name for images and VM Template - INCREMENT_ID to restore disks from a given increment (not always the last one) * Common logic has been added to BackupImage class (backup.rb) * Includes the following fixes: - Fix when increment includes blocks larger than max qemu-io request size - Fix IMAGE counter for quotas on backup images - Fix rsync restore NO_IP / NO_NIC attributes TODO: * Mimic increment_flatten logic and restore images on the backup server * Suntone restore options co-authored-by:Michal Opala <mopala@opennebula.io>

Some backup drivers may not allow delete (forget) operations while an active backup is being performed. This commits fails the one.image.delete API call in that case. A new configuration attribute to describe the related driver behavior has been added CONCURRENT_FORGET for DS_MAD_CONF.

This prevents a race condition on when flatten and backup actions are triggered simultaneously.

Extending the BACKUP state may lead to unneed wait times to perform VM management operations Fixes reset of backup chains for poweroff state

When doing incremental backups it MUST reuse the current datastore ID. This commits forces this setting ignoring the API parameter in this case.

The result of a flatten operation can be optionally sparsify to free space. This needs 'virt-sparsify' tool from libguestfs package. Note: if not present the sparsify step will be skipped.

* Fix rsync datastore monitoring * Add increment flatten for rsync backups * Fix/Refactor rsync restore action handler * Fix/Refactor rsync rm action handler * Add dedicated downloader code for rsync

…#2520)

* Add sparsify for rsync + merge_chain refactor * Linting

* F #6029: Fix rsync downloader full backup handling - "short-circuit" full backups - disable sparsify for rsync downlaoder * F #6029: Fix restic downloader full backup handling - "short-circuit" full backups - disable sparsify for restic downlaoder

Possible fix for "Message received: BACKUP SUCCESS 96 -" (?)

rsmontero added Category: Core & System Category: Drivers - Storage Type: Feature Sponsored Status: Abandoned Priority: High labels Dec 2, 2022

rsmontero added this to the Release 6.6.1 milestone Dec 2, 2022

rsmontero assigned sk4zuzu Dec 14, 2022

stale bot removed the Status: Abandoned label Dec 14, 2022

tinova assigned FrederickBor and unassigned FrederickBor Jan 30, 2023

rsmontero added a commit to OpenNebula/docs that referenced this issue Feb 15, 2023

F OpenNebula/one#6029: Prune, Incremental restore and retention

3ba6041

rsmontero added a commit to OpenNebula/docs that referenced this issue Feb 15, 2023

F OpenNebula/one#6029,#6074,#6062: Add to resolved issues.

37352cb

rsmontero added a commit to OpenNebula/docs that referenced this issue Feb 15, 2023

F OpenNebula/one#6029: Upgrade notes

b08eb0c

rsmontero added a commit to OpenNebula/docs that referenced this issue Feb 15, 2023

F OpenNebula/one#6029: Prune, Incremental restore and retention

407539c

(cherry picked from commit 3ba6041)

rsmontero added a commit to OpenNebula/docs that referenced this issue Feb 15, 2023

F OpenNebula/one#6029,#6074,#6062: Add to what's new (backport)

48e6a7f

tinova mentioned this issue Feb 21, 2023

Automatically perform prune operations on Restic #6062

Closed

3 tasks

rsmontero closed this as completed Feb 21, 2023

rsmontero added a commit that referenced this issue Feb 21, 2023

F #6029: Extend BACKUP state to include increment_flatten action

db3b920

This prevents a race condition on when flatten and backup actions are triggered simultaneously. (cherry picked from commit e6c7c51)

rsmontero pushed a commit that referenced this issue Feb 21, 2023

F #6029: Create Restic repo per VM (#2495)

063789c

(cherry picked from commit 694ff37)

rsmontero pushed a commit that referenced this issue Feb 28, 2023

F #6029: Fix xml handling in rsync backup actions (#2518)

a97c383

(cherry picked from commit 0528055)

rsmontero pushed a commit that referenced this issue Feb 28, 2023

F #6029: Make restic downloader merge increments on the backup server (…

0763ebd

…#2520) (cherry picked from commit 3937182)

rsmontero pushed a commit that referenced this issue Feb 28, 2023

F #6029: Add sparsify for rsync + merge_chain refactor (#2521)

f511d4d

* Add sparsify for rsync + merge_chain refactor * Linting (cherry picked from commit b369372)

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Normalize exit codes for rsync backup actions (#2535)

d0b81ec

Possible fix for "Message received: BACKUP SUCCESS 96 -" (?) (cherry picked from commit e6652bd)

rsmontero added a commit that referenced this issue Mar 1, 2023

F #6029: Extend BACKUP state to include increment_flatten action

e6c7c51

This prevents a race condition on when flatten and backup actions are triggered simultaneously.

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Create Restic repo per VM (#2495)

694ff37

rsmontero added a commit that referenced this issue Mar 1, 2023

F #6029: Signal increment flatten with an attribute

377db96

Extending the BACKUP state may lead to unneed wait times to perform VM management operations Fixes reset of backup chains for poweroff state

rsmontero added a commit that referenced this issue Mar 1, 2023

F #6029: Force to re-use current datastore ID

01c490a

When doing incremental backups it MUST reuse the current datastore ID. This commits forces this setting ignoring the API parameter in this case.

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Fix xml handling in rsync backup actions (#2518)

0528055

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Make restic downloader merge increments on the backup server (…

3937182

…#2520)

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Add sparsify for rsync + merge_chain refactor (#2521)

b369372

* Add sparsify for rsync + merge_chain refactor * Linting

rsmontero pushed a commit that referenced this issue Mar 1, 2023

F #6029: Normalize exit codes for rsync backup actions (#2535)

e6652bd

Possible fix for "Message received: BACKUP SUCCESS 96 -" (?)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retention for incremental backups #6029

Retention for incremental backups #6029

rsmontero commented Dec 2, 2022 •

edited by tinova

Loading

nachowork90 commented Jan 5, 2023

FrederickBor commented Feb 15, 2023

Retention for incremental backups #6029

Retention for incremental backups #6029

Comments

rsmontero commented Dec 2, 2022 • edited by tinova Loading

Progress Status

nachowork90 commented Jan 5, 2023

FrederickBor commented Feb 15, 2023

rsmontero commented Dec 2, 2022 •

edited by tinova

Loading