[Contribution] qubes-incremental-backup-poc OR Wyng backup #858
Comments
marmarek
added
T: bug
This comment was marked as outdated.
This comment was marked as outdated.
marmarek
added
C: core
and removed
C: artwork
marmarek
added
T: enhancement
|
Comment by joanna on 14 May 2014 10:42 UTC https://groups.google.com/d/msg/qubes-devel/Gcrb7KQVcMk/CK-saQU_1HYJ |
andrewdavidwong
added
the
help wanted
It looks like Duplicity supports making an incremental backup even when part of a file was changed (include diff of file, not full changed files). So indeed it may be good idea to somehow use it. But mounting VM filesystem in dom0 is a big NO-GO. On the other hand, it may be good idea to simply run duplicity in the VM, and collect its output. Take a look at the linked discussion for an idea how to handle powered off VMs (in short: launch minimal "stubdomain" like system with access to its disk for the backup purpose). Of course all this requires evaluation whether duplicity correctly handle encryption / validation. To not make things worse than they currently are... |
|
On 06/12/2016 07:04 PM, Marek Marczykowski-Górecki wrote:
No need to mount any VM filesystem in dom0. That is also not how The way it would work with Duplicity, is a backend would need to be
See this: https://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/files/head:/duplicity/backends/ Later today, or perhaps tomorrow, I will use my Qubes bombshell-client |
|
Better still: Remarkably much like my Qubes Ansible plugin. |
|
Can you explain more how exactly that would work?
Generally dom0 shouldn't be exposed to VM filesystem in any way (regardless of the form: mounting directly, accessing via sftp-like service etc). If incremental data needs to be computed at VM-file level, it should be done in separate VM and dom0 should treat the result as opaque blob. Also during restore. |
|
On 06/12/2016 08:15 PM, Marek Marczykowski-Górecki wrote:
Duplicity + hypotheticalplugin (from now on, HP) would never retrieve
Incremental data is computed in dom0 using both the rdifflib database
That cannot work because that involves copying the IMG files into the Duplicity's backends are designed to make the storage opaque to |
Not necessary - it can be attached with qvm-block-like mechanism. Anyway, IIUC you want to backup |
|
Well, look at what I have got here: I have a (shit) complete Qubes VM backend for duplicity which you can add to your dom0's duplicity backends directory, and then run something like this: duplicity /var/lib/qubes qubesvm://backupvm/Backupsmountpoint/mydom0/qubesvms Very early working prototype and building block. Enjoy! |
|
I have some experience with Duplicity:
I would prefer using Duplicity per-VM, so one could exclude caches etc. from the backup. My idea is something like the following process invoked from dom0:
As you can see, this would require some data stored in dom0 to be also backed up, but this could be handled by some very similar mechanism. (It would probably run directly in dom0 rather than in dom0.) Some implementation and security notes on those points:
|
# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
#
# This file is NOT part of duplicity. It is an extension to Duplicity
# that allows the Qubes OS dom0 to back up to a Qubes OS VM.
#
# Duplicity is free software, and so is this file. This file is
# under the same license as the one distributed by Duplicity.
import os
import pipes
import subprocess
import duplicity.backend
from duplicity.errors import *
BLOCKSIZE = 1048576 # for doing file transfers by chunk
MAX_LIST_SIZE = 10485760 # limited to 10 MB directory listings to avoid problems
class QubesVMBackend(duplicity.backend.Backend):
"""This backend accesses files stored on a Qubes OS VM. It is intended to
work as a backed within a Qubes OS dom0 (TCB) for the purposes of using
Duplicity to back up to a VM. No special tools are required other than
this backend file itself installed to your Duplicity backends directory.
Missing directories on the remote (VM) side will NOT be created. It is
an error to try and back up to a VM when the target directory does
not already exist.
module URL: qubesvm://vmname/path/to/backup/directory
"""
def __init__(self, parsed_url):
duplicity.backend._ensure_urlparser_initialized()
duplicity.backend.urlparser.uses_netloc.append("qubesvm")
duplicity.backend.urlparser.clear_cache()
properly_parsed_url = duplicity.backend.ParsedUrl(parsed_url.geturl())
duplicity.backend.Backend.__init__(self, properly_parsed_url)
if properly_parsed_url.path:
self.remote_dir = properly_parsed_url.path
else:
self.remote_dir = '.'
self.hostname = properly_parsed_url.hostname
def _validate_remote_filename(self, op, remote_filename):
if os.path.sep in remote_filename or "\0" in remote_filename:
raise BackendException(
("Qubes VM %s failed: path separators "
"or nulls in destination file name %s") % (
op, remote_filename))
def _dd(self, iff=None, off=None):
cmd = ["dd", "status=none", "bs=%s" % BLOCKSIZE]
if iff:
cmd.append("if=%s" % iff)
if off:
cmd.append("of=%s" % off)
return cmd
def _execute_qvmrun(self, cmd, stdin, stdout):
subcmd = " ".join(pipes.quote(s) for s in cmd)
cmd = ["qvm-run", "--pass-io", "--", self.hostname, subcmd]
return subprocess.Popen(
cmd,
stdin=stdin,
stdout=stdout,
bufsize=MAX_LIST_SIZE,
close_fds=True
)
def put(self, source_path, remote_filename=None):
"""Transfers a single file to the remote side."""
if not remote_filename:
remote_filename = source_path.get_filename()
self._validate_remote_filename("put", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(off=rempath)
fobject = open(source_path.name, "rb")
try:
p = self._execute_qvmrun(cmd,
stdin=fobject,
stdout=open(os.devnull))
except Exception, e:
raise BackendException(
"Qubes VM put of %s (as %s) failed: (%s) %s" % (
source_path.name, remote_filename, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM put of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
source_path.name, remote_filename, err))
def get(self, remote_filename, local_path):
"""Retrieves a single file from the remote side."""
self._validate_remote_filename("get", remote_filename)
rempath = os.path.join(self.remote_dir, remote_filename)
cmd = self._dd(iff=rempath)
fobject = open(local_path.name, "wb")
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull),
stdout=fobject)
except Exception, e:
raise BackendException(
"Qubes VM get of %s (as %s) failed: (%s) %s" % (
remote_filename.name, local_path, type(e), e))
finally:
fobject.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM get of %s (as %s) failed: writing the "
"destination path exited with nonzero status %s") % (
remote_filename.name, local_path, err))
def _list(self):
"""Lists the contents of the one duplicity dir on the remote side."""
cmd = ["find", self.remote_dir, "-maxdepth", "1", "-print0"]
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=subprocess.PIPE)
except Exception, e:
raise BackendException(
"Qubes VM list of %s failed: %s" % (self.remote_dir, e))
data = p.stdout.read(MAX_LIST_SIZE)
p.stdout.close()
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM list of %s failed: list command finished "
"with nonzero status %s" % (self.remote_dir, err)))
if not data:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"empty" % (self.remote_dir,)))
filename_list = data.split("\0")
if filename_list[0] != self.remote_dir:
raise BackendException(
("Qubes VM list of %s failed: list command returned a "
"filename_list for a path different from the remote folder") % (
self.remote_dir,))
filename_list.pop(0)
if filename_list[-1]:
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"wrongly-terminated data or listing was too long") % (
self.remote_dir,))
filename_list.pop()
filename_list = [ p[len(self.remote_dir) + 1:] for p in filename_list ]
if any(os.path.sep in p for p in filename_list):
raise BackendException(
("Qubes VM list of %s failed: list command returned "
"a path separator in the listing") % (
self.remote_dir,))
return filename_list
def delete(self, filename_list):
"""Deletes all files in the list on the remote side."""
if any(os.path.sep in p or "\0" in p for p in filename_list):
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command asked to delete a file with a path separator "
"or a null character in the listing") % (
self.remote_dir,))
pathlist = [os.path.join(self.remote_dir, p) for p in filename_list]
cmd = ["rm", "-f", "--"] + pathlist
try:
p = self._execute_qvmrun(cmd,
stdin=open(os.devnull, "rb"),
stdout=open(os.devnull, "wb"))
except Exception, e:
raise BackendException(
"Qubes VM delete of files in %s failed: %s" % (
self.remote_dir, e))
err = p.wait()
if err != 0:
raise BackendException(
("Qubes VM delete of files in %s failed: delete "
"command finished with nonzero status %s") % (
self.remote_dir, err))
duplicity.backend.register_backend("qubesvm", QubesVMBackend)Finally: I disagree with the direction of implementation which suggests we need to play Towers of Hanoi with a backup VM and attaching disk images to it. That's entirely unnecessary complication, and it also demands VMs be off for backup purposes. Entirely unnecessary and extremely discouraging of backup procedures. The three step process is all that is necessary:
That is all that is needed. Of course, the Qubes setting of which VMs to back up, plus options like |
|
Update: Duplicity is no good for the process, because differential backups slow to a crawl. We need to research other stuff, like Attic. |
|
I have not experienced such issue with Duplicity. I have managed to perform incremental backup on roughly 15GiB (after exclusions) of various data (many small files and few large files) in ~2 minutes even on HDD + dm-crypt. (Of course, this depends on size of changes.) Encryption and compression were enabled. Maybe it skips many files just because of timestamps and diffs just few files. So, I feel you must be doing something wrong. (No offense.) What was your scenario? What files did you try to backup? (E.g., dom0's ~ with 1GiB of small files.) What was your drive setup? (E.g., 7200 RPM HDD with dm-crypt with ext4.) How much time did it take? Where do you store the backup? (Other VM using your storage backend? Well, theoretically, this should not matter so much for performance of scanning, as Duplicity caches metadata locally, AFAIK somewhere in ~/.cache. But I am curious.) Did you have the metadata cached? How much data did you add to the backup? (Well, if you backup ~ and you don't exclude ~/.cache, you might add Duplicity metadata to the backup, which could explain both some time and space penalty. I am not sure if Duplicity is smart enough to exclude this automagically.) |
|
On 07/19/2016 05:45 PM, Vít Šesták wrote:
My backup of a 60 GB disk image progressed essentially nothing over the |
|
Qubes is image-based os. All other mentioned tools are file-based tools. You can use file-based tool for raw images, but that requires full re-scan of that images to detect changes for incremental backup. Wyng is different, it is image-based tool and filesystem-agnostic and it does not require full re-scan, how is this possible ? LVM magic! |
|
Update: Wyng has entered its final alpha for v0.4. The big changes have been completed, which include:
There is also a Qubes integration script which allows backup/restore by VM name. Currently this only works with LVM but reflink support is planned. |
|
Is it possible to see a comparison between Wyng and rdiff-backup (which is also used for incremental backup of VMs)?
|
|
Would love to see support for incremental backups based on the ZFS volume driver too. |
|
Wyng now has a format specification draft: https://github.com/tasket/wyng-backup/tree/04alpha3/doc/Wyng_Archive_Format_V3.md |
|
Once |
|
@tasket maybe a quick update is needed here? Or @marmarek, I see you removed assignment to yourself. Maybe you want to say what is missing to go forward under tasket/wyng-backup#102 instead? |
That's not specific to this issue. It's just a general change in how issue assignments are used. Before, issues were assigned to people who work in certain areas, even if there was no current work being done (including just "someday, maybe"). Now, by contrast, issues are only assigned to devs while those devs are actively working on the issues. You can read more about the new policy here. |
|
@tlaurion @marmarek Wyng is now in final beta for v0.8 and all features are frozen. Its exhibiting good stability overall, but I have added a caveat to the Readme about lvmthin's need for metadata space since adding Wyng's snapshots on top of Qubes' snapshots will naturally consume more and LVM does not have good defaults or out-of-space handling. The wyng-util-qubes wrapper for Qubes integration has just gone to v0.9 beta and pushed to main branch, as I recommend using this version now. The new version includes support for both reflink and lvmthin pools; the wrapper can now alias volume names as necessary between the two pool types during This is what a typical backup session from a Btrfs Qubes system looks like: This is a restore from a backup session containing two VMs: Currently, the Qubes default pool for newly created VMs, or whichever one an existing VM resides, is used for restore but you can also specify |
|
Also, if wyng is to be considered in newer versions of QubesOS, I would recommend thinking about seperating dom0 data from qubes hypervisor. That is a whole separated topic here, but on my side i'm used wyng against a lot of cloned templates and cloned qubes which results in the following: To talk about moving to brtfs+ bees dedup here is out of point of course and would be #6476, not here. But since the caveat of TLVM metadata is touched, I would love to take the opportunity to remind devs that having separated dom0 from vm-pool was one step into the direction of limiting impacts of using TLVM in the first place under Qubes considering that QubesOS uses a lot of sometimes stalling snapshots for back+x and volatile volumes which could have locked the user out, which impacts pool metadata, a lot, also to be discussed under #6476 not here. All of those could of course be dodged altogether if TLVM was reconsidered as first QubesOS candidate to envvision going into something more practical fitting specialized cloned templates, salting qubes and massive qubes private disk data preventing quick shutdown and all that jazz. But that would be #6476 not here. So I would just want to remind that using wyng consumes dom0 space for meta-dir (chunks mapping downloaded from the archive qube) and that dom0 LVM is 20GB considering past decisions (it could be within same pool if not TLVM), and that extending TLVM pool metadata requires to steal some of the swap space (manually), while having 20Gb today is quite limitating (cannot instlal multiple templates, be cautious and check dom0 usage) because dom0 should actually be quite static, if dom0 was just dom0 without dom0 keeping external states that should not exist in dom0 in the first place. @tasket and @marmarek @DemiMarie : amazing work outside of those critical, but constructive points, as usual :) Currently ongoing: users hacking qubes-backup encryption/authentication to use incremental backup tools: ... What about writing a grant application to achieve that pressing goal? Yes? |
|
@tlaurion Just FYI, Wyng has an issue open (109) for reducing metadata footprint. Currently it keeps both a compressed and uncompressed copy of manifests in /var. It does set the compression bit on the uncompressed files, so if /var is on Btrfs or other compressing fs then used space will be reduced somewhat. To be completely honest, the main issue with that issue is the low level of interest in it, otherwise I think it would have been done already. BTW, you can keep deleting all of the 'manifest' files, which should reduce the footprint by up to 2/3.
Its not something I'm familiar with, and the goal needs to be better-defined. AFAIC, Wyng is fully functional now and the only things a typical user would really miss would be a GUI and perhaps a way to mount archived volumes. |
@tasket Can you contact me over Matrix /mastodon/QubesOS forum? |
|
@marmarek @tasket @rapenne-s should we team up for a grant application? Interest? |
I'm be happy to work on this |
Random thoughts for grant application. What I envision to be done as of now :
@rapenne-s I think you are amazing candidate for documentation and insights on infra optimization. @marmarek @marmarta @DemiMarie UX and GUI integration is needed. Fill us in with high level requirements? @tasket of course to tackle wyng-backup wyng-util-qubes work and input other features envisioned missing to reach massive adoption with clear use cases. @tlaurion : heads and general plumbing, facing NLnet, grant application writeup, validation of proof of work (PoW, normally PR) prior of request for payment (RfP) and project management. Heads integration, possibly using SecureDrop workstation use case to drive this. @deeplow interest from FPF' SD and interfacing with them with requirements? Goal here would be to have restorable states from network booted environement to pull states from the network and prepare drive content to be ready to use under minutes, hosted on FPF infra. Maybe we should create discussion on QubesOS forum if there is interest or in a seperate issue? Whether you see fit best. Notes on grant application process. Grant work is paid upon PoW for scoped tasks. So for teaming up here, we need first acceptation (consent) of engagement into doing the work within a year after grant application is accepted. Scoping of general tasks, by whom the work will be done and required approximate funds to be budgetized do such high level tasks, to be then broken down in smaller tasks upon project approval in terms of deliverables paid upon PoW. @rapenne-s @marmarek @marmarta @deeplow @DemiMarie @tasket : would you be willing to engage into teaming up to expend on the needs of such integration and documentation, and agree on the first step, which is to consent into accomplishing such integration as a goal if such grant application was accepted to fund the work needed? |
|
Aside from organizational plans, I've decided that there will have to be a beta5 now. The metadata caching issue @tlaurion pointed out needs to be addressed before v0.8 rc, along with a unicode-handling issue I identified. Fixes are now in the 08wip branch and I expect beta5 to be available within a week. Session metadata will now be aged-out from /var/wyng before 3 days, by default. This can be controlled using an option like |
|
@tlaurion here is what I'd like from a backup solution, to be considered a replacement for the current (non-incremental) one:
Those are the main ones on the functional side. Some are optional (marked "nice to have"). Some (if not most) are already satisfied by wyng, but I've written them down anyway. Some of those apply to Wyng itself, some to its integration with Qubes OS. Lets discuss how we can make this happen :) |
@tasket yhis needs a thorough project status update from you here! |
|
@marmarek Thanks for taking time to post your queries/requirements. My answers follow:
For backing up the backup, simple CC @tlaurion |
|
One more nice-to-have thing I would like to see: possibility to exclude a directory/file (not sure if possible since wyng works on LVMs) or at least backup just the VM metadata (name, color, netvm, template name, etc.) without backing up its contents. Example use case: You have a "media" qube where you watch videos offline. You download a playlist from youtube with |
|
@marmarek @tasket #858 (comment): ping |
Community Devs: @v6ak, @tasket
@v6ak's PoC: https://github.com/v6ak/qubes-incremental-backup-poc
@tasket's PoC: https://github.com/tasket/wyng-backup | Status update as of 2022-08-16: #858 (comment)
Reported by joanna on 14 May 2014 10:38 UTC
None
Migrated-From: https://wiki.qubes-os.org/ticket/858
Note to any contributors who wish to work on this issue: Please either ask for details or propose a design before starting serious work on this.
The text was updated successfully, but these errors were encountered: