Ubuntu 20.04.2 LTS

[ccrssaa]

Hi, nice tool, thanks a lot

tried it in Ubuntu 20.04.2 LTS

1. apparmor issues - libvirtd not allowed to create sockets in /var/tmp

2021-04-20 17:38:50 INFO common - printVersion: Version: 0.17 Arguments: /usr/local/bin/virtnbdbackup -d vm-template -l full -o /mnt/backup/vm-template
2021-04-20 17:38:50 INFO virtnbdbackup - main: Domain has 1 disks attached which support changed block tracking.
2021-04-20 17:38:50 INFO virtnbdbackup - main: Looking for checkpoints
2021-04-20 17:38:50 INFO virtnbdbackup - main: Using checkpoint name: virtnbdbackup
2021-04-20 17:38:50 INFO virtnbdbackup - main: Temporary scratch file target directory: /var/tmp
2021-04-20 17:38:50 INFO virtnbdbackup - main: Starting backup job.
2021-04-20 17:38:50 WARNING libvirthelper - startBackup: Guest agent is not responding: QEMU guest agent is not connected
2021-04-20 17:38:50 ERROR virtnbdbackup - main: internal error: unable to execute QEMU command 'nbd-server-start': Failed to bind socket to /var/tmp/virtnbdbackup.868194: Permission denied

obvious workaround is to disable apparmor

extenthandler.py:
def _setRequestAligment(self):
align = self._nbdFh.get_block_size(0)
if align == 0:
align = self._align
return self._maxRequestBlock - align + 1

Ubuntu has libnbd 1.2.2 with missing get_block_size()
is it safe to skip get_block_size() and use self._align if no get_block_size() available ?

[abbbi]

hi,

tried it in Ubuntu 20.04.2 LTS
2021-04-20 17:38:50 ERROR virtnbdbackup - main: internal error: unable to execute QEMU command 'nbd-server-start': Failed to bind socket to /var/tmp/virtnbdbackup.868194: Permission denied

obvious workaround is to disable apparmor

nice catch, i will add it to the FAQ.

extenthandler.py:
def _setRequestAligment(self):
align = self._nbdFh.get_block_size(0)
if align == 0:
align = self._align
return self._maxRequestBlock - align + 1

Ubuntu has libnbd 1.2.2 with missing get_block_size()
is it safe to skip get_block_size() and use self._align if no get_block_size() available ?

i think it is. I cant recall completely the usage here, i think it has to do with a NDB Server can deny
requests from a client if not both client and server are using the same block aligment.

From the Description of get_block_size():

If non-zero, this will be a power of 2 between 1 and 64k; any client
 request that is not aligned in length or offset to this size is likely
to fail with C<EINVAL>.  The image size will generally also be a
multiple of this value (if not, the final few bytes are inaccessible
while obeying alignment constraints).  If zero, it is safest to
 assume a minimum block size of 512, 

so i think it would be save to assume 512 like it does if the function is missing entirely.
I have however not tested any older libnbd version, so there might be other issues or incompatibilities too.

[juliyvchirkov]

@abbbi seems it should be enough to add /var/tmp/virtnbdbackup.* rw, to the end of /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper config to get rid of Failed to bind socket [. . .] Permission denied error instead of disabling apparmor with aa-teardown completely

testing on Ubuntu 20.04

[juliyvchirkov]

@abbbi I'm back with details after the thorough testing

To get rid of Failed to bind socket [. . .] Permission denied error error, one should put 2 following lines

   /var/tmp/virtnbdbackup.* rw,
   /var/tmp/backup.* rw,

to 3 config files /etc/apparmor.d/local/abstractions/libvirt-qemu, /etc/apparmor.d/local/usr.sbin.libvirtd and /etc/apparmor.d/local/usr.lib.libvirt.virt-aa-helper (these files are empty by default)

Also I've designed and tested the bash bootstrap /usr/bin/kvmbackup to implement non-interactive backup operations cycle for active kvm machines with your virtnbdbackup tool

The first backup of every vm on every new month is forced to be full size, the rest backups on the month are incremental. Every backup is processed with lz4 compression 'cause this feature really saves the host drive space a lot. The bootstrap keeps reporting the flow logs on stdout if one runs it from console, in crontab these reports can be silenced in standard way

1 2 * * * root bash /usr/bin/kvmbackup >/dev/null 2>&1

I've uploaded the sources to review on my gist

Please feel free to utilize the above info and the bootstrap in your next releases, and thank you for the great simple tool for kvm backups!

[abbbi]

Please feel free to utilize the above info and the bootstrap in your next releases, and thank you for the great simple tool for kvm backups!

thanks for your feedback! i will have a look and update the README accordingly!

[tuomotalvitie]

As a side note for the AppArmor discussion, in Ubuntu 22.04 the sockets did not work just by adding the config files mentioned (and reloading apparmor) unless I made some error I could not catch at the time (despite checking things a few times)

I did get it working by: installed apparmor-profiles (and reloaded apparmor) (Note: I'm guessing that this was not necessary, but I'm including it here for completeness)

/etc/apparmor.d/abstractions/libvirt-qemu

# Site-specific additions and overrides. See local/README for details.

  include <local/abstractions/libvirt-qemu>

and set the config in
/etc/apparmor.d/local/abstractions/libvirt-qemu to

/var/tmp/virtnbdbackup.* rw,
/var/tmp/backup.* rw,

I may be able to later confirm this on another machine. (Especially if the apparmor-profiles was needed, or some other sunspot activity affected the results.)

(Once I got the socket working I found out that my images were qcow2, but progress is progress)

[tuomotalvitie]

Tested on another Ubuntu 22.04.

/etc/apparmor.d/local/abstractions/libvirt-qemu with contents of

/var/tmp/virtnbdbackup.* rw,
/var/tmp/backup.* rw,

and then reloading apparmor

service apparmor reload

was everything that was required for the creation of socket

[peter-kaagman]

I'm by no means an apparmor expert, not able to solve this on my own, but:

Was getting the error about the socket and added the lines as suggested in the REAME. This did in fact solve the issue about the socket, but that was just because apparmor no longer runs.

When trying to reload apparmor.service I get:

024-11-26T17:24:41.152366+01:00 kliko apparmor.systemd[6826]: Restarting AppArmor
2024-11-26T17:24:41.153647+01:00 kliko apparmor.systemd[6826]: Reloading AppArmor profiles
2024-11-26T17:24:41.157544+01:00 kliko kernel: kauditd_printk_skb: 113 callbacks suppressed
2024-11-26T17:24:41.157558+01:00 kliko kernel: audit: type=1400 audit(1732638281.155:415): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" p
rofile="unconfined" name="1password" pid=6835 comm="apparmor_parser"
2024-11-26T17:24:41.157560+01:00 kliko kernel: audit: type=1400 audit(1732638281.155:416): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="Discord" pid=6836 comm="apparmor_parser"
2024-11-26T17:24:41.157561+01:00 kliko kernel: audit: type=1400 audit(1732638281.155:417): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="QtWebEngineProcess" pid=6838 comm="apparmor_parser"
2024-11-26T17:24:41.157562+01:00 kliko kernel: audit: type=1400 audit(1732638281.155:418): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=6837 comm="apparmor_parser"
2024-11-26T17:24:41.158561+01:00 kliko kernel: audit: type=1400 audit(1732638281.156:419): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="balena-etcher" pid=6839 comm="apparmor_parser"
2024-11-26T17:24:41.158569+01:00 kliko kernel: audit: type=1400 audit(1732638281.156:420): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="buildah" pid=6841 comm="apparmor_parser"
2024-11-26T17:24:41.158569+01:00 kliko kernel: audit: type=1400 audit(1732638281.156:421): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="busybox" pid=6842 comm="apparmor_parser"
2024-11-26T17:24:41.159574+01:00 kliko kernel: audit: type=1400 audit(1732638281.157:422): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="brave" pid=6840 comm="apparmor_parser"
2024-11-26T17:24:41.159585+01:00 kliko kernel: audit: type=1400 audit(1732638281.157:423): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="cam" pid=6843 comm="apparmor_parser"
2024-11-26T17:24:41.159586+01:00 kliko kernel: audit: type=1400 audit(1732638281.157:424): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="ch-checkns" pid=6844 comm="apparmor_parser"
2024-11-26T17:24:41.201813+01:00 kliko apparmor.systemd[6931]: AppArmor parser error for /etc/apparmor.d in profile /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper at line 91: syntax error, unexpected TOK_MODE, expecting TOK_OPEN
2024-11-26T17:24:41.207398+01:00 kliko apparmor.systemd[6826]: Error: At least one profile failed to load
2024-11-26T17:24:41.208239+01:00 kliko systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
2024-11-26T17:24:41.208457+01:00 kliko systemd[1]: apparmor.service: Failed with result 'exit-code'.
2024-11-26T17:24:41.208806+01:00 kliko systemd[1]: Failed to start apparmor.service - Load AppArmor profiles.

Am on Ubuntu 24.04.1 LTS

[abbbi]

there is an syntax error in your apparmor configuration in line 91:

AppArmor parser error for /etc/apparmor.d in profile /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper at line 91: syntax error, unexpected TOK_MODE, expecting TOK_OPEN

[peter-kaagman]

I know, but that is the first line of the copied content from the README:

/var/tmp/virtnbdbackup.* rw,
/var/tmp/backup.* rw

BTW
Impressed by your speed of reaction, was wondering if reacting on a close issue would work :D

[abbbi]

i have actually no idea how to solve this apparmor fuckery. Its a pain in the ass.
Maybe its better to use a socketfile in a directory that libvirt allows to write, try passing a socket file in another location than /var/tmp/

virtnbdbackup -f /path/to/socketfile [..]

[peter-kaagman]

Ha ha ha... wonderfull. And I thought I was the only one disliking stuff like apparmor.

Will give it a shot, never occured to me I could specify another location.

Thanks for the reply, and thanks for the tool. Love it.

[abbbi]

On my system (debian 12) its fine with these lines:

tail -n 7 /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

  /var/tmp/virtnbdbackup.* rw,
  /var/tmp/backup.* rw,

  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.lib.libvirt.virt-aa-helper>
}

apparmor_parser -qp /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

does not report any parsing error. Maybe also depends on the apparmor version? idk.

[peter-kaagman]

The problem, as usual, occurs right between the back of my seat and the keyboard

/var/tmp/virtnbdbackup.* rw,
/var/tmp/backup.* rw

The last line is missing a komma at the end.

Furthermore: in the file /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper the lines should be put before the acolade,. Not at the very end of the file.

Thanks again for your help, and sorry for the noise.