Risk of metadata partition filling up with LVM Thin Provisioning

[qubesuser]

According to "man lvmthin" section "Metadata space exhaustion", exhaustion of a thin pool's metadata space can cause "inconsistent thin pool metadata and inconsistent file systems" (and even if that's not accurate it would necessarily stop thin volume creation and thus starting and creating VMs), so it would be important to avoid it.

Currently Qubes installer seems to create a ~100MB metadata partition and also leave ~15-16GB free. According to "man lvmthin" it seems possible to have the metadata partition automatically grow, but it's not clear how, and not clear whether this is enabled by the Qubes installer.

It would be nice to make sure that either the metadata partition is automatically extended up to the 16GB limit or just create a 16GB metadata partition from the start.

[brendanhoar]

Chris (@tasket) has also noted the above issue. On my primary Qubes machine, the default lvm metadata allocation is 112MB for a 472GB data allocation. Increasing it seems like a no-brainer, as some Qubes features make heavy use of it.

Suggested mechanisms to avoid metadata full->corruption in Qubes:

1. If possible, on the next release, modify anaconda config to double the default lvm metadata allocation. I suspect that would cover 99.99% of use cases and dramatically reduce this failure mode.

2. In a future release, add metadata monitoring to the Qubes Disk Space Monitor widget.

3. In a future release, consider adding a "pause all VMs except dom0 and warn user" at threshold data and/or metadata values of usage (e.g. 95%). Via the widget or some other mechanism.

4. In addition, pull in most recent stable LVM release. E.g. April's LVM release includes some logic to auto-extend the metadata (but only when a LVM volume is resized...which can be configured to happen automatically): https://sourceware.org/git/?p=lvm2.git;a=commit;h=e27d0271557d4b93e87a70854b3c7f1cc6008155

"committer Zdenek Kabelac <zkabelac@redhat.com>
Wed, 3 Apr 2019 11:28:22 +0000 (13:28 +0200)"
"Automatically grow thin metadata, when thin data gets too big."

B

[tasket]

I definitely agree with the first three.

However #4 might not help because it appears to be using an estimation routine that 's also used when the pool is initialized. Therefore, if Qubes installer is doubling the initial tmeta size, this patch would only extend tmeta further if the user were to increase pool size to more than double.

So I think #2+3 is much more valuable, and the widget could even prompt the user, offering to extend tmeta itself.

Long-term considerations:

Recalling the discussion we had in qubes-users on this topic, I came to a conclusion that defragmentation was a missing ingredient. It seems necessary because it keeps metadata complexity/size to manageable ratios with respect to the overall pool size. With defragmentation, you can handle metadata growth the way NTFS does which is far less likely to lead to catastrophic data loss.

I don't know what the likelihood is of a pool defrag tool being accepted in the thin-provisioning-tools package. There is a defragger for regular LVM volumes which has languished for over a decade.

[awokd]

modify anaconda config to double the default lvm metadata allocation

Can this be a "must have" for R4.1? Quadruple even. The current failure mode is punishing.

[marmarek]

I'm trying to find the right value and I'm confused. Metadata size is chosen here. Internally it uses thin_metadata_size tool.
So, lets play a little. Input data:

$ sudo lvs -o+chunksize -a qubes_dom0/pool00 qubes_dom0/pool00_tmeta qubes_dom0/pool00_tdata
  LV             VG         Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Chunk  
  pool00         qubes_dom0 twi-aotz-- 208.20g             94.03  79.93                            128.00k
  [pool00_tdata] qubes_dom0 Twi-ao---- 208.20g                                                          0 
  [pool00_tmeta] qubes_dom0 ewi-ao---- 108.00m                                                          0 

First, lets try to reproduce this calculation (108MB of metadata volume):

thin_metadata_size - 53.22 mebibytes estimated metadata area size for "--block-size=128kibibytes --pool-size=208gibibytes --max-thins=100"

Something is off. Maybe chunk size?

thin_metadata_size - 106.04 mebibytes estimated metadata area size for "--block-size=64kibibytes --pool-size=208gibibytes --max-thins=100"

Ok, looks like it (blivet apply rounding on top of this number).
So, lets try to calculate more appropriate number for Qubes use case, for much more snapshots/devices:

thin_metadata_size - 109.56 mebibytes estimated metadata area size for "--block-size=64kibibytes --pool-size=208gibibytes --max-thins=1000"

It's almost the same! Maybe even more:

thin_metadata_size - 144.71 mebibytes estimated metadata area size for "--block-size=64kibibytes --pool-size=208gibibytes --max-thins=10000"

Ok, here I got a bigger number.
But it still looks wrong. This pool has about 90 LVs (including snapshots) and is almost full (both data and metadata). Choosing "--max-thins=1000" wouldn't help that much.
That's all using block size 64k, even though LVM reports chunk size of 128k - which would result in even smaller numbers.

I consider using whatever number it calculate and doubling it.

[marmarek]

Additionally, I don't see any clean way to specify metadata size while using automatic partitioning. There are multiple abstraction layers, without much flexibility in extra arguments. I'm afraid I'll need to settle down on a blivet patch adjusting the defaults.

[qubesos-bot]

Automated announcement from builder-github

The package pykickstart-2.32-4.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package pykickstart-2.32-4.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update