Feature-request: native encryption with ZFS

[kvaps]

OpenZFS does support native encryption since 0.8.0.

Encryption can be enabled for whole pool, dataset or per ZVOL (our case).
It would be nice to add its support in roadmap for LINSTOR.

Here is examples:
https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/

[ghernadi]

For me this sounds like a feature request for the linstor physical-storage create-device-pool command.

As for an encryption of the whole zpool, the user would have to enable encryption before linstor sp c zfs ...
and for per ZVOL, you should be able to use the new StorDriver/ZfscreateOptions #128

[kvaps]

Yes looks like StorDriver/ZfscreateOptions #128 is exactly what we need, but the questions is how to input encryption password :)

[ghernadi]

So you also want to have a password?

Well.. in that case, I have to agree, that makes it a bit more difficult.

I am not a zfs expert, but if I understand this zfs-encryption correctly (for some reasons I could test it myself for now, I'll have to look into my test-setup) and you need to re-enter the password when you are mounting / start using the device, that will cause some problems in Linstor which would require quite a lot of changes...

First, right now, basically every layer in Linstor might be "on top" of the StorageLayer with a ZFS provider (except the StorageLayer itself).
That means, all of those layers have to be ready to enter that password before they start using the device.
This is something we highly tried to avoid, as basically every layer would have to access layer-specific data from the storage layer. Right now, (with only several small exceptions) every layer can "do its own thing" with its own layer specific data without having to rely on any other layer apart from "I need a device 'this' large" or "delete this device" or similar highly generic commands.

Second, storing / managing the password itself is a problem. The LuksLayer requires the user to remember only one passphrase (the master-passphrase), which Linstor never remembers and keeps in memory as short as possible. The actual Luks-passwords are generated, salted and encrypted (with the master-passphrase) before stored in the database.
With zfs-encryption, Linstor has to either let the user enter the password (upon resource create ..?) and has to provider an API showing the password in plaintext.
Yes, Linstor could somehow "securely" store that password, similar to the way LuksLayer's passwords are stored, but also that would require quite some rework.

All in all, although I do understand that one might want to use the feature especially "if it is already there". But on the other hand, Linstor already has a dedicated LuksLayer for encryption, which also works fine with ZFS. Unless someone explains me what other differences there are than "it is not the ZFS built in encryption", I guess this issue will get such a low priority, that it most likely will never get implemented.

[gmelikov]

Some small notes on topic:

• IIRC you may preload key once with zfs load-key [-nr] [-L keylocation] -a | filesystem https://openzfs.github.io/openzfs-docs/man/8/zfs-load-key.8.html?highlight=encrypt and mount/unmount/etc later
• most interesting thing in zfs native encryption is it's ability to scrub/send/recv/etc with encrypted data without keys, but I'm not sure how can linstor use it

[ghernadi]

Thanks for the info.

The load-key sounds interesting, especially if it can be used without the Linstor-changes I mentioned above.

Regarding the benefits: Linstor does not call / use scrub, but Linstor does use send/recv for snapshot shipping. But as you mentioned, if that also works without keys, load-key could be the answer to kvaps request :)

[ggzengel]

@kvaps What do you expect with encryption?

I use encryption to send disks to repair or throw away broken disks or protect data on stolen hardware.
For this the root filesystem is installed on a encrypted LUKS LV and asks for keys on boot. The keys for ZFS are stored on root filesystem and inherited to sub volumes.

[kvaps]

Well, right now I don't need an encryption for the LINSTOR.
But possible it will be needed in the future, I would prefer to choose native ZFS encryption instead of addition layer of LUKS

[ggzengel]

Only my root filesystem is LVM LUKS EXT4, because I don't trust to boot with ZFS and it's easier to repair grub and initramfs with debian's netinst.iso thru an IPMI session.
Everything else is native ZFS encryption.

If you encouraged you can boot from an encrypted ZFS.

[zrav]

Another vote for ZFS encryption handling.
We need encryption at rest, so my suggestion as to how this would work is based on that: The password should only have to be entered once on cluster start, then Linstor keeps the key in RAM and automatically handles any load-key operations. That would mean the same key is used for all storage pools/volumes, but that would be fine in our case.

[bernardgut]

+1 as I think it would be faster than LUKS for the ZFS case

IMO one key for everything is wrong. You encrypt zfs datasets not pools. Last time I checked, the encryption feature is turned on by default on all new Zpool (zfs-2.2.2). So you only need to decide if you turn it on/off at the dataset level. So in this case it would make sense to have a a per-pvc annotation along the lines of :

apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-encryptedVolume
  annotations:
     linstor.csi.linbit.com/encryption : true                # turns on native encryption if available. Only available at creation time
     linstor.csi.linbit.com/encryprionAlgorithm: " <aes-128-ccm | aes-192-ccm | aes-256-ccm | aes-128-gcm | aes-192-gcm | aes-256-gcm>"
     linstor.csi.linbit.com/encryptionKey: <128bit_KEY|256bit_KEY>      # encryption key to mount volume with. In Hexadecimal. Do not lose this.
specs:
...

1. The datapath doesnt change. (Unless I am missing something)
2. You need to add some extra code in the volume create path. The doc for the command is

zfs create \
    -o encryption=<off | on | aes-128-ccm | aes-192-ccm | aes-256-ccm | aes-128-gcm | aes-192-gcm | aes-256-gcm> \
    -o keysource=<raw | hex | passphrase>,<prompt | file://...> \
    -o pbkdf2iters=<n> \
    <dataset>

Assuming you dont allow passphrases (it doesn't makes sense. this is a key used by a machine) you are left with having to add the following things to your volume creation path:

key = $(etcdctl get pcv.annotations.linstor.csi.linbit.com/encryptionKey)
keyID = $(sha256sum key)
algo =  $(etcdctl get pcv.annotations.linstor.csi.linbit.com/encryptionAlgorithm) 
echo "$key" > /tmpfs/"$keyID"
zfs create \
    -o encryption=$algo \
    -o keysource=<hex,file:///tmpfs/$keyID \
    <dataset>
rm /tmpfs/$keyID

3. And the volume mount path needs to add a few things:

key = $(etcdctl get pcv.annotations.linstor.csi.linbit.com/encryptionKey)
keyID = $(sha256sum key)
echo "$key" > /tmpfs/"$keyID"
zfs load-key -L /tmpfs/"$keyID" <dataset>
zfs mount  <dataset>
rm /tmpfs/$keyID

This implementation would mean that your disks are encrypted until they are mounted by a running linstor-server instance that has access to your kubernetes cluster etcd. IMO this is the correct way to do "encryption at rest in this case"

This is just pseudocode. Of course I understand this is the k8s case and linstor-server also exists outside of k8s but you get the idea. But I think it would be a good start. We can also discuss later about the security of having keys in tmps and in an annotation, or the optimization of loading the keys locally at container boot time instead as for each mount, but these are problems that can be solved subsequently.

What do you think?