Receiving stream with bad parameters results in misleading error "kernel modules must be upgraded"

[stewartadam]

Creating separate issue from 'Scenario 4' described in #6547 per request.

System information

Type	Version/Name
Distribution Name	Fedora
Distribution Version	25
Linux Kernel	4.12.8-200.fc25
Architecture	x86_64
ZFS Version	0.7.1-1 (with encryption patches)
SPL Version	0.7.1-1

I've installed SPL from the upstream f25 repo and compiled 0.7.1 from source, patching in:

• Add libtpool (thread pools) (46364cb)
• Send / Recv Fixes following b52563 (9b84076)
• Native Encryption for ZFS on Linux (b525630)

Describe the problem you're observing

When resuming a zfs send transfer but specifying an incorrect destination, ZFS produces an incorrect error message cannot receive incremental stream: kernel modules must be upgraded to receive this stream..

The problem is unrelated entirely to kernel modules and rather that the wrong dataset has been specified for the zfs recv operation.

Describe how to reproduce the problem

truncate -s 100M /root/src.img
truncate -s 100M /root/replica.img
zpool create src /root/src.img
zpool create replica /root/replica.img
zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt replica/encrypted
zfs create src/myfiles
dd if=/dev/urandom of=/src/myfiles/random.bin bs=1M count=30
zfs snapshot src/myfiles@migration
zfs create src/myfiles/child
dd if=/dev/urandom of=/src/myfiles/child/random.bin bs=1M count=3
zfs snapshot src/myfiles/child@migration
zfs send src/myfiles@migration | zfs recv -s replica/encrypted/myfiles
zfs send src/myfiles/child@migration | pv -L 500K | zfs recv -s replica/encrypted/myfiles/child
# press ^c after a second or two of transfer starting

TOKEN=$(zfs get -H receive_resume_token "replica/encrypted/myfiles/child" | cut -f3 | grep -ve '^-$')

zfs send -t "$TOKEN" | zfs recv -s replica/encrypted/myfiles
# cannot receive incremental stream: kernel modules must be upgraded to receive this stream.

zfs send -t "$TOKEN" | zfs recv -s replica/encrypted/myfiles@migration
# cannot receive incremental stream: kernel modules must be upgraded to receive this stream.

zfs send -t "$TOKEN" | zfs recv -s replica/encrypted/myfiles/child@migration
# works

[stewartadam]

See also #6547 (comment) by @tcaputi:

What's happening here is that the kernel is doing a bunch of sanity checks to make sure that its resuming the correct receive. If any of these checks fail it will return EINVAL and all EINVALs end up printing that message.

[jgottula]

See also #6366 (comment) [specifically section 2 of that comment], where I talk complain about essentially the exact same situation.

In the course of doing something like receiving a snapshot, there's a decent number of reasons why the ioctl might result in an errno like EEXIST or EINVAL due to various different problems in the kernel module. But libzfs, at least in zfs_receive_one and probably elsewhere, takes a fairly dumb approach and mostly just uses the errno itself and maybe some other limited hints to infer what went wrong, and can very easily jump to the wrong conclusion as a result.

[LnxGnome]

+1 "cannot receive incremental stream: kernel modules must be upgraded to receive this stream" error with CentOS 7.8.2003 (Kernel 3.10.0-1127.8.2.el7.x86_64) & zfs-0.7.13-1.el7_6.x86_64 (OS updated > 7.6, zfs held back), no patches, no encryption.

zfs send -t "$TOKEN" | mbuffer -O ip:port   
-/\/-  
mbuffer -I port | zfs recv -uvsF replica
# fails

zfs send -t "$TOKEN" | mbuffer -O ip:port   
-/\/-  
mbuffer -I port | zfs recv -uvsF replica/myfiles/child
# works

Using different hosts with "-/\/-" being an IPoIB network

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.