make buildworld on a ZoL-originated pool results in invalid binaries

[rincebrain]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	13-CURRENT
Architecture	amd64
ZFS Version	zfs-kmod-v2020022700-ZoF_5db583ba1

Describe the problem you're observing

Someone reported in #zfsonlinux that they were having a problem where if they ran make buildworld on a root zpool that was created on ZoL, running make installworld would result in their system no longer booting, reporting exec format error on binaries.

I successfully reproduced this in a VM, and here I am reporting it.

Describe how to reproduce the problem

On ZoL:
( Some of these parameters are redundant, but they're what I used when I reproduced it)

• zpool create -d -o ashift=12 -o feature@lz4_compress=enabled -O acltype=posixacl -O canmount=on -O compression=lz4 -O devices=on -O normalization=formD -O xattr=on -O atime=on -O relatime=off testpool disk1
  On ZoF:
• create one or more datasets backing /usr/src and /usr/obj (/usr/obj is the one which this bug didn't happen unless I did, /usr/src may be optional.)
• make buildworld buildkernel
• make installkernel
• reboot
• make installworld, hopefully error out with exec format error (I lucked out and it errored out on /rescue/rescue with exec format error, the original reporter @javashin had make installworld not error out, but then break on next reboot)

Include any warning/errors/backtraces from the system logs

No obvious errors except the output from make installworld,

One thing I should mention - the reason I wanted to reproduce this was to see what kind of files resulted in the "Exec format error" - the usr/obj/.../rescue/rescue that was the reason make installworld failed for me was a 40 MB file of all zeroes.

The original reporter could reproduce this every time with ZoF and not FreeBSD-builtin ZFS; I'm currently running make buildworld to try and reproduce it again.

I can provide VM disk images of the machine and the zpool doing this, and I don't think it's somehow a fluke of my environment.

[rincebrain]

Footnote: this reminds me of openzfs#8816 .

[javashin]

Hello nice to know that someone (PMT, on #zfsonlinux freenode) has reproduced this problem i encounter long ago now it can be resolved in the mean time this can be avoided fetching last sources svn or github then recompile make buildkernel installkernel ---> reboot on new kernel rm -rf /usr/obj/* ; cd /usr/src ; make -j9 clean ---> recompile world make buildworld installworld just not rebooting installing world just after compile the new world that lives in /usr/obj then make make buildkernel installkernel meta-mode and ccache can be used to make things faster and =
WITHOUT_MAKE_CHECK_USE_SANDBOX="YES"
WITH_INSTALL_AS_USER="YES"
on src.conf and off course the Shims To Remove Zfs legacy code from opensolaris leaving the loader able to boot zfs . optinal not requiered but a good idea is go to /usr/src/release and make -j9 memstick it will fail but creates src and ports dists very fast , ok now got to the release dir in /usr/obj and copy the generated files in that dir to the dir called dists in the same location go back to /usr/src/release and make cdrom without specify any make jobs this will create bootonly.iso and disc1.iso and then after that back to the memstck ---> make memstick = dito seems like to be good idea to use it in the future and dont depend on trueos iso snapshot to create the pool with openzfs on the livecd installer after kldunload zfs opensolaris and load openzfs kmod and ran bsdinstall which is modded to be used with openzfs and install a system without legacy zfs from the same installer it will create the openzfs pool and install on it if you select the base/kernel without zfs progs kmod opensolaris ETC.

Other way to install is my own way that i use all the time here >>>> https://github.com/javashin/DiabloOS STEP#1 ..... thanks in advance would be nice to have openzfskmod inside the kernel built-in and not as module i think is the time :) nice work people the last update on ports works great :)

i tried to warm about this on #129 (comment) here

if needs to test things faster with the procedure of reproduce this compiling with WITHOUT_BOOT="YES"
WITHOUT_LIB32="YES"
just dont make delete-old boot to be able to reboot and enter in single user mode and try to install and make check will fail and prevent from installing anything because the binaries are borked "Exec Format Error"

and just compiling stand after the world build.

To Reproduce the problem

WITHOUT_MAKE_CHECK_USE_SANDBOX="YES"
WITH_INSTALL_AS_USER="YES"

Shims are not needed. just for the workaround on top.

[mattmacy]

Is this still a problem? We briefly had a problem with executables

[mattmacy]

I haven't seen any followup, so closing