Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to compile on ARM64 with clang #13260

Closed
derzahla opened this issue Mar 28, 2022 · 8 comments
Closed

Unable to compile on ARM64 with clang #13260

derzahla opened this issue Mar 28, 2022 · 8 comments
Labels
Type: Building Indicates an issue related to building binaries Type: Defect Incorrect behavior (e.g. crash, hang)

Comments

@derzahla
Copy link

derzahla commented Mar 28, 2022

System information

Type Lenovo c630 w/Snapdragon sdm850 cortex.a75-cortex.a55
Distribution Name Arch
Distribution Version Current w/ clang 13.0.1
Kernel Version 5.17.0
Architecture ARM64
OpenZFS Version 2.1.4 and latest git

Describe the problem you're observing

While building under GCC works fine, I am unable to with clang+lto or -lto

Describe how to reproduce the problem

Compile linux kernel with ZFS +clang or compile ZFS as a dkms module for a kernel compiled with clang

Include any warning/errors/backtraces from the system logs

LTO [M] drivers/bluetooth/btrsi.lto.o

LTO [M] drivers/target/target_core_user.lto.o
GEN .version
CHK include/generated/compile.h
GEN .tmp_initcalls.lds
GEN .tmp_symversions.lds
LTO vmlinux.o
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2150099659
ld.lld: error: couldn't allocate output register for constraint '{v7}' at line 2150100117
ld.lld: error: couldn't allocate input reg for constraint '{v1}' at line 2150100832
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2150101251
ld.lld: error: couldn't allocate output register for constraint '{v7}' at line 2150101709
ld.lld: error: couldn't allocate input reg for constraint '{v1}' at line 2150102484
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157696494
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157700699
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157704887
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157709206
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157714137
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157718730
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157724053
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157728241
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157732948
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157738385
ld.lld: error: couldn't allocate output register for constraint '{v16}' at line 2157781218
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157782350
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157786517
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157789877
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
make: *** [Makefile:1156: vmlinux] Error 1
==> ERROR: A failure occurred in build().
Aborting...

More detail is obtained when not using LTO, here is output without LTO and using with "make v=1" :

CC fs/proc/inode.o
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:151:2: error: couldn't allocate output register for constraint '{v0}'
NEON_INIT_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:75:6: note: expanded from macro 'NEON_INIT_LOOP'
asm("eor %[ZERO].16b,%[ZERO].16b,%[ZERO].16b\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:154:3: error: couldn't allocate output register for constraint '{v7}'
NEON_MAIN_LOOP(NEON_DONT_REVERSE);
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:93:6: note: expanded from macro 'NEON_MAIN_LOOP'
asm("ld1 { %[SRC].4s }, %[IP]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:157:2: error: couldn't allocate input reg for constraint '{v1}'
NEON_FINI_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:112:6: note: expanded from macro 'NEON_FINI_LOOP'
asm("st1 { %[ACC0].4s },%[DST0]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:190:2: error: couldn't allocate output register for constraint '{v0}'
NEON_INIT_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:75:6: note: expanded from macro 'NEON_INIT_LOOP'
asm("eor %[ZERO].16b,%[ZERO].16b,%[ZERO].16b\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:193:3: error: couldn't allocate output register for constraint '{v7}'
NEON_MAIN_LOOP(NEON_DO_REVERSE);
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:93:6: note: expanded from macro 'NEON_MAIN_LOOP'
asm("ld1 { %[SRC].4s }, %[IP]\n"
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:196:2: error: couldn't allocate input reg for constraint '{v1}'
NEON_FINI_LOOP();
^
fs/zfs/zcommon/zfs_fletcher_aarch64_neon.c:112:6: note: expanded from macro 'NEON_FINI_LOOP'
asm("st1 { %[ACC0].4s },%[DST0]\n"
^
6 errors generated.
make[3]: *** [scripts/Makefile.build:288: fs/zfs/zcommon/zfs_fletcher_aarch64_neon.o] Error 1
make[2]: *** [scripts/Makefile.build:550: fs/zfs/zcommon] Error 2
make[1]: *** [scripts/Makefile.build:550: fs/zfs] Error 2
make[1]: *** Waiting for unfinished jobs....


@derzahla derzahla added the Type: Defect Incorrect behavior (e.g. crash, hang) label Mar 28, 2022
@rincebrain
Copy link
Contributor

rincebrain commented Mar 28, 2022

The problem is that, while GCC is perfectly happy with explicit register allocations when it's been told not to do certain kinds of instructions with compile flags, clang is not, and passing -mgeneral-regs-only (as Linux does generally on AArch64, among others) makes the magic smoke come out.

CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only in the Makefile will make that stop, for instance, but I think technically it could choose to emit FPU instructions outside of the kfpu_begin/end bounded regions if you do that without further changes, even though I can't imagine a reason it would decide it was beneficial here.

@derzahla
Copy link
Author

Thanks for the quick response. That is what was suggested on IRC. I did a 'make clean' and placed the line (direct copy and paste) "CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only" below the 4th line of the fs/zfs/Makefile and recompiled. It didnt seem to make any difference

@rincebrain
Copy link
Contributor

rincebrain commented Mar 28, 2022

It's true, I did suggest it there too.

That said, for builtin, it's possible it wants a path like fs/zfs/... - I don't know how the way it calculates paths interacts when you do it as builtin.

As far as as a module, it should work, removing the flag did work for me, and adding that line in the Makefile did make the flag, and that error set go away (though it promptly produced the same class of problem with the NEON raidz instructions, but the same dance over there worked too).

@derzahla
Copy link
Author

derzahla commented Mar 29, 2022

Thanks! It compiles as a module with after taking care of the two raidz neon instructions. Any change adding those two would also allow it to compile builtin?

@behlendorf behlendorf added the Type: Building Indicates an issue related to building binaries label Mar 29, 2022
@derzahla
Copy link
Author

derzahla commented Apr 4, 2022

Update: So compiling completes but the system does not boot off its ZFS root. Either when compiled as a module or built in

@derzahla
Copy link
Author

Coming back to this, I'm not sure where to add the 3 lines anymore, seems the code structure has changed? I tried adding:

CFLAGS_REMOVE_zfs_fletcher_aarch64_neon.o += -mgeneral-regs-only
CFLAGS_REMOVE_vdev_raidz_math_aarch64_neon.o += -mgeneral-regs-only
CFLAGS_REMOVE_vdev_raidz_math_aarch64_neonx2.o += -mgeneral-regs-only

to the three Makefile.in's that exist in master:

./include/Makefile.in
./module/Makefile.in
./Makefile.in

But build errors out the same as without the fix. Can you tell me what Im doing wrong, note: I am building as a module this time, not built in. output from dkms build:

 CC [M]  /var/lib/dkms/zfs/git/build/module/os/linux/zfs/zvol_os.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zcommon/zfs_fletcher_aarch64_neon.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neon.o
  CC [M]  /var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neonx2.o
  LD [M]  /var/lib/dkms/zfs/git/build/module/spl.o
/var/lib/dkms/zfs/git/build/module/os/linux/zfs/zvol_os.c:1114:3: warning: ignoring return value of function declared with 'warn_unused_result' attribute [-Wunused-result]
                add_disk(zv->zv_zso->zvo_disk);
                ^~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
1 warning generated.
/var/lib/dkms/zfs/git/build/module/zfs/vdev_raidz_math_aarch64_neonx2.c:215:24: warning: pragma diagnostic pop could not pop, no matching push [-Wunknown-pragmas]
#pragma GCC diagnostic pop
                       ^
1 warning generated.
  LD [M]  /var/lib/dkms/zfs/git/build/module/zfs.o
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157731689
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157735894
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157740082
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157744401
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157749332
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157753925
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157759248
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157763436
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157768143
ld.lld: error: couldn't allocate input reg for constraint '{v0}' at line 2157773580
ld.lld: error: couldn't allocate output register for constraint '{v16}' at line 2157812348
ld.lld: error: couldn't allocate output register for constraint '{v0}' at line 2157813480
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157817647
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157821007
ld.lld: error: couldn't allocate input reg for constraint '{v4}' at line 2157829425
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157833708
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157837068
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157840959
ld.lld: error: couldn't allocate input reg for constraint '{v4}' at line 2157845316
ld.lld: error: couldn't allocate output register for constraint '{v4}' at line 2157849552
ld.lld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
make[4]: *** [scripts/Makefile.build:418: /var/lib/dkms/zfs/git/build/module/zfs.o] Error 1
make[3]: *** [Makefile:1844: /var/lib/dkms/zfs/git/build/module] Error 2
make[3]: Leaving directory '/usr/lib/modules/5.18.0-asahi-20220608-1-ARCH/build'

@rincebrain
Copy link
Contributor

Haven't tested this, but you probably want lines like CFLAGS_REMOVE_foo.o and the corresponding CFLAGS_foo.o in module/Kbuild.in for Linux kernel land (though note that they actually need to be in Kbuild, in the end, so don't forget to run configure to regenerate Kbuild after editing Kbuild.in).

@ardbiesheuvel
Copy link

Just a drive by comment: if permitted, the compiler may use NEON registers for spilling or for inline memsets and memcpys, e.g., to initialize stack variables. If this occurs outside of kernel_neon_begin()/end(), the kernel code will be clobbering user space data, because (unlike on 32-bit ARM), we don't disable access to the FP/SIMD registers when entering the kernel, and while we preserve them eagerly (as the task may be scheduled out and migrate to another CPU), the restore is only done lazily, but only when/after kernel_neon_begin()/end() have been used.

szubersk added a commit to szubersk/zfs that referenced this issue Nov 12, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
szubersk added a commit to szubersk/zfs that referenced this issue Nov 12, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
szubersk added a commit to szubersk/zfs that referenced this issue Nov 13, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
(cherry picked from commit 8dbeb67)
szubersk added a commit to szubersk/zfs that referenced this issue Nov 13, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
(cherry picked from commit 8dbeb67)
szubersk added a commit to szubersk/zfs that referenced this issue Nov 19, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
szubersk added a commit to szubersk/zfs that referenced this issue Nov 27, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
szubersk added a commit to szubersk/zfs that referenced this issue Nov 29, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
szubersk added a commit to szubersk/zfs that referenced this issue Nov 30, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Signed-off-by: szubersk <szuberskidamian@gmail.com>
andrewc12 pushed a commit to andrewc12/openzfs that referenced this issue Dec 16, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150
andrewc12 pushed a commit to andrewc12/openzfs that referenced this issue Dec 17, 2022
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150
rincebrain pushed a commit to rincebrain/zfs that referenced this issue Jun 1, 2023
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Porting notes:
- Moved the stanzas removing -mgeneral-regs-only to Makefile.in
  since they wouldn't readily work in Kbuild.in and that did.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150
rincebrain pushed a commit to rincebrain/zfs that referenced this issue Jun 1, 2023
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Porting notes:
- Moved the stanzas removing -mgeneral-regs-only to Makefile.in
  since they wouldn't readily work in Kbuild.in and that did.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150

Closes openzfs#14624
Ported-by: Rich Ercolani <rincebrain@gmail.com
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
rincebrain pushed a commit to rincebrain/zfs that referenced this issue Jun 1, 2023
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Porting notes:
- Moved the stanzas removing -mgeneral-regs-only to Makefile.in
  since they wouldn't readily work in Kbuild.in and that did.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150

Closes openzfs#14624
Ported-by: Rich Ercolani <rincebrain@gmail.com
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
behlendorf pushed a commit that referenced this issue Jun 5, 2023
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix #13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Porting notes:
- Moved the stanzas removing -mgeneral-regs-only to Makefile.in
  since they wouldn't readily work in Kbuild.in and that did.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes #13260
Closes #14150

Closes #14624
Ported-by: Rich Ercolani <rincebrain@gmail.com
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
tonyhutter pushed a commit to tonyhutter/zfs that referenced this issue Jun 6, 2023
- Clang 15 doesn't support `-fno-ipa-sra` anymore. Do a separate
  check for `-fno-ipa-sra` support by $KERNEL_CC.

- Don't enable `-mgeneral-regs-only` for certain module files.
  Fix openzfs#13260

- Scope `GCC diagnostic ignored` statements to GCC only. Clang
  doesn't need them to compile the code.

Porting notes:
- Moved the stanzas removing -mgeneral-regs-only to Makefile.in
  since they wouldn't readily work in Kbuild.in and that did.

Reviewed-by: Richard Yao <richard.yao@alumni.stonybrook.edu>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: szubersk <szuberskidamian@gmail.com>
Closes openzfs#13260
Closes openzfs#14150

Closes openzfs#14624
Ported-by: Rich Ercolani <rincebrain@gmail.com
Signed-off-by: Rich Ercolani <rincebrain@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Building Indicates an issue related to building binaries Type: Defect Incorrect behavior (e.g. crash, hang)
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants