lua: avoid gcc -Wreturn-local-addr bug

[rlibby]

Avoid a bug with gcc's -Wreturn-local-addr warning with some
obfuscation. In buggy versions of gcc, if a return value is an
expression that involves the address of a local variable, and even if
that address is legally converted to a non-pointer type, a warning may
be emitted and the value of the address may be replaced with zero.
Howerver, buggy versions don't emit the warning or replace the value
when simply returning a local variable of non-pointer type.

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90737

Signed-off-by: Ryan Libby rlibby@FreeBSD.org

Motivation and Context

Part of a small set of changes for building with gcc on FreeBSD.

This change addresses this warning, which also leads to a code generation error.

/usr/src/freebsd/sys/contrib/openzfs/module/lua/ldo.c: In function 'stack_remaining':
/usr/src/freebsd/sys/contrib/openzfs/module/lua/ldo.c:43:10: error: function returns address of local variable [-Werror=return-local-addr]
   return (intptr_t)(&local - (char *)curthread->td_kstack);
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Description

Avoid -Wreturn-local-addr by reformulating the computation in a way that gcc doesn't recognize as returning the address of a local.

This is admittedly ugly, as there is nothing wrong with the previous formulation, and without examining gcc internals it's not obvious why this formulation is any less likely to encounter the bug. However, it is a simple solution to the problem.

An alternative solution which may be more clear may be to define stack_remaining() as taking a parameter representing the amount of stack space needed and then returning a boolean, rather than returning the available space remaining. On the other hand, changing the function prototype would cause slightly more code churn.

Note that -Wno-return-local-addr is not a solution: that suppresses the warning but still leads to wrong code generation.

How Has This Been Tested?

Compile tested in the downstream FreeBSD build with gcc and clang.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[behlendorf]

Thanks for linking to the original gcc bug in the PR and commit comment, that was very helpful. It's good to see this has been fixed in gcc 9.2, and 10. Since we can avoid this with a little minor code restructuring on these buggy versions that does seem preferable.