Error: unsupported instruction `sysenter'

[vanhauser-thc]

Describe the bug
when I try to compile dynamorio it aborts with an error:

# make
-- Version number: 7.91.18289
-- WARNING: vera++ not found: disabling code style checks
-- Found Qt 5: DrGUI will be built
[...]
prg/tmp/dynamorio/core/arch/x86/x86.asm: Assembler messages:
/prg/tmp/dynamorio/core/arch/x86/x86.asm:805: Error: unsupported instruction `sysenter'
make[2]: *** [core/CMakeFiles/dynamorio_static.dir/build.make:822: core/CMakeFiles/dynamorio_static.dir/arch/x86/x86.asm.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:143: core/CMakeFiles/dynamorio_static.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

I thought this is make due a change in the code and reverted to an old version of dynamorio that previously compiled fine for me (dynamorio-cronbuild-7.90.18005 from April 2019) - but same issue there.

I also tried gcc-7 and gcc-9, same outcome.

My guess its about the kernel/libc/headers? I am running Linux 5.4.0-amd64 and using libc 2.29

To Reproduce

git clone https://github.com/DynamoRIO/dynamorio/
cd dynamorio
mkdir build
cd build
cmake ..
make

Versions
github checkout, but same for older versions e.g. 7.0 and 7.1 releases.

[johnfxgalea]

I had a quick look in x86.asm

GLOBAL_LABEL(global_do_syscall_sysenter:)
#if defined(X64) && defined(WINDOWS)
        syscall  /* FIXME ml64 won't take "sysenter" so half-fixing now */
#else
        sysenter
#endif

I haven't studied much of this code yet but maybe syscall should be used instead?

[vanhauser-thc]

Yes, switching this to syscall compiles. Not sure it then works like it should but basic functionality testing - its fine

For fixing this it needs more information why its not working, ifits a glibc version thing, kernel version thing ... or something else

[johnfxgalea]

Are you on AMD64 or intel?

[vanhauser-thc]

5.4.0-kali3-amd64
Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz

[derekbruening]

There are separate routines for separate system call gateway methods. global_do_syscall_sysenter is for executing SYSENTER, global_do_syscall_syscall is for executing SYSCALL, etc. DR observes what the app does, and then picks which method to use for syscalls that should look like the app. I believe that back when DR was ported to amd64 we were not 100% sure that no 64-bit OS would use SYSENTER and so we had to keep the code for it available in a 64-bit build of DR -- yet you can see that for Windows it was hacked to get it to compile. At this point we can say with reasonable certainty that SYSENTER is not going to be used in 64-bit apps, even for mixed-mode like in WOW64 (though there IIRC the WOW64 ntdll does have code for SYSENTER in one of its gateways -- maybe why we were worried about -- but AFAIK it is never executed), and so probably it can be ifdef-ed out. Or, just put in the raw bytes instead of the opcode to get it to assemble.

[derekbruening]

Best to use syscall not raw bytes b/c amd64 (unless in legacy mode) does not have sysenter.

Like I said, if DR is going to use SYSCALL it would use global_do_syscall_syscall. global_do_syscall_sysenter is for SYSENTER. It would likely be used only for something like WOW64 where it would be 32-bit mode at the time but it would be a 64-bit DR build (i.e., mixed mode).

[johnfxgalea]

Yea, I had got it.. hence deletion of the comment.

[johnfxgalea]

Thanks @derekbruening for the info. @vanhauser-thc do you want to make a PR to fix the issue? I do not mind doing it myself if you prefer.

[vanhauser-thc]

I would not mind doing a PR, however I cant do anything serious the next 7 days :-(

[johnfxgalea]

@vanhauser-thc okay, I'll do it then. Thanks for your report!

[vanhauser-thc]

thanks for fixing it! I am really surprise this did not bite anyone else and I was the first ...

[derekbruening]

thanks for fixing it! I am really surprise this did not bite anyone else and I was the first ...

You must have an unusual version of the assembler. No other toolchain in Travis or various users has hit this now or in the past. It is not clear how to reproduce it. Fortunately fixing it was easy.

[vanhauser-thc]

I am running Debian Testing --- gcc version 9.2.1 20200123 (Debian 9.2.1-25)
Thats why I was the first one to hit this I guess :)