No Control-flow Enforcement Technology support #6

lgv5 · 2023-09-10T03:07:46Z

Much like @omar-polo at taisei-project/taisei#372 , I'm trying Taisei 1.4 in OpenBSD. Much like Omar, the game crashes at the beginning of the first level, but for a different reason: OpenBSD has CET enabled by default in -current and my machine does support it (11th gen Intel).

At the port level, I worked it around by removing CET enforcement, but it would be nice if this library could work without it. Boost has some workarounds for it which I tried to apply at least for my arch, make_x86_64_sysv_elf_gas.S, which is adding the _CET_ENDBR macro at the beginning of each function. That isn't enough to solve it for me, as then I run into another issue:

Program terminated with signal SIGILL, Illegal instruction.
#0  koishi_fiber_swap (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:43
43              from->fctx = tf.fctx;
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (process 270966)]
(gdb) bt
#0  koishi_fiber_swap (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:43
#1  koishi_swap_coroutine (from=<optimized out>, to=<optimized out>, state=3) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:46
#2  koishi_resume (co=<optimized out>, arg=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:90
#3  cotask_resume_internal (task=0x23f89a860e0, arg=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:237
#4  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:39
#5  0x0000023d66f98876 in stage1_bg_init_fullstage () at ../taisei-1.4/src/stages/stage1/background_anim.c:72
#6  stage1_start () at ../taisei-1.4/src/stages/stage1/stage1.c:71
#7  0x0000023d66ed8a2a in COTASK_stage_comain (_cotask_args=<optimized out>) at ../taisei-1.4/src/stage.c:1018
#8  0x0000023d66ed60cc in COTASKTHUNK_stage_comain (arg=<optimized out>, arg_size=<optimized out>) at ../taisei-1.4/src/stage.c:1014
#9  0x0000023d66ef6a1b in cotask_entry (varg=0x723cefc249b0) at ../taisei-1.4/src/coroutine/cotask.c:371
#10 0x0000023d67048207 in koishi_entry (co=0x23fb5d59160) at ../taisei-1.4/subprojects/koishi/src/fcontext/../fiber.h:68
#11 0x0000023d670481ea in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/fcontext/fcontext.c:50
#12 0x0000023d66ea0dd7 in make_fcontext () at ../taisei-1.4/subprojects/koishi/src/fcontext/asm/make_x86_64_sysv_elf_gas.S:78
Backtrace stopped: Cannot access memory at address 0x23ffc717000

I have the core file around if it helps, and I'll gladly test patches.

The text was updated successfully, but these errors were encountered:

Akaricchi · 2023-09-10T03:39:48Z

Boost has some workarounds for it

Koishi is able to link to boost.context for the fcontext functions, so you can try that first and it might just work. In Taisei build directory: meson configure -Dkoishi:impl=boost_fcontext.

You can also try the ucontext and ucontext_sjlj backends. ucontext is a deprecated POSIX API that's infamous for being usually quite slow, but should be a safe choice if your libc provides it. ucontext_sjlj is a hack that uses a combination of ucontext and longjmp to speed up the context switching; it relies on undefined behavior and thus may or may not work.

If your libc lacks ucontext, you can try https://github.com/kaniini/libucontext, but Koishi will have to be lightly massaged to work with it, because the function names are different.

I'll look into backporting the CET stuff from boost later, but I can't promise anything, as I don't have any hardware to adequately test it on.

Akaricchi · 2023-09-10T03:42:40Z

Naturally, the MAP_STACK patch is also required on OpenBSD, CET or not. It's been merged now, and the koishi submodule has been updated on Taisei's v1.4.x and master branches.

Akaricchi · 2023-09-10T03:44:40Z

make_x86_64_sysv_elf_gas.S

Did you apply it to jump_x86_64_sysv_elf_gas.S as well? (ontop can be left alone as it's currently unused).

lgv5 · 2023-09-10T12:40:15Z

Did you apply it to jump_x86_64_sysv_elf_gas.S as well? (ontop can be left alone as it's currently unused).

No, I only made the changes in jump as it was the one triggering a SIGILL. I now applied boostorg/context@b62a991 and I get the same backtrace as shown in the fisrt message.

Naturally, the MAP_STACK patch is also required on OpenBSD, CET or not. It's been merged now, and the koishi submodule has been updated on Taisei's v1.4.x and master branches.

Yes, sorry if I wasn't clear. Omar shared an update to the port that includes his patch. I hit the CET protection with that patch applied.

Koishi is able to link to boost.context for the fcontext functions, so you can try that first and it might just work. In Taisei build directory: meson configure -Dkoishi:impl=boost_fcontext.

I'll give this a shot next.

Akaricchi · 2023-09-10T20:27:19Z

I think you also need this for proper CET support boostorg/context@3484575

Which would require some C code changes to manage a shadow stack per fiber, so switching to a different backend probably won't help.

lgv5 · 2023-09-14T19:22:41Z

You're right, @Akaricchi . Even with the boost_fcontext I still run into issues. ucontext based things are out of the question, as OpenBSD's libc doesn't provide it.

Thread 1 received signal SIGSEGV, Segmentation fault.
0x00000117b2e17346 in cotask_entry (varg=0x746919a55f00) at ../taisei-1.4/src/coroutine/cotask.c:366
366             CoTaskData data = { 0 };
(gdb) bt
#0  0x00000117b2e17346 in cotask_entry (varg=0x746919a55f00) at ../taisei-1.4/src/coroutine/cotask.c:366
#1  0x00000117b2f74517 in koishi_entry (co=0x11a374137f0) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:68
#2  0x00000117b2f744f9 in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:50
#3  0x00000117b2dc0ff7 in make_fcontext ()
Backtrace stopped: Cannot access memory at address 0x11a491f0000

lgv5 · 2023-09-14T20:47:28Z

Sorry, that was a botched build (forgot to include the MAP_STACK patch). With the build fixed, this is the backtrace I get, which seems to indicate that it still runs into CET (because of the SIGILL vs the previous SIGSEGV):

Thread 1 received signal SIGILL, Illegal instruction.
_ZL17koishi_fiber_swapP14fcontext_fiberS0_ (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:43
43              from->fctx = tf.fctx;
(gdb) bt
#0  _ZL17koishi_fiber_swapP14fcontext_fiberS0_ (from=<optimized out>, to=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:43
#1  _ZL21koishi_swap_coroutineP16koishi_coroutineS0_i (from=0x2b55fdee8d0, to=0x2b59dfbd240, state=3)
    at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:46
#2  koishi_resume (co=0x2b59dfbd240, arg=<optimized out>) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:90
#3  cotask_resume_internal (task=0x2b59dfbd230, arg=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:237
#4  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:39
#5  0x000002b2e4ff770b in stage1_bg_init_fullstage () at ../taisei-1.4/src/stages/stage1/background_anim.c:72
#6  stage1_start () at ../taisei-1.4/src/stages/stage1/stage1.c:71
#7  0x000002b2e4f343fa in COTASK_stage_comain (_cotask_args=<optimized out>) at ../taisei-1.4/src/stage.c:1018
#8  0x000002b2e4f31a9c in COTASKTHUNK_stage_comain (arg=<optimized out>, arg_size=<optimized out>) at ../taisei-1.4/src/stage.c:1014
#9  0x000002b2e4f523eb in cotask_entry (varg=0x7096e967ea30) at ../taisei-1.4/src/coroutine/cotask.c:371
#10 0x000002b2e50af517 in koishi_entry (co=0x2b55fdee8d0) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/../fiber.h:68
#11 0x000002b2e50af4f9 in co_entry (tf=...) at ../taisei-1.4/subprojects/koishi/src/boost_fcontext/../fcontext/fcontext.c:50
#12 0x000002b2e4efbff7 in make_fcontext ()
Backtrace stopped: Cannot access memory at address 0x2b52ee89000

Akaricchi · 2023-09-15T02:55:49Z

I've been working on a new koishi backend for the past 2 days, based on boost.context's high-level callcc API. I believe it should work with CET, though I can't test that myself. It's in a mostly working state now. I'll post a PR for testing tomorrow after cleaning it up.

Akaricchi · 2023-09-16T02:35:29Z

@lgv5 please see if the new boost_callcc backend in #7 works for you.

lgv5 · 2023-09-16T08:50:46Z

@Akaricchi sadly, still no luck:

Program terminated with signal SIGILL, Illegal instruction.
#0  koishi_fiber_init_callcc (fiber=0x10244a49d0) at /usr/local/include/boost/context/continuation_fcontext.hpp:261
261             return { detail::jump_fcontext(
--Type <RET> for more, q to quit, c to continue without paging--
[Current thread is 1 (process 269413)]
(gdb) bt
#0  koishi_fiber_init_callcc (fiber=0x10244a49d0) at /usr/local/include/boost/context/continuation_fcontext.hpp:261
#1  0x0000000dc33786a6 in _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, 
    debug=...) at ../taisei-1.4/subprojects/koishi/src/boost_callcc/boost_callcc.cc:57
#2  0x0000000dc3357e29 in _stage_enter (stage=<optimized out>, rg=<optimized out>, next=..., quickload=<optimized out>, quicksave_is_automatic=false)
    at ../taisei-1.4/src/stage.c:1177
#3  0x0000000dc33978d8 in menu_logic_frame (arg=0x100e17eaa0) at ../taisei-1.4/src/menu/menu.c:213
#4  0x0000000dc338c5fc in run_logic_frame (frame=0xdc34eaa58 <evloop+96>) at ../taisei-1.4/src/eventloop/eventloop.c:70
#5  handle_logic (pframe=<optimized out>, ftimes=<optimized out>) at ../taisei-1.4/src/eventloop/eventloop.c:89
#6  0x0000000dc3342ab8 in eventloop_run () at ../taisei-1.4/src/eventloop/executor_synchro.c:89
#7  0x0000000dc3342ab8 in main_post_vfsinit (ccr=...)
#8  0x0000000dc333e57e in main (argc=1, argv=0x747011a1aa18) at ../taisei-1.4/src/cli.c:306

Akaricchi · 2023-09-16T08:53:47Z

@Akaricchi sadly, still no luck:

Please attach the build log (with -v). I wanna take a look at the compile flags.

lgv5 · 2023-09-16T11:30:09Z

Here you go https://gist.github.com/lgv5/725db362d823586f9d1fcd0e483f4f8d . I think the line steps you're interested into are 20 and 23, starting at https://gist.github.com/lgv5/725db362d823586f9d1fcd0e483f4f8d#file-make-log-txt-L411 .

Akaricchi · 2023-09-17T09:31:14Z

I think I need to see the preprocessed version of boost_callcc.cc. In the build directory, run:

c++ -Isubprojects/koishi/src/libkoishi.a.p -I../taisei-1.4/subprojects/koishi/include -I/usr/local/include -fvisibility=hidden -flto -fcolor-diagnostics -DNDEBUG -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wextra -Wpedantic -O2 -pipe -g -fPIC -DBOOST_ALL_NO_LIB -DKOISHI_THREAD_LOCAL= -DKOISHI_HAVE_MMAP -DKOISHI_MAP_ANONYMOUS=MAP_ANONYMOUS -DKOISHI_HAVE_SYSCONF -DKOISHI_SC_PAGE_SIZE=_SC_PAGE_SIZE -DKOISHI_HAVE_GETPAGESIZE -DKOISHI_HAVE_ALIGNED_ALLOC -DKOISHI_HAVE_POSIX_MEMALIGN -DKOISHI_SJLJ_SIG -DBUILDING_KOISHI -D_BSD_SOURCE -D_DARWIN_C_SOURCE -D_DEFAULT_SOURCE -D_GNU_SOURCE -D_POSIX_C_SOURCE=200809L -D_XOPEN_SOURCE=700 -Wall -Wpedantic -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -Wmissing-prototypes -Wstrict-prototypes -o /tmp/boost_callcc.ii -E ../taisei-1.4/subprojects/koishi/src/boost_callcc/boost_callcc.cc

It should output into /tmp/boost_callcc.ii; please attach that file. Note that it might be huge (several megabytes).

lgv5 · 2023-09-17T09:54:32Z

https://gist.github.com/lgv5/e98628f6ca85420d82fa2720b0ac47d7

Akaricchi · 2023-09-17T10:20:08Z

See if meson configure -Dcpp_args=-DSHADOW_STACK_SYSCALL=1 changes anything

lgv5 · 2023-09-17T13:17:26Z

Slightly different backtrace. OpenBSD currently ships Boost 1.80.0, and the whole SHADOW_STACK_SYSCALL was introduced in 1.81.0, so I'm not super sure where the change comes from.

Thread 1 received signal SIGILL, Illegal instruction.
0x00000c2eb1c11c13 in void* boost::context::detail::create_context1<boost::context::detail::record<boost::context::continuation, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
(gdb) bt
#0  0x00000c2eb1c11c13 in void* boost::context::detail::create_context1<boost::context::detail::record<boost::context::continuation, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>, fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
#1  0x00000c2eb1c118c8 in boost::context::continuation boost::context::callcc<fake_allocator, koishi_fiber_init_callcc(boost_fiber*)::$_0>(std::__1::allocator_arg_t, fake_allocator&&, koishi_fiber_init_callcc(boost_fiber*)::$_0&&) ()
#2  0x00000c2eb1c11760 in koishi_fiber_init_callcc(boost_fiber*) ()
#3  0x00000c2eb1c116bf in koishi_fiber_init(boost_fiber*, unsigned long) ()
#4  0x00000c2eb1c11665 in koishi_init ()
#5  0x00000c2eb1ac71a5 in cotask_new_internal (entry_point=<optimized out>) at ../taisei-1.4/src/coroutine/cotask.c:210
#6  _cosched_new_task (sched=<optimized out>, func=<optimized out>, arg=<optimized out>, arg_size=<optimized out>, is_subtask=<optimized out>, debug=...)
    at ../taisei-1.4/src/coroutine/cosched.c:19
#7  0x00000c2eb1aa69e9 in _stage_enter (stage=<optimized out>, rg=<optimized out>, next=..., quickload=<optimized out>, quicksave_is_automatic=false)
    at ../taisei-1.4/src/stage.c:1177
#8  0x00000c2eb1ae59b8 in menu_logic_frame (arg=0xc30bbe551e0) at ../taisei-1.4/src/menu/menu.c:213
#9  0x00000c2eb1ada71c in run_logic_frame (frame=0xc2eb1c43658 <evloop+96>) at ../taisei-1.4/src/eventloop/eventloop.c:70
#10 handle_logic (pframe=<optimized out>, ftimes=<optimized out>) at ../taisei-1.4/src/eventloop/eventloop.c:89
#11 0x00000c2eb1a91748 in eventloop_run () at ../taisei-1.4/src/eventloop/executor_synchro.c:89
#12 0x00000c2eb1a91748 in main_post_vfsinit (ccr=...)
#13 0x00000c2eb1a8d20e in main (argc=1, argv=0x78947d004a38) at ../taisei-1.4/src/cli.c:306

Akaricchi · 2023-09-17T20:12:25Z

You have to update it then. Shadow stack support is the whole point.

lgv5 · 2024-11-05T17:42:45Z

Long time no see, @Akaricchi ! I got back to this. It turns out I was missing a single endbr64 right after the call to jump_fcontext at https://github.com/taisei-project/koishi/blob/master/src/fcontext/fcontext.c#L69 . It's needed in here because the very last instruction of jump_fcontext is an indirect branch, https://github.com/taisei-project/koishi/blob/master/src/fcontext/asm/jump_x86_64_sysv_elf_gas.S#L87 . With that endbr64 in place, I can run Taisei in my IBT-enabled CPU without triggering a SIGILL. Please check https://marc.info/?l=openbsd-ports&m=173082527119567&w=2 for the addition of _CET_ENDBR in the assembler files too. I believe that the patches can be safely applied to this repo as-is, but I'm not able to test it on a 32-bits kernel.

Akaricchi · 2024-11-07T07:17:00Z

Interesting find, thanks for looking into this. I have just pushed an update-fcontext branch that updates the asm files to a more recent version from boost-context. They contain the asm-side _CET_ENDBR changes. Could you please rebase your patch on top of that branch, test is, and open a pull request?

Also, is there a reason why the extra endbr instruction on the C side needs to be there instead of at the end of jump_x86_64_sysv_elf_gas.S? Boost-context seems to do neither. I wonder why.

Akaricchi · 2024-11-07T07:19:44Z

It's also interesting that this works without shadow stack support. Does OpenBSD not enable that part of CET?

lgv5 · 2024-11-07T11:40:30Z

Indeed, OpenBSD doesn't support shadow stacks. We only support the BTI bits.

_CET_ENDBR needs to be the first instruction after a call or indirect jump. Placing it after the indirect jump jmp *%r8 won't help here. My guess is that koishi_fiber_swap is executed in a fiber itself, which yield after a jump_fcontext and eventually needs to return to execution, but those returns are done via indirect jumps instead of a ret. This also explains "Boost-context seems to do neither. I wonder why.": _CET_ENDBR must be placed after the jump target.

I created PR #8 adding the macro definitions and _CET_ENDBR on the C++ side. It's on top of update-fcontext branch.

Akaricchi · 2024-11-07T12:23:17Z

My guess is that koishi_fiber_swap is executed in a fiber itself, which yield after a jump_fcontext and eventually needs to return to execution, but those returns are done via indirect jumps instead of a ret.

Basically koishi_fiber_swap is the only way out of a fiber. If a fiber needs to return/yield, it has to call koishi_fiber_swap with the previous context. src/fiber.h implements the asymmetrical resume/yield interface on top of the symmetrical fibers, where swap is the only context-switching operation.

This also explains "Boost-context seems to do neither. I wonder why.": _CET_ENDBR must be placed after the jump target.

It makes sense why koishi needs it. But jump_fcontext is always an indirect jump, so I'm confused about how boost.context handles the same scenario… my head hurts trying to follow this.

lgv5 · 2024-11-07T16:05:01Z

Can you point me to the boost.context bits?

Akaricchi · 2024-11-07T16:32:49Z

There are a few calls to jump_fcontext but I believe this is the important one: https://github.com/boostorg/context/blob/46effe4588a6edd64d49a853454b5f67f7196e57/include/boost/context/continuation_fcontext.hpp#L316

Trying to comprehend C++ code is giving me an aneurysm.

Akaricchi · 2024-11-07T16:34:07Z

Also, closing this as completed by #8, though we can continue the discussion here.

If we ever need shadow stack support, it's probably better to create a separate issue.

lgv5 changed the title ~~No Control-flow Enforcement Technology supporthttps://github.com/taisei-project/taisei/issues/372~~ No Control-flow Enforcement Technology support Sep 10, 2023

Akaricchi mentioned this issue Sep 16, 2023

Add a boost.context call/cc backend #7

Open

Akaricchi closed this as completed Nov 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No Control-flow Enforcement Technology support #6

No Control-flow Enforcement Technology support #6

lgv5 commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

lgv5 commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

lgv5 commented Sep 14, 2023

lgv5 commented Sep 14, 2023

Akaricchi commented Sep 15, 2023

Akaricchi commented Sep 16, 2023

lgv5 commented Sep 16, 2023

Akaricchi commented Sep 16, 2023

lgv5 commented Sep 16, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Sep 17, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Sep 17, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Nov 5, 2024

Akaricchi commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

lgv5 commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

lgv5 commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

No Control-flow Enforcement Technology support #6

No Control-flow Enforcement Technology support #6

Comments

lgv5 commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

lgv5 commented Sep 10, 2023

Akaricchi commented Sep 10, 2023

lgv5 commented Sep 14, 2023

lgv5 commented Sep 14, 2023

Akaricchi commented Sep 15, 2023

Akaricchi commented Sep 16, 2023

lgv5 commented Sep 16, 2023

Akaricchi commented Sep 16, 2023

lgv5 commented Sep 16, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Sep 17, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Sep 17, 2023

Akaricchi commented Sep 17, 2023

lgv5 commented Nov 5, 2024

Akaricchi commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

lgv5 commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

lgv5 commented Nov 7, 2024

Akaricchi commented Nov 7, 2024

Akaricchi commented Nov 7, 2024