PPU LLVM arm64+macOS port #12115

sguo35 · 2022-05-28T23:27:42Z

Fixes PPU LLVM recompiler to work for arm64 (only ppu_thread.elf works so far). Fixes virtual memory handling and W^X JIT restriction handling on macOS to work for Apple Silicon (lot of hacks 😢 ). Also fixes debug symbols on Mac.

Potentially breaking changes:

~~x86_pshufb renamed to pshufb~~
All calls out of JIT code to rpcs3 C++ code are marked non-tail now (as they aren't tail calls)

Build:

brew install molten-vk vulkan-headers sdl2 nasm qt@5 ninja cmake glew ffmpeg
export Qt5_DIR=$(brew --prefix)/opt/qt5
export VULKAN_SDK=$(brew --prefix)/opt/molten-vk
# taken from build-mac.sh
ln -s "$VULKAN_SDK/lib/libMoltenVK.dylib" "$VULKAN_SDK/lib/libvulkan.dylib"
export VK_ICD_FILENAMES=$VULKAN_SDK/share/vulkan/icd.d/MoltenVK_icd.json

git clone git@github.com:sguo35/rpcs3.git
cd rpcs3
git checkout arm64
git submodule update --init

# taken from build-mac.sh
sed -i '' "s/extern const double NSAppKitVersionNumber;/const double NSAppKitVersionNumber = 1343;/g" 3rdparty/hidapi/hidapi/mac/hid.c

mkdir build
cd build
cmake .. -DPNG_ARM_NEON=on -DUSE_ALSA=OFF -DUSE_PULSE=OFF -DUSE_AUDIOUNIT=ON -DUSE_NATIVE_INSTRUCTIONS=OFF -DUSE_SYSTEM_FFMPEG=on -DCMAKE_OSX_ARCHITECTURES="arm64" -DLLVM_TARGETS_TO_BUILD="X86;AArch64;ARM" -GNinja -DCMAKE_BUILD_TYPE=RelWithDebInfo -DUSE_SYSTEM_MVK=on
ninja && codesign --force --deep --sign - bin/rpcs3.app/Contents/MacOS/rpcs3

# Run it
./bin/rpcs3.app/Contents/MacOS/rpcs3

You may get some MoltenVK errors in build, in that case install VulkanSDK and run this (replace the 1.3.204.1 with your actual version number):

 export VULKAN_SDK=$HOME/VulkanSDK/1.3.204.1/macOS
 export VK_ICD_FILENAMES=$HOME/VulkanSDK/1.3.204.1/macOS/share/vulkan/icd.d/MoltenVK_icd.json
 export DYLD_LIBRARY_PATH="$VULKAN_SDK/lib:${DYLD_LIBRARY_PATH:-}"
 export VK_LAYER_PATH="$VULKAN_SDK/share/vulkan/explicit_layer.d"

Tested only on ppu_thread.elf, I tried with an actual game and something in SPU code will segfault. You may also get a PPU thread segfault the first time you run ppu_thread.elf after compiling, but it should go away if you run it again. I haven't been able to track down why yet.

Major hacks:

~~unmapping memory is done by just telling the OS to swap it out and marking it RW, because of the 16K page issue~~
mmap overwriting existing mmap'd memory with MAP_JIT tag is done by munmap and then mmap the same location, since Apple bans MAP_FIXED + MAP_JIT from overwriting existing mmap entries

Megamouse

Just looking at the code style

rpcs3/Emu/Cell/SPURecompiler.cpp

Megamouse · 2022-05-29T00:02:03Z

rpcs3/util/vm_native.cpp

@@ -135,7 +140,11 @@ namespace utils
 			size += 0x10000;
 		}

+#ifdef __APPLE__
+        auto ptr = ::mmap(use_addr, size, PROT_NONE, MAP_ANON | MAP_PRIVATE | MAP_JIT | c_map_noreserve, -1, 0);


indentation

I think you're using spaces instead of tabs, which is why it may appear as correct on your end

Weird that only these lines were wrong. I replaced with tabs it looks good now.

rpcs3/util/vm_native.cpp

rpcs3/Emu/Cell/PPUThread.cpp

rpcs3/Emu/Cell/PPUTranslator.cpp

rpcs3/util/vm_native.cpp

rpcs3/Emu/Memory/vm.cpp

Utilities/JIT.cpp

rpcs3/Emu/CPU/CPUTranslator.cpp

sguo35 · 2022-06-01T20:16:19Z

@Megamouse Not sure why GH is showing some of the indents, I checked in vim and Github and they are not there (see https://github.com/RPCS3/rpcs3/blob/91add25d9cc26ec74aa5eff785d8776197759b43/rpcs3/util/vm_native.cpp for example). Fixed the rest of the formatting, sorry.

@Nekotekina I cherry-picked your commits and seems to work.

Use naive function pointer on Apple arm64 because ASLR breaks asmjit. See BufferUtils.cpp comment for explanation on why this happens and how to fix if you want to use asmjit.

Tell Qt not to strip debug symbols when we're in debug or relwithdebinfo modes.

Force MachO on macOS to fix LLVM being unable to patch relocations during codegen. Adds Aarch64 NEON intrinsics for x86 intrinsics used by PPUTranslator/Recompiler.

Temporary hack to get things working by using 16k pages instead of 4k pages in VM emulation.

Fixes some intrinsics usage and patches usages of asmjit to properly emit absolute jmps so ASLR doesn't cause out of bounds rel jumps. Also patches the SPU recompiler to properly work on arm64 by telling LLVM to target arm64.

Fixes W^X on macOS aarch64 by setting all JIT mmap'd regions to default to RW mode. For both SPU and PPU execution threads, when initialization finishes we toggle to RX mode. This exploits Apple's per-thread setting for RW/RX to let us be technically compliant with the OS's W^X enforcement while not needing to actually separate the memory allocated for code/data.

Implements ppu_gateway for arm64 and patches LLVM initialization to use the correct triple. Adds some fixes for macOS W^X JIT restrictions when entering/exiting JITed code.

Strictly speaking, rpcs3 JIT -> C++ calls are not tail calls. If you call a function inside e.g. an L2 syscall, it will clobber LR on arm64 and subtly break returns in emulated code. Only JIT -> JIT "calls" should be tail.

Tag mmap calls with MAP_JIT to allow W^X on macOS. Fix mmap calls to existing mmap'd addresses that were tagged with MAP_JIT on macOS. Fix memory unmapping on 16K page machines with a hack to mark "unmapped" pages as RW.

sguo35 · 2022-06-09T14:56:23Z

@Nekotekina Is there a blocker preventing this from being merged?

Utilities/JIT.cpp

rpcs3/Emu/CPU/CPUTranslator.h

rpcs3/Emu/Cell/PPUThread.cpp

Nekotekina · 2022-06-09T15:12:59Z

Probably no, just needs some corrections.

kd-11 · 2022-06-12T14:55:49Z

I can confirm this also fixes the ppu test on linux aarch64 (asahi)
Trying any actual apps crashes with segfaults when invoking module entrypoints (specifically cellVideoOutGetState) but that is outside the scope here.

sampletext32 · 2022-06-12T21:02:50Z

rpcs3/Emu/Cell/PPUTranslator.cpp

+#ifdef ARCH_X64
 	SetFpr(op.frd, m_ir->CreateXor(xormask, Call(GetType<s32>(), "llvm.x86.sse2.cvtsd2si", m_ir->CreateInsertElement(GetUndef<f64[2]>(), b, u64{0}))));
+#else
+	SetFpr(op.frd, m_ir->CreateXor(xormask, Call(GetType<s32>(), "llvm.aarch64.neon.fcvtns.i32.f64", b)));
+#endif


I might be dumb, but why are you calling the x86 instruction inside ifdef ARCH_X64, and the aarch64 inside the else. Shouldn't it be the opposite way?

X64 is not aarch64

It basically says "if ARCH_x64 (aka Intel x86_64), use SSE2; else, assume it's AArch64 (aka ARM64) and use Neon."

Megamouse reviewed May 29, 2022

View reviewed changes

Megamouse added OS: macOS CPU labels May 29, 2022

Nekotekina mentioned this pull request May 31, 2022

Obtain system memory page size #12139

Merged

Nekotekina reviewed May 31, 2022

View reviewed changes

rpcs3/Emu/Memory/vm.cpp Outdated Show resolved Hide resolved

Nekotekina reviewed May 31, 2022

View reviewed changes

Utilities/JIT.cpp Outdated Show resolved Hide resolved

Nekotekina reviewed May 31, 2022

View reviewed changes

rpcs3/Emu/CPU/CPUTranslator.cpp Outdated Show resolved Hide resolved

sguo35 force-pushed the arm64 branch from e4eb995 to 0f1f95e Compare June 1, 2022 20:10

sguo35 added 13 commits June 2, 2022 17:17

BufferUtils: use naive function pointer on Apple arm64

0c27e5a

Use naive function pointer on Apple arm64 because ASLR breaks asmjit. See BufferUtils.cpp comment for explanation on why this happens and how to fix if you want to use asmjit.

build-macos: fix source maps for Mac

05e2623

Tell Qt not to strip debug symbols when we're in debug or relwithdebinfo modes.

LLVM PPU: fix aarch64 on macOS

64cb8ef

Force MachO on macOS to fix LLVM being unable to patch relocations during codegen. Adds Aarch64 NEON intrinsics for x86 intrinsics used by PPUTranslator/Recompiler.

virtual memory: use 16k pages on aarch64 macOS

064732b

Temporary hack to get things working by using 16k pages instead of 4k pages in VM emulation.

PPU/SPU: fix NEON intrinsics and compilation for arm64 macOS

9868adc

Fixes some intrinsics usage and patches usages of asmjit to properly emit absolute jmps so ASLR doesn't cause out of bounds rel jumps. Also patches the SPU recompiler to properly work on arm64 by telling LLVM to target arm64.

PPU: implement aarch64 specific functions

e0abfe5

Implements ppu_gateway for arm64 and patches LLVM initialization to use the correct triple. Adds some fixes for macOS W^X JIT restrictions when entering/exiting JITed code.

PPU: Mark rpcs3 calls as non-tail

7f91fad

Strictly speaking, rpcs3 JIT -> C++ calls are not tail calls. If you call a function inside e.g. an L2 syscall, it will clobber LR on arm64 and subtly break returns in emulated code. Only JIT -> JIT "calls" should be tail.

macOS/arm64: compatibility fixes

c2c9d5f

vm: patch virtual memory for arm64 macOS

907ab19

Tag mmap calls with MAP_JIT to allow W^X on macOS. Fix mmap calls to existing mmap'd addresses that were tagged with MAP_JIT on macOS. Fix memory unmapping on 16K page machines with a hack to mark "unmapped" pages as RW.

PPU: remove wrong comment

0196c30

PPU: fix a merge regression

46fddf8

vm: remove 16k page hacks

93697c8

sguo35 force-pushed the arm64 branch 2 times, most recently from e2e1365 to 9f9cb81 Compare June 5, 2022 04:05

sguo35 added 2 commits June 4, 2022 21:10

PPU: formatting fixes

f61dd50

PPU: fix arm64 null function assembly

f9360d8

sguo35 force-pushed the arm64 branch from 9f9cb81 to f9360d8 Compare June 5, 2022 04:10

Nekotekina reviewed Jun 9, 2022

View reviewed changes

Utilities/JIT.cpp Show resolved Hide resolved

Nekotekina reviewed Jun 9, 2022

View reviewed changes

Utilities/JIT.cpp Outdated Show resolved Hide resolved

Nekotekina reviewed Jun 9, 2022

View reviewed changes

rpcs3/Emu/CPU/CPUTranslator.h Outdated Show resolved Hide resolved

Nekotekina reviewed Jun 9, 2022

View reviewed changes

rpcs3/Emu/Cell/PPUThread.cpp Show resolved Hide resolved

sampletext32 reviewed Jun 12, 2022

View reviewed changes

ppu: clean up arch-specific instructions

9ffe988

sguo35 force-pushed the arm64 branch from 294b3ba to 9ffe988 Compare June 14, 2022 03:37

Nekotekina merged commit cefc37a into RPCS3:master Jun 14, 2022

Yahfz mentioned this pull request Jun 14, 2022

Regression: RPCS3 Closes itself after building modules (#12115) #12226

Closed

Nekotekina mentioned this pull request Jun 14, 2022

SPU/PPU LLVM: fix triple setup (regression fix) #12228

Merged

sguo35 mentioned this pull request Jun 18, 2022

[ppu] Fix arm64 trampoline #12242

Merged

sguo35 mentioned this pull request Jul 10, 2022

spu: LLVM arm64 + macOS port #12338

Merged

This was referenced Aug 30, 2022

[macOS] Ask CMake to look in the right place for include & link directories #12516

Merged

[Feature request] Add macOS arm64 to CI #12599

Closed

shinra-electric mentioned this pull request Sep 7, 2022

Fails to build on aarch64 #12577

Closed

zintus mentioned this pull request Feb 11, 2023

[macOS] [Arm Native] LLVM: Segfault before launching any game #12882

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PPU LLVM arm64+macOS port #12115

PPU LLVM arm64+macOS port #12115

sguo35 commented May 28, 2022 •

edited

Loading

Megamouse left a comment

Megamouse May 29, 2022

Megamouse Jun 2, 2022

sguo35 Jun 3, 2022

sguo35 commented Jun 1, 2022

sguo35 commented Jun 9, 2022

Nekotekina commented Jun 9, 2022

kd-11 commented Jun 12, 2022

sampletext32 Jun 12, 2022

AniLeo Jun 12, 2022 •

edited

Loading

nastys Jun 12, 2022 •

edited

Loading

PPU LLVM arm64+macOS port #12115

PPU LLVM arm64+macOS port #12115

Conversation

sguo35 commented May 28, 2022 • edited Loading

Megamouse left a comment

Choose a reason for hiding this comment

Megamouse May 29, 2022

Choose a reason for hiding this comment

Megamouse Jun 2, 2022

Choose a reason for hiding this comment

sguo35 Jun 3, 2022

Choose a reason for hiding this comment

sguo35 commented Jun 1, 2022

sguo35 commented Jun 9, 2022

Nekotekina commented Jun 9, 2022

kd-11 commented Jun 12, 2022

sampletext32 Jun 12, 2022

Choose a reason for hiding this comment

AniLeo Jun 12, 2022 • edited Loading

Choose a reason for hiding this comment

nastys Jun 12, 2022 • edited Loading

Choose a reason for hiding this comment

sguo35 commented May 28, 2022 •

edited

Loading

AniLeo Jun 12, 2022 •

edited

Loading

nastys Jun 12, 2022 •

edited

Loading