[Issue]: Atomic optimizer reorder causes memory access fault in Blender #58
Comments
|
I can confirm that reverting that commit (30a3adf) fixes the crash with other scenes like classroom but not with blender-3.2 (https://cloud.blender.org/p/gallery/629f23f908e12d4ff15241d3) which still crashed with the similar error (happens only with rocm-6.x): |
|
@LAKostis this is weird, because the blender-3.2 splashscreen renders fine for me on all the devices I've tested. Are you sure your Blender is not using your cycles cache? Maybe try clearing I also see that you seem to be using the Blender provided by your distro. For consistency, can you download the official binaries from Blender and try testing with that instead? |
Yes, I've specially tested this before rocm-6.0.x migration and this demo started failing only with rocm-6.0.x. Cache clearing doesn't help. It can be device specific issue - this demo crashes on my RX 6700 XT (gfx1031) but works on gfx900 (with rendering artifacts and only after setting HSA_ENABLE_SMDA=0) Regarding the blender build - I'm the blender package maintainer in this distro, so I'm sure what build options where used there :) For the sake of clarity official blender builds crash exactly the same way. I can provide any additional information or logs if you need any. |
|
Thanks for the quick response!
Just curious, does compiling against |
No, with HSA_OVERRIDE_GFX_VERSION=gfx1030 it doesn't start with error But interesting, it works with lowering -O level: with
I'm using 6.6.25 kernel + patches up to v6.5-2638-gbf901afac5d5f from amd-staging-drm-next |
|
UPDATE: more funny things with compiler:
This command produces workable kernel despite of warnings about deprecated commands. But this command:
Produces kernel which crashes. And those kernels are not equal: |
- Applied fixes for cycles:
+ cycles/hip: reduce opt level and enable hipcc-func-supp for
gfx1031 kernel (see ROCm/llvm-project#58)
|
UPDATE: checked with recent rocm and blender 4.2.0 kernel 6.9.10 If I compile gfx1031 with previous workaround (--hipcc-func-supp -O1) blender crashes on every rendering with errors: If I compile kernel with default upstream options (-O3) it renders most scenes (bmw/classroom) but still crashes on Blender 3 scene: If I compile with -O2, still the same behavior, Blender 3 crashes. But everything works with -O1: |
|
Does reverting 30a3adf still help? I'm considering just reverting this patch for Solus's ROCm 6.1.2. |
|
I've tested with reverted commit, can try again without reverting.
…On Mon, 22 Jul 2024, 19:28 Gavin Zhao, ***@***.***> wrote:
Does reverting 30a3adf
<30a3adf>
still help? I'm considering just reverting this patch for Solus's ROCm
6.1.2.
—
Reply to this email directly, view it on GitHub
<#58 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACMG6QPSN2R6CGG5PFJCJZTZNU6LZAVCNFSM6AAAAABFUT43JCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBTGQ3DENRRHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Can try after applying following patches: |
Without reverting that commit everything works with -O1 but fails if O >=2. |
Hey! Those patches are already applied in rocm-llvm somewhere in this bulk commit (1ce2523) |
|
ROCm 6.0.0 might have this issue. Looks like those patches (fixes) landed in next rocm-release immediately. |
) Currently, process of replacing bitwise operations consisting of `LSR`/`LSL` with `And` is performed by `DAGCombiner`. However, in certain cases, the `AND` generated by this process can be removed. Consider following case: ``` lsr x8, x8, #56 and x8, x8, #0xfc ldr w0, [x2, x8] ret ``` In this case, we can remove the `AND` by changing the target of `LDR` to `[X2, X8, LSL #2]` and right-shifting amount change to 56 to 58. after changed: ``` lsr x8, x8, #58 ldr w0, [x2, x8, lsl #2] ret ``` This patch checks to see if the `SHIFTING` + `AND` operation on load target can be optimized and optimizes it if it can.
ppanchad-amd
added
Under Investigation
generic
|
Hi @GZGavinZhao @LAKostis, as @pravinjagtap mentioned, the issue should be fixed now. Can you try the installing the latest version of ROCm and confirm on your system so the issue can be closed? I was able to reproduce the issue on ROCm 6.0 by following the steps @GZGavinZhao mentioned. My system is running Ubuntu 20.04 and has 3 MI210s (gfx90a). 
I also tried ROCm 6.3.0 and was able to render successfully:
|
|
|
|
gfx90c is not officially supported on ROCm, is this issue still present on any of the supported GPUs? |
|
Not to my knowledge then. |
|
Hello this issue is back with 9070 so it should be probably reopened.
This is present in blender, but in other apps as well I was able to reproduce this with
|
Problem Description
Following https://projects.blender.org/blender/blender/issues/112084, I've bisected the
rocm-6.0.xbranch and found that commit 30a3adf caused any Blender render (using HIP, of course) to crash with message along the lines of "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."Operating System
Solus 4.5 Resilience
CPU
AMD Ryzen 7 5800H with Radeon Graphics
GPU
AMD Instinct MI250, AMD Radeon VII
ROCm Version
ROCm 6.0.0
ROCm Component
llvm-project
Steps to Reproduce
Build this project at the commit mentioned.
Download the Blender 4.1 release binaries:
curl -O https://download.blender.org/release/Blender4.1/blender-4.1.0-linux-x64.tar.xz,tar xf blender-4.1.0-linux-x64.tar.xz. You should now have a folderblender-4.1.0-linux-x64.Clone Blender. Just cloning the
v4.1.0tag is enough:git clone https://projects.blender.org/blender/blender.git --depth 1 --branch v4.1.0.In the Blender repo, compile the HIP fatbin used to run Blender render:
hipcc --offload-arch=$arch --genco intern/cycles/kernel/device/hip/kernel.cpp -D CCL_NAMESPACE_BEGIN= -D CCL_NAMESPACE_END= -D HIPCC -I intern/cycles/kernel/.. -I intern/cycles/kernel/device/hip -ffast-math -o kernel_$arch.fatbin. AdjustHIP_ROCCLR_HOME,HIP_CLANG_PATHas necessary to point to the Clang you just compiled. Replace$archwith the GPU architecture to run on, e.g.gfx900orgfx1030. Don't add extra attributes like:xnack-.If you want to run on multiple architectures, repeat step 4 and 5 for each architecture.
Put this file into
blender-4.1.0-linux-x64/4.1/scripts/addons/cycles/lib/kernel_$arch.fatbin.Get the BMW27 Blender demo file.
curl -O https://download.blender.org/demo/test/BMW27.blend.zip,unzip BMW27.blend.zip. You should have a fileBMW27.blend.Now run Blender render.
blender-4.1.0-linux-x64/blender -b <path-to-BMW27.blend> -f 0 -- --cycles-device HIP. By default it runs on GPU with device ID 0, so adjustHIP_VISIBLE_DEVICESas necessary to run on the desired GPU.You should almost immediately see Blender crash with an error message similar to "Memory access fault by GPU node-1 (Agent handle: 0x7f1db8337e00) on address 0x7f1bf177e000. Reason: Page not present or supervisor privilege."
Now, build LLVM at 1 commit prior, e.g.
git switch --detach 30a3adf50e2d49dfc97c1b614d9b93638eba672d~1. Repeat step 4-7, and Blender should render normally.All of this is on ROCm 6.0.0. If you get a hang instead of a crash when running Blender (likely your on an APU), Ctrl+C and run again with environment variable
HSA_ENABLE_SMDA=0.(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support
Additional Information
This behavior has been reproduced on MI250, RX6600M, Vega 10, and Ryzen 7 5800H. (gfx90a, gfx1032, gfx900, gfx90c, respectively)
Kernel version:
6.6.22-281.current, with torvalds/linux@96c211f reverted (ref: https://lists.freedesktop.org/archives/amd-gfx/2023-October/100298.html and ROCm/ROCm#2596 (comment))The text was updated successfully, but these errors were encountered: