cmd/compile,runtime: race support for riscv64

[mauri870]

There was a previous issue requesting support for riscv64 (#58739), but it was closed because upstream support in LLVM was lacking.

On Oct 6 support for riscv64 was finally merged into the LLVM tree.

I propose we reconsider adding race detector support to Risc-V. From the top of my head this would require:

• Update the compiler to recognize -race on linux/riscv64
• clone llvm and compile a race_linux_riscv64.syso file, push the necessary changes upstream
• linux-riscv64-race builder?
• Add a new platform to x/build/cmd/racebuild to build syso files for riscv64
• Add race instrumentation (race_riscv64.s) to runtime
• Document support on https://go.dev/doc/articles/race_detector
• freebsd/riscv64 support is possible in the future?

Happy to work on this for the early stage of Go 1.23.

cc @golang/runtime @golang/riscv64

[mengzhuo]

I can setup a linux-riscv64-race builder(LUCI) for you.

[mauri870]

Thanks @mengzhuo, appreciate if you could handle that 🙏

[mauri870]

@mengzhuo I tried compiling the syso file for riscv64 with CC=clang and got stuck in this part:

runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LBB438_37 (type=291 (R_RISCV_ADD32) stype=1 (STEXT))
runtime/race(.rodata): unexpected relocation type 295 (R_RISCV_SUB32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LJTI438_0 (type=295 (R_RISCV_SUB32) stype=9 (SRODATA))runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LBB438_169 (type=291 (R_RISCV_ADD32) stype=1 (STEXT))
runtime/race(.rodata): unexpected relocation type 295 (R_RISCV_SUB32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LJTI438_0 (type=295 (R_RISCV_SUB32) stype=9 (SRODATA))runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LBB438_169 (type=291 (R_RISCV_ADD32) stype=1 (STEXT))
runtime/race(.rodata): unexpected relocation type 295 (R_RISCV_SUB32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LJTI438_0 (type=295 (R_RISCV_SUB32) stype=9 (SRODATA))runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LBB438_169 (type=291 (R_RISCV_ADD32) stype=1 (STEXT))
runtime/race(.rodata): unexpected relocation type 295 (R_RISCV_SUB32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LJTI438_0 (type=295 (R_RISCV_SUB32) stype=9 (SRODATA))runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LBB438_169 (type=291 (R_RISCV_ADD32) stype=1 (STEXT))
runtime/race(.rodata): unexpected relocation type 295 (R_RISCV_SUB32)
runtime/race(.rodata): unsupported dynamic relocation for symbol .LJTI438_0 (type=295 (R_RISCV_SUB32) stype=9 (SRODATA))runtime/race(.rodata): unexpected relocation type 291 (R_RISCV_ADD32)
/home/ubuntu/git/go/pkg/tool/linux_riscv64/link: too many errors

Maybe I have to use some specific compiler flag? I see that these relocations are currently supported in the go linker

[mengzhuo]

CC @4a6f656c
@mauri870 builder is up, I had same issue like yours.

[4a6f656c]

@mauri870 that looks like the riscv64 linker is missing handling for some relocation types - do you have a branch handy that I can build/test from?

[mauri870]

@4a6f656c I have some changes in one of my branches, hopefully you can still build from it, if not then let me know:

https://github.com/golang/go/compare/master...mauri870:go:feature/race-riscv64?expand=1

Edit: If you wanna compile the syso file yourself, then you have to get a hold of a llvm copy and then cd compiler-rt/lib/tsan/go && CC=clang ./buildgo.sh

[4a6f656c]

@mauri870 I've added the missing relocation handling to the internal linker and fixed up some other aspects of the riscv64 race code:

https://github.com/4a6f656c/go/tree/riscv64-race

We can now call into the TSAN code, however it fails on the call to __tsan_map_shadow with:

==2568052==ERROR: ThreadSanitizer failed to allocate 0x1f10000 (32571392) bytes at address 200000660000 (errno: 12)

At a first guess, this looks like it is using the 48-bit VMA mapping even though it is running on a 39-bit VMA platform.

[mauri870]

@4a6f656c Error 12 is ENOMEM, perhaps you are running out of memory on this machine?

https://github.com/torvalds/linux/blob/6f52b16c5b29b89d92c0e7236f4655dc8491ad70/include/uapi/asm-generic/errno-base.h#L16

Looking at the riscv64 tsan commit in llvm the 1000 0000 00 - 4000 0000 00 range seems to be correct for shadow memory on a 39-bit VMA.

[4a6f656c]

@4a6f656c Error 12 is ENOMEM, perhaps you are running out of memory on this machine?

https://github.com/torvalds/linux/blob/6f52b16c5b29b89d92c0e7236f4655dc8491ad70/include/uapi/asm-generic/errno-base.h#L16

Looking at the riscv64 tsan commit in llvm the 1000 0000 00 - 4000 0000 00 range seems to be correct for shadow memory on a 39-bit VMA.

The issue is caused by the fact that there is no Go mapping configuration for riscv64, which means we're falling through to the default.

I've pushed some further fixes to https://github.com/4a6f656c/go/tree/riscv64-race which makes the Go side of things correct, however even with a Go riscv64 mapping added to llvm, I've not been able to get it to detect races (the Go code is instrumented correctly and calls into the race detector code). The next step is to figure out the correct TSAN mappings for riscv64 in VMA 39 and VMA 48 modes (and we'll likely need to adjust Go's runtime/malloc.go to match).

[mengzhuo]

After some work on @4a6f656c 's riscv64-race branch, it's working now.
https://github.com/mengzhuo/go/tree/riscv64-race

But it requires llvm/llvm-project#127295 to be merged first.

[markdryan]

After some work on @4a6f656c 's riscv64-race branch, it's working now. https://github.com/mengzhuo/go/tree/riscv64-race

But it requires llvm/llvm-project#127295 to be merged first.

Great. I did try this out on QEMU and it did indeed detect a race. Is there any reason why the llvm patch seems to be sv48 only? When I tried the patch on my VF2 running Ubuntu 24.04 I got

FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48

[mengzhuo]

After some work on @4a6f656c 's riscv64-race branch, it's working now. https://github.com/mengzhuo/go/tree/riscv64-race
But it requires llvm/llvm-project#127295 to be merged first.

Great. I did try this out on QEMU and it did indeed detect a race. Is there any reason why the llvm patch seems to be sv48 only? When I tried the patch on my VF2 running Ubuntu 24.04 I got

FATAL: ThreadSanitizer: unsupported VMA range
FATAL: Found 39 - Supported 48

TSAN shadow memory requires at least 4 times of app VM.
Go allocates 896GiB app VM for TSAN (executable and heap) already greater than VMA39 maximum size (256 GiB).

You can check this memory mappings: https://github.com/llvm/llvm-project/pull/127295/files#diff-2c1abce7a33228a6123799fcd314048c1a4df96d93dbad00e2b614a7ee4dde53R684-R690

[4a6f656c]

Getting it working on sv48 is great progress, however I think we'll need to figure out a way to get it working on sv39... otherwise it is going to be of limited use and difficult to test via a race builder.

[4a6f656c]

After some work on @4a6f656c 's riscv64-race branch, it's working now. https://github.com/mengzhuo/go/tree/riscv64-race

Does this pass an ./all.bash run on sv48?

[mengzhuo]

After some work on @4a6f656c 's riscv64-race branch, it's working now. https://github.com/mengzhuo/go/tree/riscv64-race

Does this pass an ./all.bash run on sv48?

No. The race test always failed with difference testcase :( To figure out what problem is,
I have run bytes.test with race yesterday and it's ~1/10 will failed, I think there are some race bug in runtime (closure?).