regexp: Invalid character class on Windows ARM

[anacrolix]

Go version

master and 1.22 (and probably other versions)

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/anacrolix/Library/Caches/go-build'
GOENV='/Users/anacrolix/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/anacrolix/go/pkg/mod'
GOOS='darwin'
GOPATH='/Users/anacrolix/go'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.3/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.3/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.3'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/m4/f67w9zfx1pl386_2yjs7xtkm0000gn/T/go-build3366901238=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

I'm using github.com/google/go-cmp on Windows ARM. It was compiled for amd64. This is a playground for it, but obviously it's not cross compiled: https://go.dev/play/p/8a4Kt1DD-al?v=gotip.

What did you see happen?

A panic during init. The init code looks fine. The binary also appears to be loading fine, it just doesn't like this character class.

panic: regexp: Compile(`[_\p{L}][_\p{L}\p{N}]*$`): error parsing regexp: invalid character class range: `\p{L}`

goroutine 1 [running]:
regexp.MustCompile({0x7ff66897b52c, 0x17})
	/opt/homebrew/Cellar/go/1.22.3/libexec/src/regexp/regexp.go:317 +0xb4
github.com/google/go-cmp/cmp/internal/function.init()
	/Users/anacrolix/go/pkg/mod/github.com/google/go-cmp@v0.6.0/cmp/internal/function/func.go:68 +0x7c

What did you expect to see?

go-cmp's init to work normally and main to start.

[anacrolix]

I reported this downstream google/go-cmp#359, but I believe the issue is in go.

[gabyhelp]

Related Issues

• regex Windows panic MustCompile/MatchString #28765 (closed)
• regexp: regex pattern matching error #52460 (closed)
• regexp/syntax: panic: unhandled case in compile - x{2} #68067 (closed)
• cmd/compile: panic during sccp while compiling main runtime error: index out of range [-5] #64826 (closed)
• cmd/go: bootstrap with Go 1.21.x fail at tip, panic: godebug: Value of name not listed in godebugs.All: gotypesalias #64106 (closed)
• runtime: encountered a SIGSEGV during 'go run' in go1.16beta1 #43284 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[anacrolix]

None of the related issues match this problem.

[thanm]

Hi @anacrolix I am having trouble reproducing this problem -- when I build + run your test case on our windows/arm64 builders, it seems to work just fine. I tried both with tip of master and with Go 1.22.4.

Your go env output shows a Darwin machine, just confirming that this is indeed a windows issue and not a Darwin issue? Might help if you could tell me a bit more about the setup/configuration of the windows machine where you're seeing the problem. Thanks.

[anacrolix]

Thanks for looking. Specifically I cross compiled from darwin to windows/amd64, then ran that binary on Windows 11 on windows/arm64.

[thanm]

Still can't reproduce. From my darwin arm64 machine I built the program you listed via

GOOS=windows GOARCH=amd64 go build prog.go

downloaded the resulting executable to a windows/arm64 builder, still works fine. What am I missing?

[mauri870]

I couldn't reproduce as well. This feels like the error you would get by copying the regexp from the internet and some character in the regexp string would have a different encoding. But given this happens with a library upstream I'd say it is probably something to do with the OS.

[anacrolix]

Could it be CGO related? I am linking in a bunch of other stuff, and using a cross compiler (x86_64-unknown-linux-gnu-gcc) for some of those cgo parts.

At a stretch, could it be triggered by some kind of memory or emultation corruption? Are your windows/arm64 instances emulated (mine is).

The (downstream) library you mention is heavily used, AFAICT the character classes it's using are well defined too.

Let me know what else I can provide? Thanks

[thanm]

CGO: sure, I can certainly imagine problems there, but it depends on the code of course.

Are your windows/arm64 instances emulated (mine is).

The Go team windows/arm64 builders are Azure VMs, and I am pretty sure they are backed by actual arm hardware.

[anacrolix]

I'll try to construct a minimal reproduction. Thanks for looking into it

[anacrolix]

I'm only able to reproduce with upx involved. I've mentioned an issue in upx that looks related. Obviously this could be out of the scope of Go, however I'm able to reproduce it from a part of my code that additionally seems to incorrectly depend on go-cmp.

I've created a repo to reproduce at https://github.com/anacrolix/go-issue-68268. It contains some code guarded by conditions, and additionally that can be commented out to avoid compiling in the dependencies. The behaviour varies significantly if you mess around with it. The justfile contains some build steps, and copies to my Windows ARM VM.

If I use my package github.com/anacrolix/torrent/bencode, a dependency is generated to github.com/google/go-cmp/cmp via github.com/anacrolix/torrent/bencode.test (Comment out all the other lines to see this behaviour). When the resulting binary is run through upx, something must be stripped out. It's not clear why this code is included in the first place however.

> go mod why -m github.com/google/go-cmp
# github.com/google/go-cmp
github.com/anacrolix/go-issue-68268
github.com/anacrolix/torrent/bencode
github.com/anacrolix/torrent/bencode.test
github.com/google/go-cmp/cmp

CGO is not related at all. I don't have real ARM hardware running Windows, but I also think that's not the problem either.

[jreiser]

@anacrolix The best way to help make progress on the upx side is to "Paste, drop, or click to add files" (or give the URL) of an actual (never compressed yet) executable file which runs, but which fails when compressed by upx. (ref: upx/upx#443) So far I cannot find such a file here.

[anacrolix]

@jreiser I thought it would be preferable to build it yourself from source (per the repo). Here's the exe that runs fine before upx.
fine-before-upx.zip

[jreiser]

@anacrolix Thank you for the file. An actual executable file is a cut point in the graph of debugging possibilities, so it is the first place to look. Repo, integrated development environment (options, menu settings, environment variables, compiler/linker, Python site packages integrations, etc.), and system or local static libraries often introduce dozens of variables. Variations in the OS (Windows 11, latest 10, early 10, 7) sometimes present their own problems, but having a specific actual executable file still helps.

[anacrolix]

@jreiser what's the next step? Should I open a new issue in upx? Is there any possibility that Go is at fault here?

Since this issue only occurs if upx is involved, should I close it?

[jreiser]

@jreiser what's the next step? Should I open a new issue in upx? Is there any possibility that Go is at fault here?

Yes, please open a new issue in upx, and close this issue with golang.

We have seen issues where a program that has been compressed by upx gets lost trying to inspect the address space or the compressed file. This is the fault of the program, because upx guarantees only the mappings that are described by the PE file. If there are Resources in the file that are not mapped into the address space, then you should try the various upx compression options for a PE file (upx --help):

Options for win32/pe, win64/pe & rtm32/pe:
  --compress-exports=0    do not compress the export section
  --compress-exports=1    compress the export section [default]
  --compress-icons=0      do not compress any icons
  --compress-icons=1      compress all but the first icon
  --compress-icons=2      compress all but the first icon directory [default]
  --compress-icons=3      compress all icons
  --compress-resources=0  do not compress any resources at all
  --keep-resource=list    do not compress resources specified by list
  --strip-relocs=0        do not strip relocations
  --strip-relocs=1        strip relocations [default]