32 bit arm panic when started in server mode

[dcarbone]

Overview of the Issue

I use Raspberry Pi4's as the hosts for my home consul cluster. Attempting to upgrade from v1.9.6 to v1.10.0 Armhfv6 resulted in a panic at boot.

Reproduction Steps

Steps to reproduce this issue:

1. Run v1.10.0 with a 32bit OS on ARM.

Consul info for both Client and Server

Unable to run consul info

Operating system and Environment details

Hardware: Pi4 4GB
OS: PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"

Log Fragments

panic: unaligned 64-bit atomic operation
goroutine 53 [running]:
runtime/internal/atomic.panicUnaligned()
        /usr/local/go/src/runtime/internal/atomic/unaligned.go:8 +0x24
runtime/internal/atomic.Xadd64(0x5611a64, 0x1, 0x0, 0x10, 0x56ec060)
        /usr/local/go/src/runtime/internal/atomic/asm_arm.s:233 +0x14
github.com/hashicorp/consul/agent/grpc.(*statsHandler).HandleConn(0x5611a60, 0x2749ad4, 0x4c58060, 0x272fb00, 0x56e805b)
        /home/circleci/project/consul/agent/grpc/stats.go:96 +0x1a0
google.golang.org/grpc/internal/transport.newHTTP2Server(0x2752d1c, 0x56ec030, 0x53a7ee8, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/http2_server.go:251 +0x105c
google.golang.org/grpc/internal/transport.NewServerTransport(...)
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/internal/transport/transport.go:533
google.golang.org/grpc.(*Server).newHTTP2Transport(0x5030960, 0x2752d1c, 0x56ec030, 0x0, 0x0, 0x0, 0x0)
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:702 +0xd8
google.golang.org/grpc.(*Server).handleRawConn(0x5030960, 0x2752d1c, 0x56ec030)
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:669 +0x394
google.golang.org/grpc.(*Server).Serve.func3(0x5030960, 0x2752d1c, 0x56ec030)
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:639 +0x2c
created by google.golang.org/grpc.(*Server).Serve
        /go/pkg/mod/google.golang.org/grpc@v1.25.1/server.go:638 +0x1f8

[mikemorris]

The offending snippet looks to be

consul/agent/grpc/stats.go

Lines 93 to 101 in 794029b

	var count uint64
	switch s.(type) {
	case *stats.ConnBegin:
	count = atomic.AddUint64(&c.activeConns, 1)
	c.metrics.IncrCounter([]string{"grpc", label, "connection", "count"}, 1)
	case *stats.ConnEnd:
	// Decrement!
	count = atomic.AddUint64(&c.activeConns, ^uint64(0))
	}

Looks like this might be hitting the Go bug documented at https://golang.org/pkg/sync/atomic/#pkg-note-BUG

On ARM, 386, and 32-bit MIPS, it is the caller's responsibility to arrange for 64-bit alignment of 64-bit words accessed atomically. The first word in a variable or in an allocated struct, array, or slice can be relied upon to be 64-bit aligned.

I'm not sure if the right fix here woud be to switch the count here to uint32, or if there might be a better way to ensure alignment?