runtime: several FreeBSD syscall wrappers seem to mishandle errors

[benesch]

I'm in the process of porting gccgo to FreeBSD and noticed something sketchy about some of the FreeBSD-specific syscall wrappers in the runtime package. Consider sys_umtx_op for example:

go/src/runtime/sys_freebsd_amd64.s

Lines 13 to 22 in 89f465c

	TEXT runtime·sys_umtx_op(SB),NOSPLIT,$0
	MOVQ addr+0(FP), DI
	MOVL mode+8(FP), SI
	MOVL val+12(FP), DX
	MOVQ uaddr1+16(FP), R10
	MOVQ ut+24(FP), R8
	MOVL $454, AX
	SYSCALL
	MOVL AX, ret+32(FP)
	RET

The caller of this function expects negative values to be errors and positive values to be successes:

go/src/runtime/os_freebsd.go

Lines 168 to 171 in 904e113

	ret := sys_umtx_op(addr, _UMTX_OP_WAIT_UINT_PRIVATE, val, unsafe.Sizeof(*utp), utp)
	if ret >= 0 || ret == -_EINTR {
	return
	}

But sys_umtx_op returns the result of the syscall (the AX register) directly, and according to the FreeBSD calling convention, the carry flag indicates whether AX contains an error code or successful return code, not the sign of the result. Most of the other syscalls in the package seem to get this right (anything that contains a JCC instruction after SYSCALL, roughly speaking, looks right to me), but a few others seem broken in the same way, like thr_new and pipe2.

#10052 seems to be some prior art on this subject. As a result a number of these syscall wrappers were fixed to inspect the carry flag, but not all of them.

Clearly in practice this doesn't seem to matter much, but probably worth fixing nonetheless.

[gopherbot]

Change https://golang.org/cl/276892 mentions this issue: runtime: correct errors handling in several FreeBSD syscall wrappers