runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824

ianlancetaylor · 2022-07-13T01:04:06Z

Linux kernel 5.11 (February, 2021) added the epoll_pwait2 system call. It is exactly like epoll_pwait except that the timeout argument is a struct timespec. This permits specifying a timeout with nanosecond precision, where epoll_pwait and epoll_wait only permit millisecond precision. We should change runtime/netpoll_epoll.go to use epoll_pwait2 when available to get higher precision delays.

CC @golang/runtime

The text was updated successfully, but these errors were encountered:

prattmic · 2022-07-13T15:02:06Z

Specifically, this will help with #44343 (comment) on Linux.

dAdAbird · 2022-07-15T13:22:04Z

Does anybody work on this? I can take on it.

ianlancetaylor · 2022-07-15T14:04:31Z

Thanks. I suggest that you sketch out what you will do before implementing, to make sure that it seems like a good approach.

dAdAbird · 2022-07-15T16:38:46Z

@ianlancetaylor, I've just discovered the patch from @prattmic: https://go.dev/cl/363417.
As far as I can tell, it only lacks runtime·epollwait2 implementations for other than amd64 platforms. Are there some reasons you didn't move with the CL 363417 forward? Thanks!

prattmic · 2022-07-15T16:44:21Z

Mostly because I simply haven't gotten around to it. You are welcome to base a change off of my CL.

I also did want to do some benchmarking (probably with https://cs.opensource.google/go/x/benchmarks/+/master:sweet/) to check for performance regressions, since more precise timeouts could potentially lead to more frequent wakeups.

dAdAbird · 2022-07-15T16:49:48Z

Thanks, @prattmic. I'll prepare CL then and will do some benchmarking.

aclements · 2022-07-16T20:44:24Z

Since this is Linux-only, it could use the new runtime/internal/syscall package, rather than implementing it in assembly for all architectures.

dAdAbird · 2022-07-18T12:05:01Z

Oh, cool, thanks for the advice Austin!

dAdAbird · 2022-07-22T13:47:34Z

I have a few questions. @prattmic, in your CL you use _ENOSYS to define if the syscall exists. But I cannot find its definition anywhere other than in the "golang.org.x/sys/unix" package (ENOSYS there).

And in a broader context, if we go with runtime/internal/syscall, where should syscall numbers (e.g. SYS_epoll_pwait = 281, SYS_epoll_pwait2 = 441 for amd64) be defined? Currently, they are in runtime/sys_linux_xxx.s along with the syscall assembly implementations. I reckon all these definitions should end up in runtime/internal/syscall at the end of the day. But what to do with epoll_pwait2 while it's not there? I don't think using definitions from "golang.org.x/sys/unix" in runtime is a good idea. Or is it? Should I (try to) add sysnums to the runtime/internal/syscall?

Thanks!

ianlancetaylor · 2022-07-22T19:27:47Z

The constant _ENOSYS was removed from the runtime package in https://go.dev/cl/389354. It would be fine to revert that part of that CL for this purpose.

aclements · 2022-07-22T21:52:05Z

For the syscall number constants, I would create new per-arch files in runtime/internal/syscall to define those constants. Eventually, we'll want to shift all of the constants out of sys_linux_xxx.s and into that package (and ideally generate them).

Which packages the runtime can import are very restricted, so it wouldn't be able to import golang.org/x/sys/unix. We might eventually want to use the same code generation mechanism for both, but that's down the road.

dAdAbird · 2022-07-25T10:17:38Z

For the syscall number constants, I would create new per-arch files in runtime/internal/syscall to define those constants. Eventually, we'll want to shift all of the constants out of sys_linux_xxx.s and into that package (and ideally generate them).

Yes, this is what I was thinking to do. To generate those constants (like it's done in golang.org/x/sys/unix) and put them into runtime/internal/syscall.

Or in this CL I should only have SYS_epoll_pwait, SYS_epoll_pwait2 constants. And all the rest should be generated in the separate CL later on?

dAdAbird · 2022-07-25T12:12:39Z

Ah, as far as I can see there is no generator for the runtime now. Hence no easy way to generate syscall defs.
So I'll probably just add constants manually. Still, should be there only SYS_EPOLL_PWAIT and SYS_EPOLL_PWAIT2, or copy all syscalls from golang.org/x/sys/unix's zsysnum_linux_xxx.go files?

prattmic · 2022-07-25T14:42:38Z

I think for now you can just add the epoll constants you need for the CL.

dAdAbird · 2022-07-29T13:12:20Z

Hi, I'm again with questions.

I made functions for epoll_pwait and epoll_pwait2 calls in runtime/netpoll_epoll.go instead of assembler once, which looks like this:

var _zero uintptr
func epollwait2(epfd int, events []epollevent, maxev int, ts *timespec) (n, errno int) {
	var ev unsafe.Pointer
	if len(events) > 0 {
		ev = unsafe.Pointer(&events[0])
	} else {
		ev = unsafe.Pointer(&_zero)
	}
	r1, _, e := syscall.Syscall6(syscall.SYS_EPOLL_PWAIT2, uintptr(epfd), uintptr(ev), uintptr(maxev), uintptr(unsafe.Pointer(&ts)), 0, 0)
	return int(r1), int(e)
}

...

But then I thought, it makes sense to define these funcs in runtime/internal/syscall if we are switching to this package anyway. But for that, I'd have to move at least type epollevent struct definition from runtime/defs_linux_xxx.go to runtime/internal/syscall/defs_linux_xxx.go (which I've already created for SYS_EPOLL_PWAIT and SYS_EPOLL_PWAIT2 definitions). What do you think about that?
It that case, it's also worth moving into internal/syscall defs of _EINTR, _EAGAIN, _ENOMEM, _ENOSYS and making syscall.EpollPwait return Go error instead of errno. But it's for sure should be in a different CL, rather related to #51087.

Is there any reason why arguments and return values of epoll* funcs are int32 not int? Should I stay with int32 or may I switch to int in new epollwait* funcs?
I'd also redo other syscalls in runtime/netpoll_epoll.go to use runtime/internal/syscall for consistency reasons. But I'm not sure if to make it in this CL or make a new one (and an issue) for that.

Cheers!

prattmic · 2022-07-29T17:35:52Z

But then I thought, it makes sense to define these funcs in runtime/internal/syscall if we are switching to this package anyway. But for that, I'd have to move at least type epollevent struct definition from runtime/defs_linux_xxx.go to runtime/internal/syscall/defs_linux_xxx.go (which I've already created for SYS_EPOLL_PWAIT and SYS_EPOLL_PWAIT2 definitions). What do you think about that?

I think that sounds good.

_It that case, it's also worth moving into internal/syscall defs of _EINTR, _EAGAIN, _ENOMEM, _ENOSYS and making syscall.EpollPwait return Go error instead of errno.

error is an interface type, which may not be safe to use everywhere in the runtime that we need to make syscalls. But we potentially use an explicit Errno type for a bit stronger typing than the existing uintptr.

Is there any reason why arguments and return values of epoll* funcs are int32 not int? Should I stay with int32 or may I switch to int in new epollwait* funcs?

They use int32 because the kernel interface only accepts 32-bit integers. Note that int is 32-bits in the kernel on all architectures (see page 2).

I'd also redo other syscalls in runtime/netpoll_epoll.go to use runtime/internal/syscall for consistency reasons. But I'm not sure if to make it in this CL or make a new one (and an issue) for that.

I think you should have at least 2 CLs: one (or more) moving things to runtime/internal/syscall with no behavior changes, and one adding and using epoll_pwait2.

dAdAbird · 2022-07-29T18:44:24Z

Hi Michael, thank you for the feedback and explanations.

I think you should have at least 2 CLs: one (or more) moving things to runtime/internal/syscall with no behavior changes, and one adding and using epoll_pwait2.

Sounds good to me.

Cheers!

gopherbot · 2022-08-08T13:30:21Z

Change https://go.dev/cl/421994 mentions this issue: runtime: move epoll syscalls to runtime/internal/syscall

This change moves Linux epoll's syscalls implementation to the "runtime/internal/syscall" package. The intention in this CL was to minimise behavioural changes but make the code more generalised. This also will allow adding new syscalls (like epoll_pwait2) without the need to implement assembly stubs for each arch. It also drops epoll_create as not all architectures provide this call. epoll_create1 was added to the kernel in version 2.6.27 and Go requires Linux kernel version 2.6.32 or later since Go 1.18. So it is safe to always use epoll_create1. For #53824 For #51087 Change-Id: I9a6a26b7f2075a38e041de1bab4691da0ecb94fc Reviewed-on: https://go-review.googlesource.com/c/go/+/421994 Reviewed-by: Michael Pratt <mpratt@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com>

gopherbot · 2022-09-30T18:52:54Z

Change https://go.dev/cl/437295 mentions this issue: Revert "runtime: move epoll syscalls to runtime/internal/syscall"

This reverts CL 421994. Reason for revert: breaks runtime.TestCheckPtr2 For #53824 For #51087 Change-Id: I044ea4d6efdffe0a4b7fb0d2bb3717d9f391fc59 Reviewed-on: https://go-review.googlesource.com/c/go/+/437295 TryBot-Result: Gopher Robot <gobot@golang.org> Run-TryBot: Michael Pratt <mpratt@google.com> Auto-Submit: Michael Pratt <mpratt@google.com> Reviewed-by: Bryan Mills <bcmills@google.com>

gopherbot · 2022-10-07T10:28:22Z

Change https://go.dev/cl/440115 mentions this issue: runtime: move epoll syscalls to runtime/internal/syscall

This change moves Linux epoll's syscalls implementation to the "runtime/internal/syscall" package. The intention in this CL was to minimise behavioural changes but make the code more generalised. This also will allow adding new syscalls (like epoll_pwait2) without the need to implement assembly stubs for each arch. It also drops epoll_create as not all architectures provide this call. epoll_create1 was added to the kernel in version 2.6.27 and Go requires Linux kernel version 2.6.32 or later since Go 1.18. So it is safe to always use epoll_create1. This is a resubmit as the previous CL 421994 was reverted due to test failures after the merge with the master. The issue was fixed in CL 438615 For #53824 For #51087 Change-Id: I1bd0f23a85b4f9b80178c5dd36fd3e95ff4f9648 Reviewed-on: https://go-review.googlesource.com/c/go/+/440115 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Michael Pratt <mpratt@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com>

This change moves Linux epoll's syscalls implementation to the "runtime/internal/syscall" package. The intention in this CL was to minimise behavioural changes but make the code more generalised. This also will allow adding new syscalls (like epoll_pwait2) without the need to implement assembly stubs for each arch. It also drops epoll_create as not all architectures provide this call. epoll_create1 was added to the kernel in version 2.6.27 and Go requires Linux kernel version 2.6.32 or later since Go 1.18. So it is safe to always use epoll_create1. This is a resubmit as the previous CL 421994 was reverted due to test failures after the merge with the master. The issue was fixed in CL 438615 For golang#53824 For golang#51087 Change-Id: I1bd0f23a85b4f9b80178c5dd36fd3e95ff4f9648 Reviewed-on: https://go-review.googlesource.com/c/go/+/440115 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Michael Pratt <mpratt@google.com> Reviewed-by: Michael Pratt <mpratt@google.com> Run-TryBot: Michael Pratt <mpratt@google.com>

gopherbot · 2022-11-11T14:03:25Z

Change https://go.dev/cl/449815 mentions this issue: runtime: use epoll_pwait2 where available

mknyszek · 2023-06-09T19:25:54Z

It seems this has missed 2 releases. Moving to backlog.

Please comment if you plan to work on it in Go 1.22 and I'm happy to move it there. Thanks!

gopherbot · 2023-07-31T09:04:15Z

Change https://go.dev/cl/514275 mentions this issue: time: mention time resolution in docs

Updates #44343 Updates #53824 Change-Id: Ia7234fac4b1b88b3c331328aaa98dc85205e09ba Reviewed-on: https://go-review.googlesource.com/c/go/+/514275 Reviewed-by: David Chase <drchase@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@google.com> Run-TryBot: Quim Muntal <quimmuntal@gmail.com>

thepudds · 2024-02-02T17:26:15Z

Hi @dAdAbird, are you planning on returning to this?

ianlancetaylor added Performance help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jul 13, 2022

ianlancetaylor added this to the Backlog milestone Jul 13, 2022

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022

mknyszek added this to Go Compiler / Runtime Jul 13, 2022

mknyszek moved this to Triage Backlog in Go Compiler / Runtime Jul 15, 2022

prattmic assigned dAdAbird Jul 29, 2022

prattmic moved this from Triage Backlog to In Progress in Go Compiler / Runtime Jul 29, 2022

prattmic modified the milestones: Backlog, Go1.20 Jul 29, 2022

prattmic modified the milestones: Go1.20, Go1.21 Jan 11, 2023

mknyszek modified the milestones: Go1.21, Backlog Jun 9, 2023

alexbozhenko mentioned this issue Sep 21, 2023

runtime: time.Tick resolution on Linux dropped to less than 1ms starting with go 1.16? #63145

Closed

thepudds changed the title ~~runtime: use epoll_pwait2 where available~~ runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux Feb 2, 2024

thepudds mentioned this issue Feb 2, 2024

runtime: time.Sleep takes more time than expected on Windows (1ms -> 10ms) #44343

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824

runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824

ianlancetaylor commented Jul 13, 2022

prattmic commented Jul 13, 2022

dAdAbird commented Jul 15, 2022

ianlancetaylor commented Jul 15, 2022

dAdAbird commented Jul 15, 2022

prattmic commented Jul 15, 2022

dAdAbird commented Jul 15, 2022

aclements commented Jul 16, 2022 via email

dAdAbird commented Jul 18, 2022

dAdAbird commented Jul 22, 2022

ianlancetaylor commented Jul 22, 2022

aclements commented Jul 22, 2022

dAdAbird commented Jul 25, 2022

dAdAbird commented Jul 25, 2022

prattmic commented Jul 25, 2022

dAdAbird commented Jul 29, 2022 •

edited

Loading

prattmic commented Jul 29, 2022 •

edited

Loading

dAdAbird commented Jul 29, 2022

gopherbot commented Aug 8, 2022

gopherbot commented Sep 30, 2022

gopherbot commented Oct 7, 2022

gopherbot commented Nov 11, 2022

mknyszek commented Jun 9, 2023

gopherbot commented Jul 31, 2023

thepudds commented Feb 2, 2024

runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824

runtime: use epoll_pwait2 where available for sub-millisecond timeout precision on Linux #53824

Comments

ianlancetaylor commented Jul 13, 2022

prattmic commented Jul 13, 2022

dAdAbird commented Jul 15, 2022

ianlancetaylor commented Jul 15, 2022

dAdAbird commented Jul 15, 2022

prattmic commented Jul 15, 2022

dAdAbird commented Jul 15, 2022

aclements commented Jul 16, 2022 via email

dAdAbird commented Jul 18, 2022

dAdAbird commented Jul 22, 2022

ianlancetaylor commented Jul 22, 2022

aclements commented Jul 22, 2022

dAdAbird commented Jul 25, 2022

dAdAbird commented Jul 25, 2022

prattmic commented Jul 25, 2022

dAdAbird commented Jul 29, 2022 • edited Loading

prattmic commented Jul 29, 2022 • edited Loading

dAdAbird commented Jul 29, 2022

gopherbot commented Aug 8, 2022

gopherbot commented Sep 30, 2022

gopherbot commented Oct 7, 2022

gopherbot commented Nov 11, 2022

mknyszek commented Jun 9, 2023

gopherbot commented Jul 31, 2023

thepudds commented Feb 2, 2024

dAdAbird commented Jul 29, 2022 •

edited

Loading

prattmic commented Jul 29, 2022 •

edited

Loading