Excessive clock queries causing slowdown on freebsd

[Keno]

On freebsd, the jl_hrtime calls that we use to compute time spent in the compiler appear to be taking 70% of total execution time of compile-heavy julia processes. On linux it's not quite as bad, because this syscall has a special fastpath, but either way, this time is clearly excessive. cc @ianshmean who added this timer in #37678. At the very least, it should only be on in @time.

[Keno]

I'm a bit confused though, because FreeBSD appears to have the same vDSO based acceleration for this system call (https://github.com/lattera/freebsd/blob/401a161083850a9a4ce916f37520c084cff1543b/lib/libc/sys/__vdso_gettimeofday.c)

[Keno]

Hmm:

# sysctl kern.timecounter
kern.timecounter.tsc_shift: 1
kern.timecounter.smp_tsc_adjust: 0
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 1
kern.timecounter.fast_gettime: 1
kern.timecounter.tick: 1
kern.timecounter.choice: i8254(0) ACPI-fast(900) TSC-low(-100) dummy(-1000000)
kern.timecounter.hardware: ACPI-fast
kern.timecounter.alloweddeviation: 5
kern.timecounter.timehands_count: 2
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.counter: 30709
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.ACPI-fast.quality: 900
kern.timecounter.tc.ACPI-fast.frequency: 3579545
kern.timecounter.tc.ACPI-fast.counter: 7891073
kern.timecounter.tc.ACPI-fast.mask: 16777215
kern.timecounter.tc.TSC-low.quality: -100
kern.timecounter.tc.TSC-low.frequency: 1500038608
kern.timecounter.tc.TSC-low.counter: 4233127671
kern.timecounter.tc.TSC-low.mask: 4294967295

This is on AWS, but it looks like if the TSC timecounter is disabled, then so is the gettime fastpath: https://github.com/lattera/freebsd/blob/401a161083850a9a4ce916f37520c084cff1543b/sys/x86/x86/tsc.c#L760

[Keno]

Two action items here:

• Stop spamming clock queries
• Somebody should talk to upstream about the lack of vdso clock acceleration in virtualized environments (AWS in particular). Since linux can get away with it, I assume FreeBSD could too

[ararslan]

@kostikbel, apologies for the ping, but do you have any insight on the FreeBSD internals aspect here? I ask because you're on the copyright for the file in question. 🙂

[kostikbel]

Problem is that smp_tsc == 0. For start, try to set the loader tunables kern.timecounter.smp_tsc=1 and kern.timecounter.smp_tsc_adjust=1 and see

1. if smp_tsc is reported as 1 and default timecounter hardware choice changed
2. if 1 is fine, whether ntpd can pace the hardware, i.e. ntpd should not give up after one or two hours of uptime
3. if 2 is fine, whether it actually helps julia.

[Keno]

@ararslan Can you experiment with this? E.g. just time the build of corecompiler.ji with the various options.
Here's what happens if I manually turn on TSC-low:

$ time /usr/home/ec2-user/julia/usr/bin/julia -C "native" --output-ji /usr/home/ec2-user/julia/usr/lib/julia/corecompiler.ji.tmp --startup-file=no --warn-overwrite=yes -g0 -O0 compiler/compiler.jl
      740.55 real       111.85 user       615.77 sys
$ su
# sysctl kern.timecounter.hardware=TSC-low
kern.timecounter.hardware: ACPI-fast -> TSC-low 
$ time /usr/home/ec2-user/julia/usr/bin/julia -C "native" --output-ji /usr/home/ec2-user/julia/usr/lib/julia/corecompiler.ji.tmp --startup-file=no --warn-overwrite=yes -g0 -O0 compiler/compiler.jl
       93.66 real        81.22 user         0.60 sys

[Keno]

Note that I'm just using the stock AMI that's available on EC2.

[IanButterworth]

Meanwhile I'm working on a PR to disable spamming the timer unless during @time

[ararslan]

@ararslan Can you experiment with this?

Yes but only on a local machine; I don't have a virtualized FreeBSD environment. Will that still be useful?

[Keno]

I can spin you up a freebsd box on AWS with identical configuration.

[IanButterworth]

If anyone has a test system up, it may be good to double check that #38885 removes all unwanted jl_hrtime usage

[Keno]

Alright, this issue is resolved for us, but I think it's worth for @ararslan to continue following up with upstream to see why the vdso optimization is disabled on the default AWS images. After all, what's the point of such an optimization if it doesn't actually run a lot of the time.

[emaste]

For reference, upstream code review to stop avoiding TSC in VMs (which will enable the vdso optimization): https://reviews.freebsd.org/D27629