[Benchmark] replace Wall-Clock Time with Per-process CPU Time

[Raimo33]

Summary

I propose replacing the current gettimeofday()-based timing measurements in the benchmark suite with CPU cycle counting using hardware performance counters (RDTSC). This change would significantly improve benchmark reliability, precision, and consistency.

Current Issues with Time-Based Measurements

The current benchmarking implementation in bench.h uses wall-clock time measurements with microsecond precision:

static int64_t gettime_i64(void) {
    struct timeval tv;
    gettimeofday(&tv, NULL);
    return (int64_t)tv.tv_usec + (int64_t)tv.tv_sec * 1000000LL;
}

This approach has several significant limitations:

1. Limited Precision: Microsecond resolution is insufficient for modern CPUs where cryptographic operations can complete in hundreds of nanoseconds. Requiring a lot of iterations to get an acceptable result
2. System Interference: Wall-clock time is affected by:
   • OS scheduler interrupts and context switches
   • Other running processes competing for CPU time
   • Power management and frequency scaling
   • Thermal throttling
   • Shared caches between cores
3. High Variability: Benchmark results can vary by 50%+ between runs due to system noise
4. Non-deterministic: Results depend on system load, making comparisons unreliable
5. Overhead: System call overhead affects measurement accuracy
6. Lack of comparison: doesn't provide a stable, reliable metric for team-wide performance discussions and comparisons.

Proposed Implementation

Use clocks that don't include the time that the process is paused etc. I propose something like clock_gettime(CLOCK_PROCESS_CPUTIME_ID) or perf stat -e cpu-clock

[real-or-random]

I think that's a good proposal.

[Raimo33]

I think that's a good proposal.

Unfortunately I was mistaken about RDTSC. It is not immune from system interference. It merely addresses points 1, 5, 6.

After careful research, the only options are clock_gettime(CLOCK_PROCESS_CPUTIME_ID) or perf stat -e cpu-clock. They would address points 1, 2 (except for shared caches), 3, 4.

both are not portable.