README.md

Sleep Experimentation in Python

This repository experiments with differing methods for Python-based sleep functionality. The general goal is to examine the accuracy and CPU usage of various sleep functions across a range of hardware.

We examine the following sleep functions:

• Python's built-in time.sleep function
• A basic spin lock implementation
• A Python implmentation of Sleep Spin (first proposed by Blat Blatnik)

Experimental Setup

Experiments are performed on the following hardware:

• Macbook Pro (macOS 12, M1 Max)
• Desktop Computer (Windows 10, i7 7700)
• Raspberry Pi 4 Model B 8GB (Raspberry Pi OS)
• A Vagrant-based Ubuntu Virtual Machine running on macOS and Windows

Results are generated by looping 5,000 times at 132Hz or 60Hz and recording various statistics. We examine the follwing variables across different loops:

• Use of different sleep functions
• Use of different clock functions (time.perf_counter vs. time.perf_counter_ns).
  • Note that nanoseconds were used for both clocks. time.perf_counter was converted to nanoseconds and time.perf_counter_ns was used as-is.
• Experimentation on idle hardware vs. hardware under (artificial) load.
  • Load is produced by using the multiprocessing library to occupy all available cores with relatively strenuous busy-work. For more information, please reference the stress.py Python file located at the root of this repo.

The following statistics are collected/generated while/after looping:

• The time for each loop during all 5,000 loops (aka the period).
• The mean absolute error for all periods vs. the true frequency.
• The min, max, mean, standard deviation of the periods, and number of periods 25% above the true frequency.
• The CPU usage of the entire loop.

Results

Experimental results from test runs are detailed below. For further examination, data can be found in the results folder located at the repo's root.

Results reported with "NS" are using the time.perf_counter_ns clock and results without "NS" are using the time.perf_counter clock.

Period Duration over Time

A compiled display can be found at the following page. Uncompiled graphics can be found in the results/spin_plot and results/sleep_plot folders.

Period Mean Absolute Error and Boxplots

A compiled display can be found at the following page. Uncompiled graphics can be found in the results/mae_boxplots folder.

Loop Statistics and CPU Usage

Please refer to the compiled result CSV files in the results folder:

132Hz Compiled Results

60Hz Compiled Results

Observations

• When under no load, spin locking exhibits the highest period accuracy (vs. the true frequency), however, this comes at the cost of high CPU usage on a thread.
• Spin locking while under load can result in large latency spikes on the Raspberry Pi and Windows. CPU usage of the timing process in these situations is reduced (ex: 80% vs. the usual 99-100%), indicating contention for CPU time with other demanding processes.
• The host machine influences how sleeping occurs inside a VM. Ex: Results from the Linux VM on Windows are closer to Windows results than a Linux VM on macOS or Raspberry Pi results.
• Across all experiments, introducing (artificial) load resulted in larger maximum periods and increased variability in period times.
• Using Python's sleep function for timing typically results in a high number of inaccurate periods on all platforms. sleep also demonstrates less robustness towards load than spin locking or sleep spinning for most cases. Using sleep, though, requires the least amount of CPU usage and can provide an accurate platform for timing in some cases (ex: on the Raspberry Pi under load).
• Windows exhibits lower period accuracy than other operating systems (Linux VM on Windows included). Python-wise, perf_counter and perf_counter_ns on Windows have 21% and 19% less resolution, respectively, versus Linux (see PEP 564 for more details).
  • Timing issues may also come down to how the scheduler is choosing to allocate CPU time on Windows. One notable example is with most sleep spin experiments on Windows taking nearly 0.0% CPU usage (vs. the usual ~12-40%). This indicates that the majority of time spent is within the sleep function rather than the spin lock loop when sleep spinning. This is likely caused by the scheduler choosing not to honour sleep times.
• On Linux and macOS, perf_counter exhibits slightly higher accuracy versus perf_counter_ns. It should be noted, however, that perf_counter is being converted to nanoseconds immediately to avoid floating point rounding error for the conducted experiments. This finding is corroborated in PEP 564, with perf_counter having a resolution of 82 ns vs. perf_counter_ns's resolution of 84 ns.
• Sleep spinning generally serves as a middle ground between sleeping and spin locking. Under load, however, accuracy is reduced in situations where spin locking also has trouble keeping an accurate period.
• Maintaining 132Hz resulted in more experiments with periods over frequency * 1.25, indicating that driving a 132Hz is more difficult than 60Hz for most systems.

Reproducing Results

Virtual Machine Setup

1. Depending on the host system (the system that will be running the VM), rename the appropriate Vagrantfile located at the repository root. Ex: For a macOS system on an M-series processor, Vagrantfile_macos_mseries would be renamed to Vagrantfile.

2. Ensure that Vagrant is installed and available on the command line:

# The following should look similar
> vagrant --version
Vagrant 2.3.4

3. Boot the VM with Vagrant

# For Windows and Intel Macs
vagrant up
# For M-series processors on macOS, QEMU will need to be installed as a plugin and specified as the provider
brew install qemu
vagrant plugin install vagrant-qemu
vagrant up --provider qemu

4. SSH into the VM and run any tests

vagrant ssh
# Navigate to the project directory
cd /vagrant

Note: For M-series processors, results will need to be manually copied back from the VM. This can be done using rsync or scp.

Running Experiments

1. Install all Python dependencies. If you are using a VM, you can skip this step as dependencies are installed as part of the boot process.

pip3 install -r requirements.txt

2. Run the main.py file to run all experiments and save the results to the project root.

python3 main.py