Skip to content

Commit

Permalink
CI: use a Dev Drive to improve Windows I/O performance
Browse files Browse the repository at this point in the history
Dev Drives are a modern Windows feature based on the ReFS file system
which offers significantly better performance for developer-focused
workloads. This is perfect for pip's Windows CI which is still slower
than the Unix jobs.

Most of the implementation was borrowed from the uv project which also
uses a Dev Drive to improve their Windows CI times. There is a community
action (samypr100/setup-dev-drive) that can set up a Dev Drive for us,
but the Powershell script needed for our simple needs is really not that
bad. The small maintenance burden of doing it ourselves is perferable
over the risks of using yet another 3rd party action.

NB: We used to use a RAM disk to improve I/O performance, but the creation
of the RAM disk started to fail intermittently, annoying everyone and
eliminating any speed ups gained by the constant retrying needed.

See also: https://learn.microsoft.com/en-us/windows/dev-drive/
  • Loading branch information
ichard26 committed Dec 22, 2024
1 parent 3b91f42 commit cad5bd9
Show file tree
Hide file tree
Showing 2 changed files with 54 additions and 19 deletions.
34 changes: 15 additions & 19 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,15 @@ jobs:
strategy:
fail-fast: true
matrix:
os: [ubuntu-latest, macos-13, macos-latest]
os: [ubuntu-latest]
# os: [ubuntu-latest, macos-13, macos-latest]
python:
- "3.8"
- "3.9"
- "3.10"
- "3.11"
- "3.12"
- "3.13"
# - "3.9"
# - "3.10"
# - "3.11"
# - "3.12"
# - "3.13"

steps:
- uses: actions/checkout@v4
Expand Down Expand Up @@ -162,7 +163,7 @@ jobs:
name: tests / ${{ matrix.python }} / ${{ matrix.os }} / ${{ matrix.group }}
runs-on: ${{ matrix.os }}-latest

needs: [packaging, determine-changes]
needs: [determine-changes]
if: >-
needs.determine-changes.outputs.tests == 'true' ||
github.event_name != 'pull_request'
Expand All @@ -189,41 +190,36 @@ jobs:
python-version: ${{ matrix.python }}
allow-prereleases: true

# We use C:\Temp (which is already available on the worker)
# as a temporary directory for all of the tests because the
# default value (under the user dir) is more deeply nested
# and causes tests to fail with "path too long" errors.
- name: Create a Dev Drive
run: |
./tools/ci/devdrive.ps1 -Drive R -Size 5GB
mkdir R:\Temp
echo "TEMP=R:\\Temp" >> $env:GITHUB_ENV
- run: pip install nox
env:
TEMP: "C:\\Temp"

# Main check
- name: Run unit tests
if: matrix.group == 1
run: >-
echo $TEMP
nox -s test-${{ matrix.python }} --
-m unit
--verbose --numprocesses auto --showlocals
env:
TEMP: "C:\\Temp"
- name: Run integration tests (group 1)
if: matrix.group == 1
run: >-
nox -s test-${{ matrix.python }} --
-m integration -k "not test_install"
--verbose --numprocesses auto --showlocals
env:
TEMP: "C:\\Temp"
- name: Run integration tests (group 2)
if: matrix.group == 2
run: >-
nox -s test-${{ matrix.python }} --
-m integration -k "test_install"
--verbose --numprocesses auto --showlocals
env:
TEMP: "C:\\Temp"
tests-zipapp:
name: tests / zipapp
Expand Down
39 changes: 39 additions & 0 deletions tools/ci/devdrive.ps1
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# A Powershell script to create a Dev Drive which tends to have significantly better
# performance in developer workloads. The goal is improve pip's Windows CI times.
#
# The implementation was borrowed from the uv project which also use a Dev Drive for
# better CI performance: https://github.com/astral-sh/uv/pull/3522
#
# Windows docs:
# https://learn.microsoft.com/en-us/windows/dev-drive/
# Related GHA reports:
# https://github.com/actions/runner-images/issues/7320 (Windows slowness report)
# https://github.com/actions/runner-images/issues/8698 (feature request for
# preprovisioned Dev Drives)

[CmdletBinding()]
param(
[Parameter(Mandatory=$true,
HelpMessage="Drive letter to use for the Dev Drive")]
[String]$drive,
[Parameter(Mandatory=$true,
HelpMessage="Size to allocate to the Dev Drive")]
[UInt64]$size
)
$ErrorActionPreference = "Stop"

$OSVersion = [Environment]::OSVersion.Version
$Partition = New-VHD -Path C:/pip_dev_drive.vhdx -SizeBytes $size |
Mount-VHD -Passthru |
Initialize-Disk -Passthru |
New-Partition -DriveLetter $drive -UseMaximumSize
# Dev Drives aren't supported on all GHA Windows runners, in which case fallback to
# a ReFS disk which still offer performance gains.
if ($OSVersion -ge (New-Object 'Version' 10.0.22621)) {
Write-Output "Dev Drives are supported, one will be created at $drive:"
$Volume = ($Partition | Format-Volume -DevDrive -Confirm:$false -Force)
} else {
Write-Output "Dev Drives are unsupported, only creating a ReFS disk at $drive:"
$Volume = ($Partition | Format-Volume -FileSystem ReFS -Confirm:$false -Force)
}
Write-Output $Volume

0 comments on commit cad5bd9

Please sign in to comment.