Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CI: Fix vcpkg Windows and Linux builds #77

Merged
merged 30 commits into from
Apr 21, 2022
Merged

CI: Fix vcpkg Windows and Linux builds #77

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
a8ad8c2
Change syntax for passing in VCPKG_INSTALLATION_ROOT to windows
brendan-ward Apr 20, 2022
04b10dd
debug just using expat build
brendan-ward Apr 20, 2022
f2934d8
See what happens if we don't set install root on command line
brendan-ward Apr 20, 2022
67344f9
Try to figure out where things are getting installed
brendan-ward Apr 20, 2022
7851e6b
Looks like some env vars we expect from CI are not set?
brendan-ward Apr 20, 2022
9b02b52
More path probing
brendan-ward Apr 20, 2022
244a343
Try again with windows path
brendan-ward Apr 20, 2022
44e91b5
Try again...
brendan-ward Apr 20, 2022
b4218f2
Try with hard-coded windows path
brendan-ward Apr 20, 2022
07a7cf9
try with os-specific vcpkg directories
brendan-ward Apr 21, 2022
17a58d5
Try again using $VCPKG_INSTALLATION_ROOT but windows paths, re-enable…
brendan-ward Apr 21, 2022
933639e
Fix paths / env var
brendan-ward Apr 21, 2022
3071c40
More fixes
brendan-ward Apr 21, 2022
48c45db
try to debug build caching on macos
brendan-ward Apr 21, 2022
cbdba33
More debugging into vcpkg macos cache
brendan-ward Apr 21, 2022
fcf08cd
Now try to build and cache gdal
brendan-ward Apr 21, 2022
0c11a21
Re-enable building wheels, linux build, etc
brendan-ward Apr 21, 2022
9705b6e
try pinned manylinux image and enable debug logging
brendan-ward Apr 21, 2022
4205814
fix workflow
brendan-ward Apr 21, 2022
038bcc2
re-enable other builds and wheel testing
brendan-ward Apr 21, 2022
7b9df80
try env variable without curly braces
jorisvandenbossche Apr 21, 2022
63da1d2
Try using unquoted variables for cache and build with fresh cache jus…
brendan-ward Apr 21, 2022
9268eef
Try setting VCPKG_INSTALLATION_ROOT manually again
brendan-ward Apr 21, 2022
b206bea
Try powershell env var
brendan-ward Apr 21, 2022
309382a
and also try bash
jorisvandenbossche Apr 21, 2022
99c17e8
Disable cache, remove failing ls
brendan-ward Apr 21, 2022
50e053e
Try using log file dir without quotes
brendan-ward Apr 21, 2022
f056845
try with cache matrix var and powershell vars for windows
brendan-ward Apr 21, 2022
cc56a77
Try with absolute paths for logs
brendan-ward Apr 21, 2022
63dc5ac
Re-enable full build
brendan-ward Apr 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 16 additions & 8 deletions .github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ jobs:
uses: docker/setup-buildx-action@v1
with:
install: true
buildkitd-flags: --debug
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and BUILDKIT_PROGRESS were added in attempt to get the Docker build to emit more logs during build, so we can see what vcpkg packages get installed. I'm not sure if both are needed, but didn't want to invest the build time in trying to game these.


- name: Build Docker image with vcpkg and gdal
# using build-push-action (without push) to make use of cache arguments
Expand All @@ -69,6 +70,8 @@ jobs:
load: true
cache-from: type=gha
cache-to: type=gha,mode=max
env:
BUILDKIT_PROGRESS: plain

- name: Build wheels
uses: pypa/cibuildwheel@v2.3.1
Expand All @@ -86,8 +89,14 @@ jobs:
include:
- os: "macos-10.15"
triplet: "x64-osx-dynamic"
vcpkg_cache: "/Users/runner/.cache/vcpkg/archives"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the asset paths with hardcoded links because they didn't work properly when using the $VCPKG_INSTALLATION_ROOT form, but I don't see a big downside to having them hardcoded regardless.

vcpkg_logs: "/usr/local/share/vcpkg/buildtrees/**/*.log"

- os: "windows-2019"
triplet: "x64-windows"
# windows requires windows-specific paths
vcpkg_cache: "c:\\vcpkg\\installed"
vcpkg_logs: "c:\\vcpkg\\buildtrees\\**\\*.log"

env:
MACOSX_DEPLOYMENT_TARGET: "10.15"
Expand All @@ -99,13 +108,13 @@ jobs:
fetch-depth: 0

- name: Cache vcpkg
uses: actions/cache@v2
uses: actions/cache@v3
id: vcpkgcache
with:
path: |
/Users/runner/.cache/vcpkg/archives
C:\vcpkg\installed
key: ${{ matrix.os }}-x86_64-vcpkg-gdal3.4.2-1
${{ matrix.vcpkg_cache }}
# bump the last digit to avoid using previous build cache
key: ${{ matrix.os }}-x86_64-vcpkg-gdal3.4.2-cache4

# MacOS build requires aclocal, which is part of automake, but appears
# to be missing in default image
Expand All @@ -118,17 +127,16 @@ jobs:
- name: Install GDAL
env:
VCPKG_DEFAULT_TRIPLET: ${{ matrix.triplet }}

shell: bash
run: |
echo "VCPKG_INSTALLATION_ROOT: ${VCPKG_INSTALLATION_ROOT}"
vcpkg install --overlay-triplets=./ci/custom-triplets --feature-flags="versions,manifests" --x-manifest-root=./ci --x-install-root="${VCPKG_INSTALLATION_ROOT}/installed"
vcpkg install --overlay-triplets=./ci/custom-triplets --feature-flags="versions,manifests" --x-manifest-root=./ci --x-install-root=$VCPKG_INSTALLATION_ROOT/installed
vcpkg list

- name: Upload vcpkg build logs
if: ${{ failure() }}
uses: actions/upload-artifact@v3
with:
path: "${VCPKG_INSTALLATION_ROOT}/buildtrees/**/*.log"
path: ${{ matrix.vcpkg_logs }}

- name: Build wheels
uses: pypa/cibuildwheel@v2.3.1
Expand Down
2 changes: 1 addition & 1 deletion ci/manylinux2014_x86_64-vcpkg-gdal.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM quay.io/pypa/manylinux2014_x86_64
FROM quay.io/pypa/manylinux2014_x86_64:2022-04-03-da6ecb3

RUN yum install -y curl unzip zip tar python3

Expand Down
6 changes: 3 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ def run(self):
else:
raise Exception(
"Could not find GDAL data files for packaging. "
"Ensure to set the GDAL_DATA environment variable"
"Make sure to set the GDAL_DATA environment variable"
)

proj_data = os.environ.get("PROJ_LIB")
Expand All @@ -177,8 +177,8 @@ def run(self):
else:
raise Exception(
"Could not find PROJ data files for packaging. "
"Ensure to set the PROJ_LIB environment variable"
)
"Make sure to set the PROJ_LIB environment variable"
)

package_data = {"pyogrio": ["gdal_data/*", "proj_data/*"]}

Expand Down