Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

First try at Cirrus config #99

Merged
merged 46 commits into from
Feb 2, 2023
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
f12b224
First try at Cirrus config
sgillies Jan 27, 2023
da3daae
Add REPO_DIR: rasterio
sgillies Jan 27, 2023
7a0907b
Set BUILD_PREFIX to $HOME/local. /usr/local can't be modified.
sgillies Jan 27, 2023
129a91a
Remove dependence on /usr/local existing
sgillies Jan 27, 2023
95e8251
Set perms on $HOME/local
sgillies Jan 27, 2023
6ff7d96
Debugging
sgillies Jan 27, 2023
d24bed2
Switch to new cmake getter and chown -R root:admin
sgillies Jan 29, 2023
6338476
More BUILD_PREFIX perms
sgillies Jan 29, 2023
a23b9f0
sudo build library deps
sgillies Jan 29, 2023
cf6993f
brew remove using force
sgillies Jan 29, 2023
49b47b0
Consolidate build and test wheel scripts
sgillies Jan 29, 2023
ae2b45b
Souce files within new sudo shell
sgillies Jan 29, 2023
9b963a6
Try sudo -s
sgillies Jan 29, 2023
5022c6d
Remove erroneous BUILD_PREFIX
sgillies Jan 29, 2023
8cfa7cd
New build prefix
sgillies Jan 29, 2023
fe3860f
Add debug listings
sgillies Jan 29, 2023
963c067
Bring back debugging listings of build prefix
sgillies Jan 29, 2023
ed71139
Remove sudo
sgillies Jan 29, 2023
c70cab5
Pre-create man dirs
sgillies Jan 29, 2023
67c7b12
Fix man path
sgillies Jan 29, 2023
bec87e5
Fix zstd install directory
sgillies Jan 29, 2023
45630ae
Add missing prefix for libtiff
sgillies Jan 29, 2023
1a086f4
Switch build_geos to cmake
sgillies Jan 29, 2023
a5cea4c
Fix GEOS URL (bz2)
sgillies Jan 29, 2023
050e064
Switch build_gdal to Cmake
sgillies Jan 30, 2023
d656f11
Fixes
sgillies Jan 30, 2023
80d4428
Remove conflicting geotiff var
sgillies Jan 30, 2023
1177612
Remove patch
sgillies Jan 30, 2023
173f54b
Remove GDAL patches, enable optional raster drivers, set GDAL_CONFIG
sgillies Jan 30, 2023
df1d5aa
deployment target 12.0
sgillies Jan 30, 2023
a5391ea
Set macosx deployment target
sgillies Jan 31, 2023
cc0fbd1
GTFO GML
sgillies Jan 31, 2023
bdd9582
gdal-config broken?
sgillies Jan 31, 2023
0f3467d
Remove gdal-config
sgillies Jan 31, 2023
4a2228c
Specify GDAL_VERSION
sgillies Jan 31, 2023
abda364
GDAL 3.5.3
sgillies Jan 31, 2023
9285023
Patch multibuild and revert to gdal-config
sgillies Jan 31, 2023
249ec9d
Source missing scripts
sgillies Jan 31, 2023
2b1324b
Add 3 python versions
sgillies Jan 31, 2023
ed847e2
Get modern cmake early, once
sgillies Jan 31, 2023
caa26d7
Adaptive stripping
sgillies Feb 1, 2023
2993ed3
Don't build postgisraster
sgillies Feb 1, 2023
db7e7fb
Buh bye exr
sgillies Feb 1, 2023
e2f0305
Disable HEIF and ODBC
sgillies Feb 1, 2023
1900738
Add blosc, remove homebrew lz4
sgillies Feb 1, 2023
f5a89d5
Remove homebrew's openjpeg
sgillies Feb 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .cirrus.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
wheel_build_task:
name: Build Rasterio wheels for Mac ARM64.
macos_instance:
image: ghcr.io/cirruslabs/macos-monterey-xcode
env:
BUILD_PREFIX: /tmp/local
REPO_DIR: rasterio
BUILD_COMMIT: 1.3.5
CIRRUS_CLONE_SUBMODULES: true
PATH: /opt/homebrew/opt/python@3.10/bin:$PATH
PLAT: "arm64"
BUILD_DEPENDS: "oldest-supported-numpy cython>=0.29 wheel==0.34.2"
TEST_DEPENDS: "oldest-supported-numpy attrs==21.4.0 pytest==6.2.5 click==8.0.3 mock boto3 packaging hypothesis==4.38.1 wheel==0.34.2"
MACOSX_DEPLOYMENT_TARGET: "11.0"
matrix:
- MB_PYTHON_VERSION: "3.11"
install_pre_requirements_script:
- brew install python@3.10
- ln -s python3 /opt/homebrew/opt/python@3.10/bin/python
brew_remove_pkgs_script:
- brew remove -f --ignore-dependencies webp zstd xz libtiff libxcb curl php lcms2 xmlto ghostscript
build_wheel_script:
- mkdir -p $BUILD_PREFIX/include
- mkdir -p $BUILD_PREFIX/lib
- mkdir -p $BUILD_PREFIX/share/man/man{1,3,5,7}
- source multibuild/common_utils.sh
- source multibuild/travis_steps.sh
- python3 -m pip install virtualenv
- before_install
- clean_code $REPO_DIR $BUILD_COMMIT
- build_wheel $REPO_DIR $PLAT
test_wheel_script:
- source multibuild/common_utils.sh
- source multibuild/travis_steps.sh
- install_run $PLAT
wheels_artifacts:
path: "wheelhouse/*.whl"
158 changes: 67 additions & 91 deletions config.sh
Original file line number Diff line number Diff line change
Expand Up @@ -44,15 +44,30 @@ function fetch_unpack {
function build_geos {
CFLAGS="$CFLAGS -g -O2"
CXXFLAGS="$CXXFLAGS -g -O2"
build_simple geos $GEOS_VERSION https://download.osgeo.org/geos tar.bz2
if [ -e geos-stamp ]; then return; fi
local cmake=$(get_modern_cmake)
fetch_unpack http://download.osgeo.org/geos/geos-${GEOS_VERSION}.tar.bz2
(cd geos-${GEOS_VERSION} \
&& mkdir build && cd build \
&& $cmake .. \
-DCMAKE_INSTALL_PREFIX:PATH=$BUILD_PREFIX \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DENABLE_IPO=ON \
-DBUILD_APPS:BOOL=OFF \
-DBUILD_TESTING:BOOL=OFF \
&& $cmake --build . -j4 \
&& $cmake --install .)
touch geos-stamp
}


function build_jsonc {
if [ -e jsonc-stamp ]; then return; fi
local cmake=$(get_modern_cmake)
fetch_unpack https://s3.amazonaws.com/json-c_releases/releases/json-c-${JSONC_VERSION}.tar.gz
(cd json-c-${JSONC_VERSION} \
&& /usr/local/bin/cmake -DCMAKE_INSTALL_PREFIX=$BUILD_PREFIX . \
&& $cmake -DCMAKE_INSTALL_PREFIX=$BUILD_PREFIX . \
&& make -j4 \
&& make install)
if [ -n "$IS_OSX" ]; then
Expand All @@ -71,19 +86,20 @@ function build_proj {
CFLAGS="$CFLAGS -g -O2"
CXXFLAGS="$CXXFLAGS -g -O2"
if [ -e proj-stamp ]; then return; fi
local cmake=$(get_modern_cmake)
build_sqlite
fetch_unpack http://download.osgeo.org/proj/proj-${PROJ_VERSION}.tar.gz
(cd proj-${PROJ_VERSION} \
&& mkdir build && cd build \
&& /usr/local/bin/cmake .. \
&& $cmake .. \
-DCMAKE_INSTALL_PREFIX:PATH=$BUILD_PREFIX \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DENABLE_IPO=ON \
-DBUILD_APPS:BOOL=OFF \
-DBUILD_TESTING:BOOL=OFF \
&& cmake --build . -j4 \
&& cmake --install .)
&& $cmake --build . -j4 \
&& $cmake --install .)
touch proj-stamp
}

Expand Down Expand Up @@ -114,22 +130,6 @@ function build_expat {
}


function get_cmake {
local cmake=cmake
if [ -n "$IS_OSX" ]; then
brew install cmake > /dev/null
else
fetch_unpack https://www.cmake.org/files/v3.12/cmake-3.12.1.tar.gz > /dev/null
(cd cmake-3.12.1 \
&& ./bootstrap --prefix=$BUILD_PREFIX > /dev/null \
&& make -j4 > /dev/null \
&& make install > /dev/null)
cmake=/usr/local/bin/cmake
fi
echo $cmake
}


function build_tiff {
if [ -e tiff-stamp ]; then return; fi
build_jpeg
Expand All @@ -141,7 +141,7 @@ function build_tiff {
(cd tiff-${TIFF_VERSION} \
&& mv VERSION VERSION.txt \
&& (patch -u --force < ../patches/libtiff-rename-VERSION.patch || true) \
&& ./configure --enable-zstd --enable-webp \
&& ./configure --prefix=$BUILD_PREFIX --enable-zstd --enable-webp \
&& make -j4 \
&& make install)
touch tiff-stamp
Expand All @@ -153,7 +153,7 @@ function build_openjpeg {
build_zlib
build_tiff
build_lcms2
local cmake=$(get_cmake)
local cmake=$(get_modern_cmake)
local archive_prefix="v"
if [ $(lex_ver $OPENJPEG_VERSION) -lt $(lex_ver 2.1.1) ]; then
archive_prefix="version."
Expand All @@ -168,6 +168,9 @@ function build_openjpeg {


function build_libwebp {
ls -l $BUILD_PREFIX
ls -l $BUILD_PREFIX/share
ls -l $BUILD_PREFIX/share/man
build_libpng
build_giflib
build_simple libwebp $LIBWEBP_VERSION \
Expand Down Expand Up @@ -226,7 +229,7 @@ function build_zstd {
fi
(cd zstd-${ZSTD_VERSION} \
&& make -j4 PREFIX=$BUILD_PREFIX ZSTD_LEGACY_SUPPORT=0 \
&& make install SED_ERE_OPT=$sed_ere_opt)
&& make install PREFIX=$BUILD_PREFIX SED_ERE_OPT=$sed_ere_opt)
touch zstd-stamp
}

Expand All @@ -250,78 +253,51 @@ function build_gdal {
CXXFLAGS="$CXXFLAGS -g -O2"

if [ -n "$IS_OSX" ]; then
EXPAT_PREFIX=/usr
GEOS_CONFIG="--without-geos"
GEOS_CONFIG="-DGDAL_USE_GEOS=OFF"
else
EXPAT_PREFIX=$BUILD_PREFIX
GEOS_CONFIG="--with-geos=${BUILD_PREFIX}/bin/geos-config"
GEOS_CONFIG="-DGDAL_USE_GEOS=ON"
fi

local cmake=$(get_modern_cmake)
fetch_unpack http://download.osgeo.org/gdal/${GDAL_VERSION}/gdal-${GDAL_VERSION}.tar.gz
(cd gdal-${GDAL_VERSION} \
&& (patch -u -p2 --force < ../patches/4646.diff || true) \
&& (patch -u -p2 --force < ../patches/6194.diff || true) \
&& ./configure \
--with-crypto=yes \
--with-hide-internal-symbols \
--with-webp=${BUILD_PREFIX} \
--disable-debug \
--disable-static \
--disable-driver-elastic \
--prefix=$BUILD_PREFIX \
--with-curl=curl-config \
--with-expat=${EXPAT_PREFIX} \
${GEOS_CONFIG} \
--with-geotiff=internal \
--with-gif \
--with-grib \
--with-jpeg \
--with-libiconv-prefix=/usr \
--with-libjson-c=${BUILD_PREFIX} \
--with-libtiff=${BUILD_PREFIX} \
--with-libz=/usr \
--with-netcdf=${BUILD_PREFIX} \
--with-openjpeg \
--with-pam \
--with-png \
--with-proj=${BUILD_PREFIX} \
--with-sqlite3=${BUILD_PREFIX} \
--with-zstd=${BUILD_PREFIX} \
--with-threads \
--without-bsb \
--without-cfitsio \
--without-dwgdirect \
--without-ecw \
--without-fme \
--without-freexl \
--without-gnm \
--without-grass \
--without-ingres \
--without-jasper \
--without-jp2mrsid \
--without-jpeg12 \
--without-kakadu \
--without-libgrass \
--without-libgrass \
--without-libkml \
--without-mrf \
--without-mrsid \
--without-mysql \
--without-odbc \
--without-ogdi \
--without-pcidsk \
--without-pcraster \
--without-perl \
--without-pg \
--without-php \
--without-python \
--without-qhull \
--without-sde \
--without-sfcgal \
--without-xerces \
--without-xml2 \
&& make -j4 \
&& make install)
&& mkdir build \
&& cd build \
&& $cmake .. \
-DCMAKE_INSTALL_PREFIX=$BUILD_PREFIX \
-DCMAKE_INCLUDE_PATH=$BUILD_PREFIX/include \
-DCMAKE_LIBRARY_PATH=$BUILD_PREFIX/lib \
-DCMAKE_PROGRAM_PATH=$BUILD_PREFIX/bin \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DGDAL_BUILD_OPTIONAL_DRIVERS=OFF \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-DGDAL_BUILD_OPTIONAL_DRIVERS=OFF is quite drastic. It will disable a number of raw raster drivers that might be popular like ENVI, etc that were presumably available in the autoconf builds

-DOGR_BUILD_OPTIONAL_DRIVERS=OFF \
${GEOS_CONFIG} \
-DGDAL_USE_TIFF=ON \
-DGDAL_USE_TIFF_INTERNAL=OFF \
-DGDAL_USE_GEOTIFF=ON \
-DGDAL_USE_GEOTIFF_INTERNAL=ON \
-DGDAL_ENABLE_DRIVER_GIF=ON \
-DGDAL_ENABLE_DRIVER_GRIB=ON \
-DGDAL_ENABLE_DRIVER_JPEG=ON \
-DGDAL_USE_ICONV=ON \
-DGDAL_USE_JSONC=ON \
-DGDAL_USE_JSONC_INTERNAL=OFF \
-DGDAL_USE_ZLIB=ON \
-DGDAL_USE_ZLIB_INTERNAL=OFF \
-DGDAL_ENABLE_DRIVER_NETCDF=ON \
-DGDAL_ENABLE_DRIVER_OPENJPEG=ON \
-DGDAL_ENABLE_DRIVER_PNG=ON \
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps enable GeoPackage (was likely enabled in autoconf build) (note: this is called OGR_ENABLE_DRIVER_GPKG but this enables both vector & raster sides)

Suggested change
-DGDAL_ENABLE_DRIVER_PNG=ON \
-DGDAL_ENABLE_DRIVER_PNG=ON \
-DOGR_ENABLE_DRIVER_GPKG=ON \

-DBUILD_PYTHON_BINDINGS=OFF \
-DBUILD_JAVA_BINDINGS=OFF \
-DBUILD_CSHARP_BINDINGS=OFF \
-DGDAL_USE_SFCGAL=OFF \
-DGDAL_USE_XERCESC=OFF \
-DGDAL_USE_LIBXML2=OFF \
&& $cmake --build . -j4 \
&& $cmake --install .)
if [ -n "$IS_OSX" ]; then
:
else
Expand All @@ -343,15 +319,15 @@ function pre_build {
suppress build_nghttp2

if [ -n "$IS_OSX" ]; then
rm /usr/local/lib/libpng*
rm /usr/local/lib/libpng* || true
fi

suppress build_openssl

fetch_unpack https://curl.haxx.se/download/curl-${CURL_VERSION}.tar.gz

# Remove previously installed curl.
rm -rf /usr/local/lib/libcurl*
rm -rf /usr/local/lib/libcurl* || true

suppress build_curl
suppress build_libwebp
Expand Down