Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issues for running on Windows #313

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions .azure-pipelines.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,6 @@ jobs:
########################################################################################
- job:
displayName: 'Windows'
condition: False

pool:
vmImage: 'vs2017-win2016'
Expand All @@ -173,18 +172,12 @@ jobs:
Python37:
python.version: '3.7'
PYTHON: '3.7'
Python36:
python.version: '3.6'
PYTHON: '3.6'
#Python36:
#python.version: '3.6'
#PYTHON: '3.6'

steps:

# Install ghostscript separately since there is no Windows conda-forge package for it.
- bash: |
set -x -e
choco install ghostscript
displayName: Install ghostscript via chocolatey

- powershell: Write-Host "##vso[task.prependpath]$env:CONDA\Scripts"
displayName: Add conda to PATH

Expand All @@ -203,6 +196,13 @@ jobs:
conda list
displayName: List installed packages

# Download latest GMT
- script: git clone --branch=master --depth=1 https://github.com/GenericMappingTools/gmt.git
displayName: Download GMT

# Build GMT
- template: ci/azure-pipelines-windows.yml

# Install the package that we want to test
- bash: |
set -x -e
Expand Down
6 changes: 3 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ install:
test:
# Run a tmp folder to make sure the tests are run on the installed version
mkdir -p $(TESTDIR)
@echo ""
@cd $(TESTDIR); python -c "import $(PROJECT); $(PROJECT).print_clib_info()"
@echo ""
#@echo ""
#@cd $(TESTDIR); python -c "import $(PROJECT); $(PROJECT).print_clib_info()"
#@echo ""
cd $(TESTDIR); pytest $(PYTEST_ARGS) $(PROJECT)
cp $(TESTDIR)/.coverage* .
rm -r $(TESTDIR)
Expand Down
88 changes: 88 additions & 0 deletions ci/azure-pipelines-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Template for Windows steps in Azure Pipelines

steps:

# Cache vcpkg installed libraries
# Currently, caches are immutable and can't be updated.
# To rebuild the cache, you have to change the date in the key list
- task: CacheBeta@0
inputs:
key: $(Agent.OS) | vcpkg | 20190802
path: $(VCPKG_INSTALLATION_ROOT)/installed/
cacheHitVar: CACHE_VCPKG_RESTORED
displayName: Cache vcpkg libraries

- bash: |
set -x -e
# By default, vcpkg builds both release and debug configurations.
# Set VCPKG_BUILD_TYPE to build release only to save half time
echo 'set (VCPKG_BUILD_TYPE release)' >> ${VCPKG_INSTALLATION_ROOT}/triplets/${WIN_PLATFORM}.cmake
cat ${VCPKG_INSTALLATION_ROOT}/triplets/${WIN_PLATFORM}.cmake
# install libraries
#vcpkg install netcdf-c gdal pcre fftw3 clapack openblas --triplet ${WIN_PLATFORM}
vcpkg install netcdf-c gdal pcre fftw3 --triplet ${WIN_PLATFORM}
displayName: Install dependencies via vcpkg
env:
WIN_PLATFORM: "x64-windows"
condition: ne(variables['CACHE_VCPKG_RESTORED'], true)

- bash: |
# hook up user-wide integration
vcpkg integrate install
# list installed libraries
vcpkg list
# Executable files search for DLL files in the directories listed in the PATH environment variable.
echo "##vso[task.prependpath]${VCPKG_INSTALLATION_ROOT}/installed/${WIN_PLATFORM}/bin"
# Tools like gdal_translate, ogr2ogr are located in tools/gdal
echo "##vso[task.prependpath]${VCPKG_INSTALLATION_ROOT}/installed/${WIN_PLATFORM}/tools/gdal"
displayName: Configure vcpkg
env:
WIN_PLATFORM: "x64-windows"

- bash: |
set -x -e
choco install ninja
choco install ghostscript --version 9.26 # gs 9.27 is buggy
displayName: Install dependencies via chocolatey

- bash: |
echo "##vso[task.setvariable variable=INSTALLDIR]$BUILD_SOURCESDIRECTORY/gmt-install-dir"
echo "##vso[task.setvariable variable=COASTLINEDIR]$BUILD_SOURCESDIRECTORY/coastline"
displayName: Set install location and coastline location

- bash: echo "##vso[task.prependpath]$INSTALLDIR/bin"
displayName: Set PATH

- task: CacheBeta@0
inputs:
key: coastline
path: $(COASTLINEDIR)
cacheHitVar: CACHE_COASTLINE_RESTORED
displayName: Cache GSHHG and DCW data

- bash: ci/download-coastlines.sh
displayName: Download coastlines
condition: ne(variables['CACHE_COASTLINE_RESTORED'], true)

- bash: ci/config-gmt-windows.sh
displayName: Configure GMT

# Azure Pipelines cannot find Visual Studio correctly if Ninja is used.
# Here, we need
# 1. call the vcvars64.bat script
# 2. define CMAKE_C_COMPILER
- script: |
mkdir build
cd build
call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Auxiliary\Build\vcvars64.bat"
cmake .. -G "Ninja" -DCMAKE_TOOLCHAIN_FILE=%VCPKG_INSTALLATION_ROOT%\scripts\buildsystems\vcpkg.cmake -DCMAKE_C_COMPILER=cl.exe -DCMAKE_BUILD_TYPE=Release
cmake --build .
cmake --build . --target install
displayName: Compile GMT

- script: |
gmt --version
gmt defaults -Vd
gmt pscoast -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -P -Vd > test.ps
gmt begin && gmt coast -R0/10/0/10 -JM6i -Ba -Ggray -ENG+p1p,blue -Vd && gmt end
displayName: Check a few simple commands
41 changes: 41 additions & 0 deletions ci/config-gmt-windows.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/usr/bin/env bash
# Build and install GMT

# To return a failure if any commands inside fail
set -e

cat > cmake/ConfigUser.cmake << 'EOF'
set (CMAKE_INSTALL_PREFIX "$ENV{INSTALLDIR}")
set (GSHHG_ROOT "$ENV{COASTLINEDIR}/gshhg")
set (DCW_ROOT "$ENV{COASTLINEDIR}/dcw")
set (COPY_GSHHG TRUE)
set (COPY_DCW TRUE)
set (GMT_INSTALL_MODULE_LINKS FALSE)
set (CMAKE_C_FLAGS "/D_CRT_SECURE_NO_WARNINGS /D_CRT_SECURE_NO_DEPRECATE ${CMAKE_C_FLAGS}")
set (CMAKE_C_FLAGS "/D_CRT_NONSTDC_NO_DEPRECATE /D_SCL_SECURE_NO_DEPRECATE ${CMAKE_C_FLAGS}")
EOF

if [[ "$TEST" == "true" ]]; then
cat >> cmake/ConfigUser.cmake << 'EOF'
enable_testing()
set (DO_EXAMPLES TRUE)
set (DO_TESTS TRUE)
set (DO_API_TESTS ON)
set (SUPPORT_EXEC_IN_BINARY_DIR TRUE)
EOF
fi

if [[ "$BUILD_DOCS" == "true" ]]; then
cat >> cmake/ConfigUser.cmake << 'EOF'
set (DO_ANIMATIONS TRUE)
set (CMAKE_VERBOSE_MAKEFILE ON CACHE BOOL "ON")
EOF
fi

echo ""
echo "Using the following cmake configuration:"
cat cmake/ConfigUser.cmake
echo ""

# Turn off exit on failure.
set +e
55 changes: 55 additions & 0 deletions ci/download-coastlines.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/usr/bin/env bash
# Download and install the coastlines and boundaries datasets

# General settings:
EXT="tar.gz"
GSHHG="gshhg-gmt-2.3.7"
DCW="dcw-gmt-1.1.4"
MD5_GSHHG=8ee2653f9daf84d49fefbf990bbfa1e7
MD5_DCW=4f30857a8b12af0f910222fceb591538

# Used for checking the downloaded files:
check_md5 ()
{
md5_ref=$1
file=$2

test -f "$file" || return 1
md5=$(openssl dgst -md5 $file | cut -d ' ' -f 2)
test "$md5" = "$md5_ref"
}

# To return a failure if any commands inside fail
set -e

# Create the directory for coastline data
mkdir -p $COASTLINEDIR

# GSHHG and DCW tarballs are cached here:
test -d $HOME/cache-gshhg-dcw || mkdir $HOME/cache-gshhg-dcw
cd $HOME/cache-gshhg-dcw

# GSHHG (coastlines, rivers, and political boundaries):
echo ""
echo "Downloading and unpacking GSHHG"
echo "================================================================================"
# download when md5sums don't match:
check_md5 $MD5_GSHHG $GSHHG.$EXT || curl -L -O --retry 10 "https://mirrors.ustc.edu.cn/gmt/$GSHHG.$EXT"
check_md5 $MD5_GSHHG $GSHHG.$EXT
tar xzf $GSHHG.$EXT
mv $GSHHG $COASTLINEDIR/gshhg

# DCW (country polygons):
echo ""
echo "Downloading and unpacking DCW"
echo "================================================================================"
# download when md5sums don't match:
check_md5 $MD5_DCW $DCW.$EXT || curl -L -O --retry 10 "https://mirrors.ustc.edu.cn/gmt/$DCW.$EXT"
check_md5 $MD5_DCW $DCW.$EXT
tar xzf $DCW.$EXT
mv $DCW $COASTLINEDIR/dcw

ls $COASTLINEDIR/gshhg $COASTLINEDIR/dcw

# Turn off exit on failure.
set +e
84 changes: 23 additions & 61 deletions pygmt/clib/loading.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,85 +37,47 @@ def load_libgmt(env=None):
couldn't access the functions).

"""
libpath = get_clib_path(env)
try:
libgmt = ctypes.CDLL(libpath)
check_libgmt(libgmt)
except OSError as err:
msg = "\n".join(
[
"Error loading the GMT shared library '{}':".format(libpath),
"{}".format(str(err)),
]
)
raise GMTCLibNotFoundError(msg)
return libgmt


def get_clib_path(env):
"""
Get the path to the libgmt shared library.

Determine the file name and extension and append to the path set by
``GMT_LIBRARY_PATH``, if any.

Parameters
----------
env : dict or None
A dictionary containing the environment variables. If ``None``, will
default to ``os.environ``.

Returns
-------
libpath : str
The path to the libgmt shared library.

"""
libname = clib_name()
if env is None:
env = os.environ
if "GMT_LIBRARY_PATH" in env:
libpath = os.path.join(env["GMT_LIBRARY_PATH"], libname)
else:
libpath = libname
return libpath
libnames = clib_name(os_name=sys.platform)
libpath = env.get("GMT_LIBRARY_PATH", "")
error = False
for libname in libnames:
try:
libgmt = ctypes.CDLL(os.path.join(libpath, libname))
check_libgmt(libgmt)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If both ctypes.CDLL and check_libgmt don't raise any errors, it means we find the library, then we should break out of the loop. Right?

Suggested change
check_libgmt(libgmt)
check_libgmt(libgmt)
break

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Of course! Thanks, Dongdong. I stopped coding yesterday because I couldn't even see this basic error anymore :)

Copy link
Member

@weiji14 weiji14 Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized that this change is part of what's been causing our Windows builds to fail... Since we have a list of dll files to check, if one of the dll fails, the variable error turns True and stays True even if a later dll works. Our Linux and MacOS tests work because there's only one item in the list.

break
except OSError as err:
error = err
if error:
raise GMTCLibNotFoundError(
"Error loading the GMT shared library '{}':".format(libname)
)
return libgmt


def clib_name(os_name=None, is_64bit=None):
def clib_name(os_name):
"""
Return the name of GMT's shared library for the current OS.

Parameters
----------
os_name : str or None
The operating system name as given by ``sys.platform``
(the default if None).
is_64bit : bool or None
Whether or not the OS is 64bit. Only used if the OS is ``win32``. If
None, will determine automatically.
os_name : str
The operating system name as given by ``sys.platform``.

Returns
-------
libname : str
The name of GMT's shared library.
libname : list of str
List of possible names of GMT's shared library.

"""
if os_name is None:
os_name = sys.platform

if is_64bit is None:
is_64bit = sys.maxsize > 2 ** 32

if os_name.startswith("linux"):
libname = "libgmt.so"
libname = ["libgmt.so"]
elif os_name == "darwin":
# Darwin is macOS
libname = "libgmt.dylib"
libname = ["libgmt.dylib"]
elif os_name == "win32":
if is_64bit:
libname = "gmt_w64.dll"
else:
libname = "gmt_w32.dll"
libname = ["gmt.dll", "gmt_w64.dll", "gmt_w32.dll"]
else:
raise GMTOSError('Operating system "{}" not supported.'.format(sys.platform))
return libname
Expand Down
Loading