Skip to content

Commit

Permalink
Use preprocessed OGGM data and migrate to pure Julia instead of relyi…
Browse files Browse the repository at this point in the history
…ng on PyCall (#65)

* Add version number to all dependencies

* Update CI.yml with new micromamba version

* Update environment.yml

* Changing python imports

* Python environment working in local - testing CI

* Update CI.yml - Testing in multiple versions & OS

* Update CI.yml

* Remove cache environments

* Remove unnecessary `which python`

* Update CI.yml - bring some specific os info

* Rename environment

* Update CI.yml - Ask more network variables

* Update CI.yml - typo in NetworkOptions

* Ping path for Mac - not working for Ubuntu

* Update CI.yml - checking path

* Update CI.yml

* Update CI.yml

* Update CI.yml

* Update CI.yml

* Update CI.yml

* Update config.jl - remove catch warning

* Update CI.yml - Now maybe exporiting global env variable

* Update CI.yml - Fix export on environmnet names

* Update CI.yml - bump micromamba version

* testing idea from the discourse

* Update CI.yml

* Update action version of `checkout` and `setup-python`

* Update CI.yml - fix typo with SSH -> SSL

* Update CI.yml - export SSL_CERT_FILE

* Update CI.yml - Change Python path

* Update CI.yml

* Update CI.yml

* Update CI.yml - Trying to reproduce past behaviour

* Update CI.yml

* Update config.jl - remove try's

* Update CI.yml - Julia 1.11

* Update CI.yml - remove Python setup and let this to micromamba

* Update CI.yml - conda -> micromamba for environment inspection

* Remove some redundancy and things that may fail

* Update CI.yml

* Update CI.yml

* [WIP] Removing Python dependencies and migrating to Rasters.jl

Ongoing fix, still a few lines missing, and tests have still not been run.

* Remove environment.yml file

* [WIP] continue the migration, tests of params_construction.jl are passing

* Migrate to pure julia (#62)

* migrate all 2D files to purejulia

* update test references

* remove PyCall dependencies in the 1D files

* automatic download of preprocessed data + remove DataFrames dependency

* try to remove Python from the CI

* remove remaining implicit dependencies to Python objects

* create get_rgi_paths

* fix various bugs

* remove absolute paths from get_rgi_paths to make the tests platform agnostic

* fix bugs with the climate_2D_step structure

* some cleaning

* bump version

* fix dependencies after rebase

---------

Co-authored-by: Facundo Sapienza <fsapienza@berkeley.edu>
Co-authored-by: Jordi <jordi.bolibar@gmail.com>
  • Loading branch information
3 people authored Jan 23, 2025
1 parent 15f3942 commit a539180
Show file tree
Hide file tree
Showing 28 changed files with 425 additions and 695 deletions.
124 changes: 81 additions & 43 deletions .github/workflows/CI.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,66 +21,104 @@ jobs:
fail-fast: false
matrix:
version:
- '1.10'
python:
- '3.12'
os:
- ubuntu-latest
- '1.11'
# python:
# - 3.9
# - 3.12
os:
# - ubuntu-latest
- macos-latest
arch:
- x64
steps:
# - name: Cancel ongoing test runs for previous commits
# uses: styfle/cancel-workflow-action@0.12.1
# with:
# access_token: ${{ github.token }}
- uses: actions/checkout@v4
- uses: julia-actions/setup-julia@latest
# - name: Set up Python 🐍 ${{ matrix.python }}
# uses: actions/setup-python@v5
# with:
# python-version: ${{ matrix.python }}
# - name: Create environment with micromamba 🐍🖤
# uses: mamba-org/setup-micromamba@v1
# with:
# micromamba-version: '2.0.2-2'
# environment-file: ./environment.yml
# environment-name: oggm_env # it is recommendable to add both name and yml file.
# init-shell: bash
# cache-environment: false
# cache-downloads: false
# # If necessary, we can include .condarc to configure environment
# # condarc-file: ./condarc.yml
# - name: Test creation of environment with micromamba 🔧🐍🖤
# run: |
# which python
# micromamba env export
# # shell: bash -el {0}
# - name: Update certifi
# run: |
# pip install --upgrade certifi
# # shell: bash -el {0}
# - name: Set ENV Variables for PyCall.jl 🐍 📞
# run: |
# echo "PYTHON=/Users/runner/micromamba/envs/oggm_env/bin/python" >> "$GITHUB_ENV"
# # shell: bash -el {0}
- uses: julia-actions/setup-julia@v1
with:
version: ${{ matrix.version }}
arch: ${{ matrix.arch }}
- name: Install dependencies on Ubuntu
if: matrix.os == 'ubuntu-latest'
run: |
sudo apt-get update
sudo apt-get install -y libxml2 libxml2-dev libspatialite7 libspatialite-dev
- name: Install dependencies on macOS
if: matrix.os == 'macos-latest'
run: |
brew install libxml2 libspatialite
- uses: julia-actions/cache@v2
with:
cache-registries: "false"
cache-compiled: "false"
# - uses: actions/cache@v4
# env:
# cache-name: cache-artifacts
# with:
# path: ~/.julia/artifacts
# key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
# restore-keys: |
# ${{ runner.os }}-test-${{ env.cache-name }}-
# ${{ runner.os }}-test-
# ${{ runner.os }}-
- uses: julia-actions/julia-buildpkg@v1
- name: Check Julia SSL certifications 🔎🔐
run: |
julia -e 'using NetworkOptions; println(NetworkOptions.bundled_ca_roots()); println(NetworkOptions.ca_roots()); println(NetworkOptions.ca_roots_path()); println(NetworkOptions.ssh_key_path()); println(NetworkOptions.ssh_key_name()); println(NetworkOptions.ssh_pub_key_path())'
julia -e 'using NetworkOptions; println(NetworkOptions.bundled_ca_roots()); println(NetworkOptions.ca_roots_path()); println(NetworkOptions.ssh_key_path()); println(NetworkOptions.ssh_key_name()); println(NetworkOptions.ssh_pub_key_path())'
# echo "SSL_CERT_PATH=$(julia -e 'using NetworkOptions; println(NetworkOptions.bundled_ca_roots())')" >> "$GITHUB_ENV"
# shell: bash -el {0}
# - name: Install dependencies on Ubuntu
# if: matrix.os == 'ubuntu-latest'
# run: |
# sudo apt-get update
# sudo apt-get install -y libxml2 libxml2-dev libspatialite7 libspatialite-dev
# echo "LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH" >> "$GITHUB_ENV"
# - name: Install dependencies on macOS
# if: matrix.os == 'macos-latest'
# run: |
# brew install libxml2 libspatialite
# echo "PKG_CONFIG_PATH=/opt/homebrew/opt/libxml2/lib/pkgconfig" >> "$GITHUB_ENV"
# - name: Check that new paths had been exported
# if: matrix.os == 'macos-latest'
# run: |
# echo $PYTHON
# echo $PKG_CONFIG_PATH
# echo $SSL_CERT_FILE
- uses: julia-actions/cache@v1
with:
cache-registries: "true"
cache-compiled: "true"
- name: Build Julia packages in Ubuntu
uses: julia-actions/julia-buildpkg@v1
if: matrix.os == 'ubuntu-latest'
# env:
# PYTHON : /Users/runner/micromamba/envs/oggm_env/bin/python
# # The SSL certificate path can be readed from the action "Check Julia SSL certifications"
# # JULIA_SSL_CA_ROOTS_PATH: /etc/ssl/certs/ca-certificates.crt
# SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
- name: Build Julia packages in MacOS
uses: julia-actions/julia-buildpkg@v1
if: matrix.os == 'macos-latest'
# env:
# PYTHON : /Users/runner/micromamba/envs/oggm_env/bin/python
# JULIA_SSL_CA_ROOTS_PATH: /Users/runner/hostedtoolcache/julia/1.11.1/x64/share/julia/cert.pem
# SSL_CERT_FILE: /Users/runner/hostedtoolcache/julia/1.11.1/x64/share/julia/cert.pem
- name: Run tests in Ubuntu
uses: julia-actions/julia-runtest@v1
if: matrix.os == 'ubuntu-latest'
env:
# Specify the file or directory containing the certificate authority roots. See NetworkOptions.ca_roots
JULIA_SSL_CA_ROOTS_PATH: /opt/hostedtoolcache/julia/1.10.7/x64/share/julia/cert.pem
# The default value returned by ca_roots_path() may be overridden by setting the JULIA_SSL_CA_ROOTS_PATH, SSL_CERT_DIR, or SSL_CERT_FILE environment variables,
SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
# env:
# PYTHON : /Users/runner/micromamba/envs/oggm_env/bin/python
# # JULIA_SSL_CA_ROOTS_PATH: /etc/ssl/certs/ca-certificates.crt
# SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
- name: Run tests in MacOS
uses: julia-actions/julia-runtest@v1
if: matrix.os == 'macos-latest'
env:
JULIA_SSL_CA_ROOTS_PATH: /Users/runner/hostedtoolcache/julia/1.10.7/x64/share/julia/cert.pem
SSL_CERT_FILE: /Users/runner/hostedtoolcache/julia/1.10.7/x64/share/julia/cert.pem
# env:
# PYTHON : /Users/runner/micromamba/envs/oggm_env/bin/python
# JULIA_SSL_CA_ROOTS_PATH: /Users/runner/hostedtoolcache/julia/1.11.1/x64/share/julia/cert.pem
# SSL_CERT_FILE: /Users/runner/hostedtoolcache/julia/1.11.1/x64/share/julia/cert.pem
- uses: julia-actions/julia-processcoverage@v1
- uses: codecov/codecov-action@v4
with:
Expand Down
28 changes: 15 additions & 13 deletions Project.toml
Original file line number Diff line number Diff line change
@@ -1,39 +1,41 @@
name = "Sleipnir"
uuid = "f5e6c550-199f-11ee-3608-394420200519"
uuid = "10baed72-45ec-4fdd-b59b-ebd9654d36be"
authors = ["Jordi Bolibar <jordi.bolibar@gmail.com>", "Facundo Sapienza <fsapienza@berkeley.edu>"]
version = "0.7.0"
version = "0.7.1"

[deps]
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
CodecZlib = "944b1d66-785c-5afd-91f1-9de20f533193"
ComponentArrays = "b0b7db55-cfe3-40fc-9ded-d10e2dbeff66"
CondaPkg = "992eb4ea-22a4-4c89-a5bb-47a3300528ab"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
Downloads = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
HDF5 = "f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f"
Infiltrator = "5903a43b-9cc3-4c30-8d17-598619ec4e9b"
JLD2 = "033835bb-8acc-5ee8-8aae-3f567f8a3819"
Libdl = "8f399da3-3557-5675-b5ff-fb832c97cbdb"
OpenSSL_jll = "458c3c95-2e84-50aa-8efc-19380b2a3a95"
JSON = "682c06a0-de6a-54ab-a142-c8b1cf79cde6"
NCDatasets = "85f8d34a-cbdd-5861-8df4-14fed0d494ab"
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
PreferenceTools = "ba661fbb-e901-4445-b070-854aec6bfbc5"
PythonCall = "6099a3de-0909-46bc-b1f4-468b9a2dfc0d"
Rasters = "a3a2b9e3-a471-40c9-b274-f788e487c689"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
Tar = "a4e569a6-e804-4fa4-b0f3-eef7a1d5b13e"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[compat]
CairoMakie = "0.11"
CondaPkg = "0.2.24"
CSV = "0.10.15"
CodecZlib = "0.7.6"
Downloads = "1"
HDF5 = "0.17"
Infiltrator = "1"
JLD2 = "0.4"
PreferenceTools = "0.1.2"
PythonCall = "0.9.23"
JSON = "0.21.4"
NCDatasets = "0.14.6"
Rasters = "0.13.0"
Revise = "3"
Statistics = "1"
julia = "1.7"
Tar = "1.10.0"
julia = "1.9"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
Expand Down
Binary file modified data/missing_glaciers.jld2
Binary file not shown.
32 changes: 7 additions & 25 deletions src/Sleipnir.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,38 +15,20 @@ using CairoMakie
using Downloads
using HDF5
using ComponentArrays

include("setup/ssl.jl")
using PythonCall, CondaPkg
using Rasters
using CSV
using JSON
using CodecZlib
using Tar
import NCDatasets

# ##############################################
# ############ PARAMETERS ###############
# ##############################################

cd(@__DIR__)
const global root_dir::String = dirname(Base.current_project())

# ##############################################
# ############ PYTHON LIBRARIES ##############
# ##############################################

# We define empty objects for the Python packages
const netCDF4 = Ref{Py}()
const cfg = Ref{Py}()
const utils = Ref{Py}()
const workflow = Ref{Py}()
const tasks = Ref{Py}()
const global_tasks = Ref{Py}()
const graphics = Ref{Py}()
const bedtopo = Ref{Py}()
const millan22 = Ref{Py}()
const MBsandbox = Ref{Py}()
const salem = Ref{Py}()

# Essential Python libraries
const xr = Ref{Py}()
const rioxarray = Ref{Py}()
const pd = Ref{Py}()
const global prepro_dir::String = joinpath(homedir(), ".ODINN", "ODINN_prepro")

# ##############################################
# ########## SLEIPNIR LIBRARIES ##############
Expand Down
30 changes: 20 additions & 10 deletions src/glaciers/climate/Climate1D.jl
Original file line number Diff line number Diff line change
@@ -1,23 +1,33 @@

@kwdef mutable struct Climate1Dstep{F <: AbstractFloat}
@kwdef mutable struct Climate1Dstep{F <: AbstractFloat}
temp::Vector{F}
PDD::Vector{F}
snow::Vector{F}
rain::Vector{F}
gradient::Ref{F}
avg_gradient::Ref{F}
gradient::F
avg_gradient::F
x::Vector{F}
y::Vector{F}
ref_hgt::Ref{F}
ref_hgt::F
end

@kwdef mutable struct Climate1D{F <: AbstractFloat}
raw_climate::Py # Raw climate dataset for the whole simulation
Base.:(==)(a::Climate1Dstep, b::Climate1Dstep) = a.temp == b.temp && a.PDD == b.PDD &&
a.snow == b.snow && a.rain == b.rain &&
a.gradient == b.gradient && a.avg_gradient == b.avg_gradient &&
a.x == b.x && a.y == b.y && a.ref_hgt == b.ref_hgt

@kwdef mutable struct Climate1D{F <: AbstractFloat}
raw_climate::RasterStack # Raw climate dataset for the whole simulation
# Buffers to avoid memory allocations
climate_raw_step::Ref{Py} # Raw climate trimmed for the current step
climate_step::Ref{Py} # Climate data for the current step
climate_raw_step::RasterStack # Raw climate trimmed for the current step
climate_step::Dict # Climate data for the current step
climate_2D_step::Climate2Dstep # 2D climate data for the current step to feed to the MB model
longterm_temps::Vector{F} # Longterm temperatures for the ice rheology
avg_temps::Ref{Py} # Intermediate buffer for computing average temperatures
avg_gradients::Ref{Py} # Intermediate buffer for computing average gradients
avg_temps::F # Intermediate buffer for computing average temperatures
avg_gradients::F # Intermediate buffer for computing average gradients
end

Base.:(==)(a::Climate1D, b::Climate1D) = a.raw_climate == b.raw_climate && a.climate_raw_step == b.climate_raw_step &&
a.climate_step == b.climate_step && a.climate_2D_step == b.climate_2D_step &&
a.longterm_temps == b.longterm_temps && a.avg_temps == b.avg_temps &&
a.avg_gradients == b.avg_gradients
30 changes: 20 additions & 10 deletions src/glaciers/climate/Climate2D.jl
Original file line number Diff line number Diff line change
@@ -1,25 +1,35 @@

export Climate2Dstep, Climate2D

@kwdef mutable struct Climate2Dstep{F <: AbstractFloat}
@kwdef mutable struct Climate2Dstep{F <: AbstractFloat}
temp::Matrix{F}
PDD::Matrix{F}
snow::Matrix{F}
rain::Matrix{F}
gradient::Ref{F}
avg_gradient::Ref{F}
gradient::F
avg_gradient::F
x::Vector{F}
y::Vector{F}
ref_hgt::Ref{F}
ref_hgt::F
end

@kwdef mutable struct Climate2D{F <: AbstractFloat}
raw_climate::Py # Raw climate dataset for the whole simulation
Base.:(==)(a::Climate2Dstep, b::Climate2Dstep) = a.temp == b.temp && a.PDD == b.PDD &&
a.snow == b.snow && a.rain == b.rain &&
a.gradient == b.gradient && a.avg_gradient == b.avg_gradient &&
a.x == b.x && a.y == b.y && a.ref_hgt == b.ref_hgt

@kwdef mutable struct Climate2D{F <: AbstractFloat}
raw_climate::RasterStack # Raw climate dataset for the whole simulation
# Buffers to avoid memory allocations
climate_raw_step::Ref{Py} # Raw climate trimmed for the current step
climate_step::Ref{Py} # Climate data for the current step
climate_raw_step::RasterStack # Raw climate trimmed for the current step
climate_step::Dict # Climate data for the current step
climate_2D_step::Climate2Dstep # 2D climate data for the current step to feed to the MB model
longterm_temps::Vector{F} # Longterm temperatures for the ice rheology
avg_temps::Ref{Py} # Intermediate buffer for computing average temperatures
avg_gradients::Ref{Py} # Intermediate buffer for computing average gradients
avg_temps::F # Intermediate buffer for computing average temperatures
avg_gradients::F # Intermediate buffer for computing average gradients
end

Base.:(==)(a::Climate2D, b::Climate2D) = a.raw_climate == b.raw_climate && a.climate_raw_step == b.climate_raw_step &&
a.climate_step == b.climate_step && a.climate_2D_step == b.climate_2D_step &&
a.longterm_temps == b.longterm_temps && a.avg_temps == b.avg_temps &&
a.avg_gradients == b.avg_gradients
Loading

0 comments on commit a539180

Please sign in to comment.