Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BiasCorr classes and rename previous coreg.BiasCorr in coreg.VerticalShift #158

Merged
merged 61 commits into from
Jul 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
f7f6108
Merge pull request #6 from GlacioHack/main
rhugonnet Mar 27, 2021
9c97f54
Merge branch 'GlacioHack:main' into main
rhugonnet Jun 17, 2021
f0ea514
rename BiasCorr into VerticalShift, refactor all sources and adapt do…
rhugonnet Jun 26, 2021
34dbefc
add biascorr, draft along/across track classes
rhugonnet Jun 26, 2021
bf3e1cd
update doc and NuthKaab with new VerticalShift variable naming
rhugonnet Jun 26, 2021
2c79310
move ZScaleCorr to TerrainBias, wrap Along and Across into Directiona…
rhugonnet Jun 26, 2021
497cc1a
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Sep 8, 2021
08ef092
refactor post-merge biascorr tests with vshift naming
rhugonnet Sep 8, 2021
6edc5e2
refactor remaining biascorr in vshift
rhugonnet Sep 8, 2021
a39f7d4
improve with erik comments
rhugonnet Sep 9, 2021
5122625
draft structure bias corr
rhugonnet Sep 10, 2021
38edb2e
finish biascorr structure + start tests
rhugonnet Sep 15, 2021
fb93c87
advancing bias corr
rhugonnet Sep 19, 2021
83d90c6
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Apr 23, 2022
a262e89
Remove spatial_tools import
rhugonnet Apr 23, 2022
ba48754
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Sep 20, 2022
d25e9ec
Fix linting errors
rhugonnet Sep 20, 2022
0f60e09
Fix flake8
rhugonnet Sep 20, 2022
e0e8589
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Apr 28, 2023
e0abf74
Incremental fixes on tests
rhugonnet May 7, 2023
b5b8371
Incremental commit on bias correction classes
rhugonnet May 18, 2023
215ac32
Incremental commit on biascorr
rhugonnet May 19, 2023
9338b7b
Incremental commit on biascorr
rhugonnet May 20, 2023
b350820
Fix subsampling and parameter unpacking
rhugonnet May 20, 2023
3af1028
Homogenize func parameter ungrouping, fix binning and continue tests
rhugonnet May 22, 2023
960b856
Finalize first round of tests
rhugonnet May 23, 2023
e3f9de4
Linting all but mypy
rhugonnet May 23, 2023
5fb8d6d
Incremental commit on biascorr tests
rhugonnet May 26, 2023
63b1f61
Finalize bias correction tests
rhugonnet May 26, 2023
6372348
Finalize last test
rhugonnet May 27, 2023
6840923
Linting (including mypy!)
rhugonnet May 27, 2023
8defc14
Move richdem to pip to force install
rhugonnet May 27, 2023
31fd35c
Add future annotation import in test_biascorr
rhugonnet May 28, 2023
ab3c327
Add basic documentation page and refactor coreg.Deramp into coreg.Tilt
rhugonnet May 29, 2023
b0aeb16
Linting
rhugonnet May 29, 2023
ba3bc92
Try to circumvent richdem issues
rhugonnet May 29, 2023
42130a4
Linting
rhugonnet May 29, 2023
fb83de2
Skip or ignore new terrain error, opening issue
rhugonnet May 30, 2023
db30a9d
Add random state for all relevant tests
rhugonnet May 30, 2023
c1ae323
Linting
rhugonnet May 30, 2023
a838531
Use lambda functions that converge for the tests
rhugonnet May 31, 2023
b4bcdc3
Add bin_and_fit option
rhugonnet May 31, 2023
240f38e
Linting
rhugonnet May 31, 2023
38f40f9
Fix dimensions in sumsin calculation to take any number of frequencie…
rhugonnet May 31, 2023
982dfdb
Linting
rhugonnet May 31, 2023
26aa866
Add draft gallery example and new methods to API
rhugonnet May 31, 2023
9567df2
Linting
rhugonnet May 31, 2023
91efd8b
Fix test that randomly fails
rhugonnet Jun 1, 2023
173749a
Merge branch 'main' into biascorr
rhugonnet Jun 13, 2023
427a98f
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Jun 13, 2023
0098b8f
Linting
rhugonnet Jun 13, 2023
06536fa
Merge branch 'biascorr' of https://github.com/rhugonnet/xdem into bia…
rhugonnet Jun 13, 2023
9ee8015
Eriks comments
rhugonnet Jun 20, 2023
4f73d3b
Linting
rhugonnet Jun 20, 2023
48bed27
Fix residuals
rhugonnet Jun 20, 2023
1e0c554
Re-structure coreg module following discussed plan in PR comments
rhugonnet Jul 29, 2023
94507b1
Linting
rhugonnet Jul 29, 2023
7557c82
Fix documentation
rhugonnet Jul 29, 2023
f6ab095
Fix small indent warning
rhugonnet Jul 29, 2023
8c1f861
Merge remote-tracking branch 'upstream/main' into biascorr
rhugonnet Jul 29, 2023
cb1aad5
Linting
rhugonnet Jul 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 54 additions & 10 deletions doc/source/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,35 +95,79 @@ A {class}`~xdem.DEM` inherits four unique attributes from {class}`~geoutils.Rast

## Coreg

### Coregistration object and pipeline
**Overview of co-registration class structure**:

```{eval-rst}
.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
:top-classes: xdem.Coreg
```

### Coregistration, pipeline and blockwise

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.coreg.Coreg
xdem.coreg.CoregPipeline
xdem.coreg.BlockwiseCoreg
```

### Affine coregistration methods


**Generic parent class:**

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.coreg.AffineCoreg
```

**Convenience classes for specific coregistrations:**

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.coreg.VerticalShift
xdem.coreg.NuthKaab
xdem.coreg.ICP
xdem.coreg.Tilt
```

### Bias-correction (including non-affine coregistration) methods

**Generic parent class:**

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.Coreg
xdem.CoregPipeline
xdem.coreg.BiasCorr
```

### Rigid coregistration methods
**Classes for any 1-, 2- and N-D biases:**

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.BiasCorr
xdem.NuthKaab
xdem.ICP
xdem.Deramp
xdem.coreg.BiasCorr1D
xdem.coreg.BiasCorr2D
xdem.coreg.BiasCorrND
```

### Spatial coregistration
**Convenience classes for specific corrections:**

```{eval-rst}
.. autosummary::
:toctree: gen_modules/

xdem.BlockwiseCoreg
xdem.coreg.Deramp
xdem.coreg.DirectionalBias
xdem.coreg.TerrainBias
```

## Terrain attributes
Expand Down
168 changes: 164 additions & 4 deletions doc/source/biascorr.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,173 @@
---
file_format: mystnb
jupytext:
formats: md:myst
text_representation:
extension: .md
format_name: myst
kernelspec:
display_name: xdem-env
language: python
name: xdem
---

(biascorr)=

# Bias corrections
# Bias correction

In xDEM, bias-correction methods correspond to non-rigid transformations that cannot be described as a 3-dimensional
affine function (see {ref}`coregistration`).

Contrary to rigid coregistration methods, bias corrections are not limited to the information in the DEMs. They can be
passed any external variables (e.g., land cover type, processing metric) to attempt to identify and correct biases in
the DEM. Still, many methods rely either on coordinates (e.g., deramping, along-track corrections) or terrain
(e.g., curvature- or elevation-dependant corrections), derived solely from the DEM.

## The {class}`~xdem.BiasCorr` object

Each bias-correction method in xDEM inherits their interface from the {class}`~xdem.Coreg` class (see {ref}`coreg_object`).
This implies that bias-correction methods can be combined in a {class}`~xdem.CoregPipeline` with any other methods, or
applied in a block-wise manner through {class}`~xdem.BlockwiseCoreg`.

**Inheritance diagram of co-registration and bias corrections:**

```{eval-rst}
.. inheritance-diagram:: xdem.coreg.base xdem.coreg.affine xdem.coreg.biascorr
:top-classes: xdem.Coreg
```

As a result, each bias-correction approach has the following methods:

- {func}`~xdem.BiasCorr.fit` for estimating the bias.
- {func}`~xdem.BiasCorr.apply` for correcting the bias on a DEM.

## Modular estimators

Bias-correction methods have 3 main ways of estimating and correcting a bias, both relying on one or several variables:

- **Performing a binning of the data** along variables with a statistic (e.g., median), and applying the statistics in each bin,
- **Fitting a parametric function** to the variables, and applying that function,
- **(Recommended<sup>1</sup>) Fitting a parametric function on a data binning** of the variable, and applying that function.

```{margin}
<sup>1</sup>DEM alignment is a big data problem often plagued by outliers, greatly **simplified** and **accelerated** by binning with robust estimators.
```

To define the parameters related to fitting and/or binning, every {func}`~xdem.BiasCorr` is instantiated with the same arguments:

- `fit_or_bin` to either fit a parametric model to the bias by passing "fit", perform an empirical binning of the bias by passing "bin", or to fit a parametric model to the binning with "bin_and_fit" **(recommended)**,
- `fit_func` to pass any parametric function to fit to the bias,
- `fit_optimizer` to pass any optimizer function to perform the fit minimization,
- `bin_sizes` to pass the size or edges of the bins for each variable,
- `bin_statistic` to pass the statistic to compute in each bin,
- `bin_apply_method` to pass the method to apply the binning for correction.

```{code-cell} ipython3
:tags: [hide-input, hide-output]

import geoutils as gu
import numpy as np

import xdem

# Open a reference DEM from 2009
ref_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_ref_dem"))
# Open a to-be-aligned DEM from 1990
tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem")).reproject(ref_dem, silent=True)

# Open glacier polygons from 1990, corresponding to unstable ground
glacier_outlines = gu.Vector(xdem.examples.get_path("longyearbyen_glacier_outlines"))
# Create an inlier mask of terrain outside the glacier polygons
inlier_mask = glacier_outlines.create_mask(ref_dem)
```

(biascorr-deramp)=

## Deramping

{class}`xdem.coreg.Deramp`

Bias corrections correspond to transformations that cannot be described as a 3-dimensional affine function (see {ref}`coregistration`).
- **Performs:** Correct biases with a 2D polynomial of degree N.
- **Supports weights** Yes.
- **Recommended for:** Residuals from camera model.

Deramping works by estimating and correcting for an N-degree polynomial over the entire dDEM between a reference and the DEM to be aligned.
This may be useful for correcting small rotations in the dataset, or nonlinear errors that for example often occur in structure-from-motion derived optical DEMs (e.g. Rosnell and Honkavaara [2012](https://doi.org/10.3390/s120100453); Javernick et al. [2014](https://doi.org/10.1016/j.geomorph.2014.01.006); Girod et al. [2017](https://doi.org/10.5194/tc-11827-2017)).

### Limitations

Deramping does not account for horizontal (X/Y) shifts, and should most often be used in conjunction with other methods.

1st order deramping is not perfectly equivalent to a rotational correction: values are simply corrected in the vertical direction, and therefore includes a horizontal scaling factor, if it would be expressed as a transformation matrix.
For large rotational corrections, [ICP] is recommended.

### Example

```{code-cell} ipython3
from xdem import coreg

# Instantiate a 1st order deramping
deramp = coreg.Deramp(poly_order=1)
# Fit the data to a suitable polynomial solution
deramp.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)

# Apply the transformation
corrected_dem = deramp.apply(tba_dem)
```

## Directional biases

TODO: In construction
{class}`xdem.coreg.DirectionalBias`

- **Performs:** Correct biases along a direction of the DEM.
- **Supports weights** Yes.
- **Recommended for:** Undulations or jitter, common in both stereo and radar DEMs.

The default optimizer for directional biases optimizes a sum of sinusoids using 1 to 3 different frequencies, and keeping the best performing fit.

### Example

```{code-cell} ipython3
# Instantiate a directional bias correction
dirbias = coreg.DirectionalBias(angle=65)
# Fit the data
dirbias.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)

# Apply the transformation
corrected_dem = dirbias.apply(tba_dem)
```

## Terrain biases

TODO: In construction
{class}`xdem.coreg.TerrainBias`

- **Performs:** Correct biases along a terrain attribute of the DEM.
- **Supports weights** Yes.
- **Recommended for:** Different native resolution between DEMs.

The default optimizer for terrain biases optimizes a 1D polynomial with an order from 1 to 6, and keeping the best performing fit.

### Example

```{code-cell} ipython3
# Instantiate a 1st order terrain bias correction
terbias = coreg.TerrainBias(terrain_attribute="maximum_curvature")
# Fit the data
terbias.fit(ref_dem, tba_dem, inlier_mask=inlier_mask)

# Apply the transformation
corrected_dem = terbias.apply(tba_dem)
```

## Generic 1-D, 2-D and N-D classes

All bias-corrections methods are inherited from generic classes that perform corrections in 1-, 2- or N-D. Having these
separate helps the user navigating the dimensionality of the functions, optimizer, binning or variables used.

{class}`xdem.coreg.BiasCorr1D`
{class}`xdem.coreg.BiasCorr2D`
{class}`xdem.coreg.BiasCorrND`

- **Performs:** Correct biases with any function and optimizer, or any binning, in 1-, 2- or N-D.
- **Supports weights** Yes.
- **Recommended for:** Anything.
4 changes: 3 additions & 1 deletion doc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,9 @@
"geoutils.georaster.satimg.SatelliteImage": "geoutils.SatelliteImage",
"geoutils.geovector.Vector": "geoutils.Vector",
"xdem.dem.DEM": "xdem.DEM",
"xdem.dem.Coreg": "xdem.Coreg",
"xdem.coreg.base.Coreg": "xdem.Coreg",
"xdem.coreg.affine.AffineCoreg": "xdem.AffineCoreg",
"xdem.coreg.biascorr.BiasCorr": "xdem.BiasCorr",
}

# To have an edge color that works in both dark and light mode
Expand Down
Loading