Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Forman 222 rasterize features into dataset #227

Merged
merged 5 commits into from
Nov 11, 2019
Merged

Forman 222 rasterize features into dataset #227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6a95610
modules xcube.core.geom and xcube.core.verify no longer depend on 'lo…
forman Nov 10, 2019
4311ef4
new API function rasterize_features_into_dataset()
forman Nov 10, 2019
c4ae482
module xcube.core.new no longer depend on 'lon' and 'lat' coords
forman Nov 11, 2019
2afa022
api doc update
forman Nov 11, 2019
c736756
update
forman Nov 11, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@
function that can be used to generate an output cube computed from a Python
function that is applied to one or more input cubes. Replaces the formerly
hidden `xcube apply` command. (#167)

* Added new function `xcube.core.geom.rasterize_features_into_dataset()`
to rasterize features into a dataset. (#222)

### Enhancements

Expand Down
4 changes: 2 additions & 2 deletions test/cli/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ def test_dump_ds(self):
"Attributes:\n"
" Conventions: CF-1.7\n"
" title: Test Cube\n"
" time_coverage_start: 2010-01-01 00:00:00\n"
" time_coverage_end: 2010-01-06 00:00:00\n"
" time_coverage_start: 2010-01-01T00:00:00\n"
" time_coverage_end: 2010-01-06T00:00:00\n"
" geospatial_lon_min: -180.0\n"
" geospatial_lon_max: 180.0\n"
" geospatial_lon_units: degrees_east\n"
Expand Down
10 changes: 5 additions & 5 deletions test/core/test_extract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ class GetCubeValuesForPointsTest(unittest.TestCase):
def _new_test_cube(self):
return new_cube(width=2000,
height=1000,
lon_start=0.0,
lat_start=50.0,
spatial_res=4.0 / 2000,
x_start=0.0,
y_start=50.0,
x_res=4.0 / 2000,
time_start="2010-01-01",
time_periods=20,
variables=dict(precipitation=0.6, temperature=276.2))
Expand Down Expand Up @@ -107,11 +107,11 @@ def test_get_dataset_indexes_without_bounds(self):
self._assert_get_dataset_indexes_works(dataset)

def test_get_dataset_indexes_with_bounds_inverse_lat(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def test_get_dataset_indexes_with_bounds_inverse_lat(self):
def test_get_dataset_indexes_with_bounds_inverse_y(self):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lat is fine here, because get_dataset_indexes is still aware of lat, not y.

dataset = new_cube(width=360, height=180, inverse_lat=True, drop_bounds=False)
dataset = new_cube(width=360, height=180, inverse_y=True, drop_bounds=False)
self._assert_get_dataset_indexes_works(dataset, inverse_lat=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self._assert_get_dataset_indexes_works(dataset, inverse_lat=True)
self._assert_get_dataset_indexes_works(dataset, inverse_y=True)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here


def test_get_dataset_indexes_without_bounds_inverse_lat(self):
dataset = new_cube(width=360, height=180, inverse_lat=True, drop_bounds=True)
dataset = new_cube(width=360, height=180, inverse_y=True, drop_bounds=True)
self._assert_get_dataset_indexes_works(dataset, inverse_lat=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
self._assert_get_dataset_indexes_works(dataset, inverse_lat=True)
self._assert_get_dataset_indexes_works(dataset, inverse_y=True)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here :)


def _assert_get_dataset_indexes_works(self, dataset, inverse_lat=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def _assert_get_dataset_indexes_works(self, dataset, inverse_lat=False):
def _assert_get_dataset_indexes_works(self, dataset, inverse_y=False):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here...

Expand Down
86 changes: 82 additions & 4 deletions test/core/test_geom.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,88 @@
import shapely.geometry
import xarray as xr

from xcube.core.new import new_cube
from xcube.core.chunk import chunk_dataset
from xcube.core.geom import get_dataset_geometry, get_dataset_bounds, get_geometry_mask, convert_geometry, \
mask_dataset_by_geometry, clip_dataset_by_geometry
from xcube.core.geom import clip_dataset_by_geometry
from xcube.core.geom import convert_geometry
from xcube.core.geom import get_dataset_bounds
from xcube.core.geom import get_dataset_geometry
from xcube.core.geom import get_geometry_mask
from xcube.core.geom import mask_dataset_by_geometry
from xcube.core.geom import rasterize_features_into_dataset
from xcube.core.new import new_cube


class RasterizeFeaturesIntoDataset(unittest.TestCase):
def test_rasterize_features_into_dataset_lonlat(self):
self._test_rasterize_features_into_dataset('lon', 'lat')

def test_rasterize_features_into_dataset_xy(self):
self._test_rasterize_features_into_dataset('x', 'y')

def _test_rasterize_features_into_dataset(self, x_name, y_name):
dataset = new_cube(width=10, height=10, x_name=x_name, y_name=y_name, x_res=10, x_start=-50, y_start=-50)
feature1 = dict(type='Feature',
geometry=dict(type='Polygon',
coordinates=[[(-180, 0), (-1, 0), (-1, 90), (-180, 90), (-180, 0)]]),
properties=dict(a=0.5, b=2.1, c=9))
feature2 = dict(type='Feature',
geometry=dict(type='Polygon',
coordinates=[[(-180, -90), (-1, -90), (-1, 0), (-180, 0), (-180, -90)]]),
properties=dict(a=0.6, b=2.2, c=8))
feature3 = dict(type='Feature',
geometry=dict(type='Polygon',
coordinates=[[(20, 0), (180, 0), (180, 90), (20, 90), (20, 0)]]),
properties=dict(a=0.7, b=2.3, c=7))
feature4 = dict(type='Feature',
geometry=dict(type='Polygon',
coordinates=[[(20, -90), (180, -90), (180, 0), (20, 0), (20, -90)]]),
properties=dict(a=0.8, b=2.4, c=6))
dataset = rasterize_features_into_dataset(dataset,
[feature1, feature2, feature3, feature4],
['a', 'b', 'c'],
var_properties=dict(
b=('b', np.float32, np.nan, dict(units='meters')),
c=('c2', np.uint8, 0, None),
),
in_place=False)
self.assertIsNotNone(dataset)
self.assertIn(x_name, dataset.coords)
self.assertIn(y_name, dataset.coords)
self.assertIn('time', dataset.coords)
self.assertIn('a', dataset)
self.assertIn('b', dataset)
self.assertIn('c2', dataset)
self.assertEquals((10, 10), dataset.a.shape)
self.assertEquals((10, 10), dataset.b.shape)
self.assertEquals((10, 10), dataset.c2.shape)
self.assertEquals(np.float64, dataset.a.dtype)
self.assertEquals(np.float32, dataset.b.dtype)
self.assertEquals(np.uint8, dataset.c2.dtype)
self.assertEquals({}, dataset.a.attrs)
self.assertEquals({'units': 'meters'}, dataset.b.attrs)
self.assertEquals({}, dataset.c2.attrs)
self.assertEquals((y_name, x_name), dataset.a.dims)
self.assertEquals((y_name, x_name), dataset.b.dims)
self.assertEquals((y_name, x_name), dataset.c2.dims)
self.assertEquals(np.array(0.5, dtype=np.float64), dataset.a.min())
self.assertEquals(np.array(0.8, dtype=np.float64), dataset.a.max())
self.assertEquals(np.array(2.1, dtype=np.float32), dataset.b.min())
self.assertEquals(np.array(2.4, dtype=np.float32), dataset.b.max())
self.assertEquals(np.array(0, dtype=np.uint8), dataset.c2.min())
self.assertEquals(np.array(9, dtype=np.uint8), dataset.c2.max())
nan = np.nan
np.testing.assert_almost_equal(
np.array([[0.5, 0.5, 0.5, 0.5, 0.5, nan, nan, 0.7, 0.7, 0.7],
[0.5, 0.5, 0.5, 0.5, 0.5, nan, nan, 0.7, 0.7, 0.7],
[0.5, 0.5, 0.5, 0.5, 0.5, nan, nan, 0.7, 0.7, 0.7],
[0.5, 0.5, 0.5, 0.5, 0.5, nan, nan, 0.7, 0.7, 0.7],
[0.5, 0.5, 0.5, 0.5, 0.5, nan, nan, 0.7, 0.7, 0.7],
[0.6, 0.6, 0.6, 0.6, 0.6, nan, nan, 0.8, 0.8, 0.8],
[0.6, 0.6, 0.6, 0.6, 0.6, nan, nan, 0.8, 0.8, 0.8],
[0.6, 0.6, 0.6, 0.6, 0.6, nan, nan, 0.8, 0.8, 0.8],
[0.6, 0.6, 0.6, 0.6, 0.6, nan, nan, 0.8, 0.8, 0.8],
[0.6, 0.6, 0.6, 0.6, 0.6, nan, nan, 0.8, 0.8, 0.8]]),
dataset.a.values)


class DatasetGeometryTest(unittest.TestCase):
Expand All @@ -28,7 +106,7 @@ def setUp(self) -> None:

self.cube = new_cube(width=width,
height=height,
spatial_res=spatial_res,
x_res=spatial_res,
drop_bounds=True,
variables=dict(temp=273.9, precip=0.9))

Expand Down
50 changes: 45 additions & 5 deletions test/core/test_schema.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,15 @@ def _assert_schema(self, schema: CubeSchema, expected_shape=None, expected_chunk
self.assertIn('lon', schema.coords)
self.assertEqual(('time', 'lat', 'lon'), schema.dims)
self.assertEqual('time', schema.time_dim)
self.assertEqual(('lon', 'lat'), schema.spatial_dims)
self.assertEqual('lon', schema.x_name)
self.assertEqual('lat', schema.y_name)
self.assertEqual('time', schema.time_name)
self.assertEqual('lon', schema.x_dim)
self.assertEqual('lat', schema.y_dim)
self.assertEqual('time', schema.time_dim)
self.assertEqual(expected_shape[-1], schema.x_var.size)
self.assertEqual(expected_shape[-2], schema.y_var.size)
self.assertEqual(expected_shape[0], schema.time_var.size)

def test_repr_html(self):
cube = new_cube(variables=dict(a=2, b=3, c=4))
Expand Down Expand Up @@ -64,6 +72,24 @@ def test_constructor_with_invalid_args(self):
self.assertEqual('coords must be a mapping from dimension names to label arrays',
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
# noinspection PyTypeChecker
CubeSchema(schema.shape, cube.coords, x_name=None)
self.assertEqual('x_name must be given',
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
# noinspection PyTypeChecker
CubeSchema(schema.shape, cube.coords, y_name=None)
self.assertEqual('y_name must be given',
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
# noinspection PyTypeChecker
CubeSchema(schema.shape, cube.coords, time_name=None)
self.assertEqual('time_name must be given',
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
CubeSchema(schema.shape[1:], schema.coords)
self.assertEqual('shape must have at least three dimensions',
Expand All @@ -76,12 +102,12 @@ def test_constructor_with_invalid_args(self):

with self.assertRaises(ValueError) as cm:
CubeSchema(schema.shape, schema.coords, dims=('lat', 'lon', 'time'))
self.assertEqual("the first name in dims must be 'time'",
self.assertEqual("the first dimension in dims must be 'time'",
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
CubeSchema(schema.shape, schema.coords, dims=('time', 'lon', 'lat'))
self.assertEqual("the last two names in dims must be either ('lat', 'lon') or ('y', 'x')",
self.assertEqual("the last two dimensions in dims must be 'lat' and 'lon'",
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
Expand All @@ -93,15 +119,15 @@ def test_constructor_with_invalid_args(self):
coords = dict(schema.coords)
del coords['lat']
CubeSchema(schema.shape, coords, dims=schema.dims, chunks=(1, 90, 90))
self.assertEqual("missing dimension 'lat' in coords",
self.assertEqual("missing variables 'lon', 'lat', 'time' in coords",
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
coords = dict(schema.coords)
lat = coords['lat']
coords['lat'] = xr.DataArray(lat.values.reshape((1, len(lat))), dims=('b', lat.dims[0]), attrs=lat.attrs)
CubeSchema(schema.shape, coords, dims=schema.dims, chunks=(1, 90, 90))
self.assertEqual("labels of 'lat' in coords must be one-dimensional",
self.assertEqual("variables 'lon', 'lat', 'time' in coords must be 1-D",
f'{cm.exception}')

with self.assertRaises(ValueError) as cm:
Expand All @@ -119,6 +145,20 @@ def test_new_with_cube(self):
self.assertEqual("cube is empty",
f'{cm.exception}')

cube = new_cube()
del cube.coords['lon']
with self.assertRaises(ValueError) as cm:
CubeSchema.new(cube)
self.assertEqual("cube has no valid spatial coordinate variables",
f'{cm.exception}')

cube = new_cube()
del cube.coords['time']
with self.assertRaises(ValueError) as cm:
CubeSchema.new(cube)
self.assertEqual("cube has no valid time coordinate variable",
f'{cm.exception}')

cube = new_cube(variables=dict(a=1, b=2))
cube['c'] = xr.DataArray(np.array([1, 2, 3, 4, 5]), dims=('q',))
with self.assertRaises(ValueError) as cm:
Expand Down
16 changes: 7 additions & 9 deletions test/core/test_verify.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,7 @@ def test_assert_cube_illegal_coord_var(self):
with self.assertRaises(ValueError) as cm:
assert_cube(cube)
self.assertEqual("Dataset is not a valid xcube dataset, because:\n"
"- coordinate variable 'lat' must have a single dimension 'lat';\n"
"- coordinate variable 'lon' must have a single dimension 'lon'.",
"- missing spatial x,y coordinate variables.",
f"{cm.exception}")

def test_assert_cube_illegal_coord_bounds_var(self):
Expand All @@ -41,12 +40,11 @@ def test_assert_cube_illegal_coord_bounds_var(self):
with self.assertRaises(ValueError) as cm:
assert_cube(cube)
self.assertEqual("Dataset is not a valid xcube dataset, because:\n"
"- bounds coordinate variable 'lat_bnds' must have dimensions ('lat', <bounds_dim>);\n"
"- shape of bounds coordinate variable 'lat_bnds' must be (180, 2) but was (5, 180, 2);\n"
"- bounds coordinate variable 'lon_bnds' must have dimensions ('lon', <bounds_dim>);\n"
"- shape of bounds coordinate variable 'lon_bnds' must be (360, 2) but was (5, 360, 2);\n"
"- type of bounds coordinate variable 'lon_bnds' must be dtype('float64')"
" but was dtype('float16').",
"- type of bounds coordinate variable 'lon_bnds' must be dtype('float64') but was dtype('float16');\n"
"- bounds coordinate variable 'lat_bnds' must have dimensions ('lat', <bounds_dim>);\n"
"- shape of bounds coordinate variable 'lat_bnds' must be (180, 2) but was (5, 180, 2).",
f"{cm.exception}")

def test_assert_cube_illegal_data_var(self):
Expand All @@ -69,7 +67,7 @@ def test_verify_cube(self):
cube = new_cube()
self.assertEqual([], verify_cube(cube))
ds = cube.drop("time")
self.assertEqual(["missing coordinate variable 'time'"], verify_cube(ds))
self.assertEqual(["missing time coordinate variable"], verify_cube(ds))
ds = ds.drop("lat")
self.assertEqual(["missing coordinate variable 'time'",
"missing coordinate variable 'lat'"], verify_cube(ds))
self.assertEqual(["missing spatial x,y coordinate variables",
"missing time coordinate variable"], verify_cube(ds))
Loading