z error when reading geojson data

[yeedle]

I'm trying to read the nytimes' elections dataset by using st_read. After downloading and extracting the file, I get the following error (sf version 0.9-7):

> data <- read_sf("~Downloads/precincts-with-results.geojson")
Error in CPL_get_z_range(obj, 3) : z error - expecting three columns;

System details

platform x86_64-pc-linux-gnu

arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 6.3
year 2020
month 02
day 29
svn rev 77875
language R
version.string R version 3.6.3 (2020-02-29)
nickname Holding the Windsock

sessionInfo()

R version 3.6.3 (2020-02-29) Platform: x86_64-pc-linux-gnu (64-bit) Running under: elementary OS 0.4.1 Loki

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
[1] LC_CTYPE=en_US.UTF-8
[2] LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8
[8] LC_NAME=C
[9] LC_ADDRESS=C
[10] LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8
[12] LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets
[6] methods base

other attached packages:
[1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.2
[4] purrr_0.3.4 readr_1.4.0 tidyr_1.1.2
[7] tibble_3.0.4 ggplot2_3.3.2 tidyverse_1.3.0
[10] sf_0.9-7

loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 cellranger_1.1.0
[3] pillar_1.4.7 compiler_3.6.3
[5] dbplyr_1.4.4 class_7.3-17
[7] tools_3.6.3 lubridate_1.7.9
[9] jsonlite_1.7.2 lifecycle_0.2.0
[11] gtable_0.3.0 pkgconfig_2.0.3
[13] rlang_0.4.9 reprex_0.3.0
[15] DBI_1.1.1 cli_2.2.0
[17] rstudioapi_0.13 haven_2.3.1
[19] e1071_1.7-4 withr_2.3.0
[21] xml2_1.3.2 httr_1.4.2
[23] fs_1.5.0 hms_0.5.3
[25] generics_0.0.2 vctrs_0.3.6
[27] classInt_0.4-3 grid_3.6.3
[29] tidyselect_1.1.0 glue_1.4.2
[31] R6_2.5.0 fansi_0.4.1
[33] readxl_1.3.1 modelr_0.1.8
[35] blob_1.2.1 magrittr_2.0.1
[37] backports_1.2.0 scales_1.1.1
[39] ellipsis_0.3.1 units_0.6-7
[41] rvest_0.3.6 assertthat_0.2.1
[43] colorspace_2.0-0 KernSmooth_2.23-17
[45] stringi_1.5.3 munsell_0.5.0
[47] broom_0.7.1 crayon_1.3.4

[edzer]

@dcooley ?

[dcooley]

yep, I get this error too. I'll take a look in the morning (Australian time).

In the meantime, this works

data <- geojsonsf::geojson_sf("~/Downloads/precincts-with-results.geojson")

[dcooley]

Actually, I've had a quick look, and there are some geometries with a z property

df <- sfheaders::sf_to_df(
  sf = sf
)

library(data.table)
setDT( df )

df[ !is.na( z ), ]
        sfg_id multipolygon_id polygon_id linestring_id         x        y z
     1:  33209              NA      33209             1 -95.42749 37.96577 0
     2:  33209              NA      33209             1 -95.42737 37.96577 0
     3:  33209              NA      33209             1 -95.42738 37.96657 0
     4:  33209              NA      33209             1 -95.42740 37.96932 0
     5:  33209              NA      33209             1 -95.42740 37.97120 0
    ---                                                                     
933443:  37279              NA      37279             1 -97.37149 37.56692 0
933444:  37279              NA      37279             1 -97.37200 37.56692 0
933445:  37279              NA      37279             1 -97.37199 37.56791 0
933446:  37279              NA      37279             1 -97.37149 37.56791 0
933447:  37279              NA      37279             1 -97.37149 37.56750 0

But they're all 0

> df[ !is.na( z ) & z > 0 ]
Empty data.table (0 rows and 7 cols): sfg_id,multipolygon_id,polygon_id,linestring_id,x,y...

---

And here is a minimally reproducible example

sf::st_sfc(
  list(
    sf::st_point(c(0,0))
    , sf::st_point(c(1,1,1))
  )
)

Error in CPL_get_z_range(unclass(obj)[sel], 0) : 
  z error - expecting three coordinates

[dcooley]

@edzer

With this example

sf::st_sfc(
  list(
    sf::st_point(c(0,0))
    , sf::st_point(c(1,1,1))
  )
)

If I let it pass through the CPL_get_z_range() C++ functions, it ends up hitting this stop point

# set n_empty, check XY* is uniform:
if (is.null(attr(lst, "n_empty")) || any(is_null)) { # n_empty is set by CPL_read_wkb:
	attr(lst, "n_empty") = sum(vapply(lst, sfg_is_empty, TRUE))
	if (length(u <- unique(sfg_classes[1L,])) > 1)
		stop(paste("found multiple dimensions:", paste(u, collapse = " ")))
}

For examle

sf::st_sfc(
	list(
		sf::st_point(c(0,0))
		, sf::st_point(c(1,1,1))
	)
 )
Error in sf::st_sfc(list(sf::st_point(c(0, 0)), sf::st_point(c(1, 1, 1)))) : 
  found multiple dimensions: XY XYZ

But the read_sf() example will work

data <- read_sf("~/Downloads/precincts-with-results.geojson")
nrow(data)
[1] 109363

Is it perhaps best to allow multiple dimensions, given the various st_ functions can handle them?

For example

geo <- '
{
	"type":"FeatureCollection",
	"features": [
	{
		"type":"Feature",
		"geometry": {
			"type": "LineString",
			"coordinates": [[0,0],[1,1]]
		}
	},
	{
		"type":"Feature",
			"geometry": {
				"type": "LineString",
				"coordinates": [[0,0,0],[1,1,0]]
			}
	}
	]
}'


sf <- st_read(geo)
sf::st_length( sf )

Units: [m]
[1] 156899.6 110574.4

(The next thing for me to understand is why adding in the Z coordinate here creates a differnet 'length').

[edzer]

Thanks for looking into this! I'd like to see st_read succeed in such cases, best would be with a warning and a suggestion how to clean up the mess.

[dcooley]

ok, I'll pull together a PR for review.

[yeedle]

I remember running into a similar issue in the past and I solved it with st_zm (which is obviously not possible with st_read.)

And thanks for the amazing work you guys are doing!

[edzer]

@dcooley any news on this one?

[dcooley]

Hi, yeah, sorry - got sidetracked with something I'm working on.

I think a limiting factor is the current requirement to have the dimension uniform for all geometries.

How do you think this should be handled?

[edzer]

The purist in me would say: leave this an error, let the user solve the problem, and let the user try again after solving it. But then: how should she solve it? Before we computed z & m ranges (and generated errors), an object was returned with mixed dimensions, raised a warning, and pointed to st_zm() to correct this (remove all z, m, or zm geometries). Would it be doable to go back to that state?

[edzer]

@dcooley the requirement you refer to is skipped when we come back from CPL_read_wkb, i.e. when we read data from file.

[dcooley]

Yes, for example this works

geo <- '{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"id":1.0},"geometry":{"type":"Point","coordinates":[1.0,2.0]}},{"type":"Feature","properties":{"id":1.0},"geometry":{"type":"LineString","coordinates":[[1.0,2.0,3.0,4.0]]}}]}'

sf <- sf::st_read( geo )

sf
# Simple feature collection with 2 features and 1 field
# geometry type:  GEOMETRY
# dimension:      XY
# bbox:           xmin: 1 ymin: 2 xmax: 1 ymax: 2
# z_range:        zmin: 3 zmax: 3
# geographic CRS: WGS 84
# id             geometry
# 1  1          POINT (1 2)
# 2  1 LINESTRING Z (1 2 3)

However, the dimension attributed is printed as XY, even though there are mixed dimensions.

Is it worth considering removing this uniform check, and perhaps printing the different dimensions that exist, something like

sf
# Simple feature collection with 2 features and 1 field
# geometry type:  GEOMETRY
# dimension:      XY, XYZ
# bbox:           xmin: 1 ymin: 2 xmax: 1 ymax: 2
# z_range:        zmin: 3 zmax: 3
# geographic CRS: WGS 84
# id             geometry
# 1  1          POINT (1 2)
# 2  1 LINESTRING Z (1 2 3)

[edzer]

Thanks, this should do it.