z error when reading geojson data #1592

yeedle · 2021-02-03T02:27:36Z

I'm trying to read the nytimes' elections dataset by using st_read. After downloading and extracting the file, I get the following error (sf version 0.9-7):

> data <- read_sf("~Downloads/precincts-with-results.geojson")
Error in CPL_get_z_range(obj, 3) : z error - expecting three columns;

System details

platform x86_64-pc-linux-gnu

arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 6.3
year 2020
month 02
day 29
svn rev 77875
language R
version.string R version 3.6.3 (2020-02-29)
nickname Holding the Windsock

sessionInfo()

R version 3.6.3 (2020-02-29) Platform: x86_64-pc-linux-gnu (64-bit) Running under: elementary OS 0.4.1 Loki

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
[1] LC_CTYPE=en_US.UTF-8
[2] LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8
[4] LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8
[6] LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8
[8] LC_NAME=C
[9] LC_ADDRESS=C
[10] LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8
[12] LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets
[6] methods base

other attached packages:
[1] forcats_0.5.0 stringr_1.4.0 dplyr_1.0.2
[4] purrr_0.3.4 readr_1.4.0 tidyr_1.1.2
[7] tibble_3.0.4 ggplot2_3.3.2 tidyverse_1.3.0
[10] sf_0.9-7

loaded via a namespace (and not attached):
[1] Rcpp_1.0.5 cellranger_1.1.0
[3] pillar_1.4.7 compiler_3.6.3
[5] dbplyr_1.4.4 class_7.3-17
[7] tools_3.6.3 lubridate_1.7.9
[9] jsonlite_1.7.2 lifecycle_0.2.0
[11] gtable_0.3.0 pkgconfig_2.0.3
[13] rlang_0.4.9 reprex_0.3.0
[15] DBI_1.1.1 cli_2.2.0
[17] rstudioapi_0.13 haven_2.3.1
[19] e1071_1.7-4 withr_2.3.0
[21] xml2_1.3.2 httr_1.4.2
[23] fs_1.5.0 hms_0.5.3
[25] generics_0.0.2 vctrs_0.3.6
[27] classInt_0.4-3 grid_3.6.3
[29] tidyselect_1.1.0 glue_1.4.2
[31] R6_2.5.0 fansi_0.4.1
[33] readxl_1.3.1 modelr_0.1.8
[35] blob_1.2.1 magrittr_2.0.1
[37] backports_1.2.0 scales_1.1.1
[39] ellipsis_0.3.1 units_0.6-7
[41] rvest_0.3.6 assertthat_0.2.1
[43] colorspace_2.0-0 KernSmooth_2.23-17
[45] stringi_1.5.3 munsell_0.5.0
[47] broom_0.7.1 crayon_1.3.4

The text was updated successfully, but these errors were encountered:

edzer · 2021-02-03T07:16:48Z

@dcooley ?

dcooley · 2021-02-03T09:55:00Z

yep, I get this error too. I'll take a look in the morning (Australian time).

In the meantime, this works

data <- geojsonsf::geojson_sf("~/Downloads/precincts-with-results.geojson")

dcooley · 2021-02-03T09:58:58Z

Actually, I've had a quick look, and there are some geometries with a z property

df <- sfheaders::sf_to_df(
  sf = sf
)

library(data.table)
setDT( df )

df[ !is.na( z ), ]
        sfg_id multipolygon_id polygon_id linestring_id         x        y z
     1:  33209              NA      33209             1 -95.42749 37.96577 0
     2:  33209              NA      33209             1 -95.42737 37.96577 0
     3:  33209              NA      33209             1 -95.42738 37.96657 0
     4:  33209              NA      33209             1 -95.42740 37.96932 0
     5:  33209              NA      33209             1 -95.42740 37.97120 0
    ---                                                                     
933443:  37279              NA      37279             1 -97.37149 37.56692 0
933444:  37279              NA      37279             1 -97.37200 37.56692 0
933445:  37279              NA      37279             1 -97.37199 37.56791 0
933446:  37279              NA      37279             1 -97.37149 37.56791 0
933447:  37279              NA      37279             1 -97.37149 37.56750 0

But they're all 0

> df[ !is.na( z ) & z > 0 ]
Empty data.table (0 rows and 7 cols): sfg_id,multipolygon_id,polygon_id,linestring_id,x,y...

And here is a minimally reproducible example

sf::st_sfc(
  list(
    sf::st_point(c(0,0))
    , sf::st_point(c(1,1,1))
  )
)

Error in CPL_get_z_range(unclass(obj)[sel], 0) : 
  z error - expecting three coordinates

dcooley · 2021-02-03T22:11:09Z

@edzer

With this example

sf::st_sfc(
  list(
    sf::st_point(c(0,0))
    , sf::st_point(c(1,1,1))
  )
)

If I let it pass through the CPL_get_z_range() C++ functions, it ends up hitting this stop point

# set n_empty, check XY* is uniform:
if (is.null(attr(lst, "n_empty")) || any(is_null)) { # n_empty is set by CPL_read_wkb:
	attr(lst, "n_empty") = sum(vapply(lst, sfg_is_empty, TRUE))
	if (length(u <- unique(sfg_classes[1L,])) > 1)
		stop(paste("found multiple dimensions:", paste(u, collapse = " ")))
}

For examle

sf::st_sfc(
	list(
		sf::st_point(c(0,0))
		, sf::st_point(c(1,1,1))
	)
 )
Error in sf::st_sfc(list(sf::st_point(c(0, 0)), sf::st_point(c(1, 1, 1)))) : 
  found multiple dimensions: XY XYZ

But the read_sf() example will work

data <- read_sf("~/Downloads/precincts-with-results.geojson")
nrow(data)
[1] 109363

Is it perhaps best to allow multiple dimensions, given the various st_ functions can handle them?

For example

geo <- '
{
	"type":"FeatureCollection",
	"features": [
	{
		"type":"Feature",
		"geometry": {
			"type": "LineString",
			"coordinates": [[0,0],[1,1]]
		}
	},
	{
		"type":"Feature",
			"geometry": {
				"type": "LineString",
				"coordinates": [[0,0,0],[1,1,0]]
			}
	}
	]
}'


sf <- st_read(geo)
sf::st_length( sf )

Units: [m]
[1] 156899.6 110574.4

(The next thing for me to understand is why adding in the Z coordinate here creates a differnet 'length').

edzer · 2021-02-03T22:25:27Z

Thanks for looking into this! I'd like to see st_read succeed in such cases, best would be with a warning and a suggestion how to clean up the mess.

dcooley · 2021-02-03T22:40:54Z

ok, I'll pull together a PR for review.

yeedle · 2021-02-03T23:27:44Z

I remember running into a similar issue in the past and I solved it with st_zm (which is obviously not possible with st_read.)

And thanks for the amazing work you guys are doing!

edzer · 2021-02-16T09:10:18Z

@dcooley any news on this one?

dcooley · 2021-02-18T08:20:56Z

Hi, yeah, sorry - got sidetracked with something I'm working on.

I think a limiting factor is the current requirement to have the dimension uniform for all geometries.

How do you think this should be handled?

edzer · 2021-02-27T14:00:55Z

The purist in me would say: leave this an error, let the user solve the problem, and let the user try again after solving it. But then: how should she solve it? Before we computed z & m ranges (and generated errors), an object was returned with mixed dimensions, raised a warning, and pointed to st_zm() to correct this (remove all z, m, or zm geometries). Would it be doable to go back to that state?

edzer · 2021-03-03T10:00:17Z

@dcooley the requirement you refer to is skipped when we come back from CPL_read_wkb, i.e. when we read data from file.

dcooley · 2021-03-03T20:36:21Z

Yes, for example this works

geo <- '{"type":"FeatureCollection","features":[{"type":"Feature","properties":{"id":1.0},"geometry":{"type":"Point","coordinates":[1.0,2.0]}},{"type":"Feature","properties":{"id":1.0},"geometry":{"type":"LineString","coordinates":[[1.0,2.0,3.0,4.0]]}}]}'

sf <- sf::st_read( geo )

sf
# Simple feature collection with 2 features and 1 field
# geometry type:  GEOMETRY
# dimension:      XY
# bbox:           xmin: 1 ymin: 2 xmax: 1 ymax: 2
# z_range:        zmin: 3 zmax: 3
# geographic CRS: WGS 84
# id             geometry
# 1  1          POINT (1 2)
# 2  1 LINESTRING Z (1 2 3)

However, the dimension attributed is printed as XY, even though there are mixed dimensions.

Is it worth considering removing this uniform check, and perhaps printing the different dimensions that exist, something like

sf
# Simple feature collection with 2 features and 1 field
# geometry type:  GEOMETRY
# dimension:      XY, XYZ
# bbox:           xmin: 1 ymin: 2 xmax: 1 ymax: 2
# z_range:        zmin: 3 zmax: 3
# geographic CRS: WGS 84
# id             geometry
# 1  1          POINT (1 2)
# 2  1 LINESTRING Z (1 2 3)

edzer · 2021-03-05T11:27:23Z

Thanks, this should do it.

dcooley · 2021-03-07T02:49:51Z

I think we also need to let it pass the CPL_get_z_range() function to avoid the original error

Error in CPL_get_z_range(obj, 3) : z error - expecting three columns;

edzer · 2021-03-07T09:32:24Z

Yes. Return NA?

…

On March 7, 2021 3:50:02 AM GMT+01:00, Dave ***@***.***> wrote: I think we also need to let it pass the `CPL_get_z_range()` function to avoid the original error ``` Error in CPL_get_z_range(obj, 3) : z error - expecting three columns; ``` -- You are receiving this because you modified the open/close state. Reply to this email directly or view it on GitHub: #1592 (comment)

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

dcooley · 2021-03-08T01:27:07Z

yes, I think so. I had a very quick go at it with this implementation a few days ago.

edzer · 2021-03-08T09:08:55Z

LGTM!

dcooley · 2021-06-09T22:37:40Z

Saw your comment .

I'm actually struggling to compile sf on my M1 Mac. Are there any open issues I can read through to get it working?

IvanPalm · 2022-07-22T11:50:17Z

FYI: this error occurs in sf v1.0-7.

The way-around suggested by @dcooley (i.e. using geojsonsf::geojson_sf) worked fine. Thanks for the hint and for the great work that you have been doing!

barryrowlingson · 2022-09-05T08:35:04Z

I'm seeing this error from the JSON file downloaded from: https://149613070.v2.pressablecdn.com/coordinates/indigenousTerritories.json (4Mb) using 1.0-8:

> library(sf)
Linking to GEOS 3.8.0, GDAL 3.0.4, PROJ 6.3.1; sf_use_s2() is TRUE
> terr = st_read("./indigenousTerritories.json")
Reading layer `indigenousTerritories' from data source 
  `/nobackup/rowlings/Downloads/SO/indigenousTerritories.json' 
  using driver `GeoJSON'
Error in CPL_get_z_range(obj, 3) : z error - expecting three columns;
> packageVersion("sf")
[1] ‘1.0.8’

Current fix is to use ogr2ogr to flatten it all to XY using the -dim option, but I'm also wondering why passing the right type= code to st_read has no effect.

gklarenberg · 2022-12-13T16:39:53Z

Is this issue still open? One of my students is running into the same issue loading a regular (ESRI) shapefile (latest R version 4.2.2, sf version 1.0.9 and rgdal version 1.6.2). The code worked last year (probably an older sf version). We used this (#2046 (comment)) solution to enable her to read in her shapefile again and re-run her analysis and map. But I would prefer to have an actual fix...

amb26 · 2022-12-23T12:31:46Z

Eager for a fix for this. This backwards incompatible change in what should be a patch release of 1.0.9 has made life difficult for us, and we have having to downgrade manually to st 1.0.8 - this kind of change should have been introduced in a 2.0 release. Would also be happy with a workaround that didn't involve a deprecated library such as rgdal.

amb26 · 2022-12-23T12:43:02Z

When using rgdal::readOGR I do indeed receive the message "Z-dimension discarded" - the shapefile in question can be found at https://github.com/IMERSS/maxwell/tree/main/spatial_data/vectors/Shp_files/Watershed_CRD and was exported from QGIS.

closes #2046 #1592

abdulrr · 2022-12-28T11:59:48Z

I still get the same error on sf_1.0-9, rgdal_1.6-3
Interestingly it works with a polygon layer with a z range, but not with a multi-polygon feature layer.
any suggestions?

kadyb · 2022-12-28T12:47:30Z

@abdulrr, make sure you updated {sf} to the devel version (1.0.10).

install.packages("sf", repos = "https://r-spatial.r-universe.dev")

or

remotes::install_github("r-spatial/sf")

abdulrr · 2022-12-28T13:00:05Z

@kadyb works now! thanks

itsmevictor · 2023-01-11T19:15:27Z

Hi, I still have this problem using 1.0-9 and cannot manage to install 1.10 using either of the two commands mentioned above. Would anyone have a solution?

itsmevictor · 2023-01-11T19:15:28Z

Hi, I still have this problem using 1.0-9 and cannot manage to install 1.10 using either of the two commands mentioned above. Would anyone have a solution?

kadyb · 2023-01-11T20:22:15Z

@itsmevictor, I think it will be better if you open a new issue and post the installation log with error and session information.

Robinlovelace · 2023-02-27T21:04:39Z

Confirmed: I just hit this issue and tried the work-around.

geojsonsf::geojson_sf("file.geojson)

worked for me

Robinlovelace · 2023-02-27T21:05:17Z

Happy to share minimal input files that caused it, would need to strip out some non public stuff for a reprex though.

kadyb · 2023-02-27T21:08:44Z

@Robinlovelace, this should be fixed in 1.0.10, isn't it?

Robinlovelace · 2023-02-27T22:41:07Z

@kadyb just checked and I tested it on 1.0.9. Will check on the latest version now...

Robinlovelace · 2023-02-27T23:03:29Z

Update: it works. Apologies for the false alarm and thanks for spotting this on, need to keep up with the times!

Output for the record:

Geometry set for 5 features 
Geometry type: GEOMETRY
Dimension:     XY, XYZ
Bounding box:  xmin: -1.261924 ymin: 51.71217 xmax: -1.175928 ymax: 51.76901
z_range:       zmin: 0 zmax: 0
Geodetic CRS:  WGS 84
POLYGON Z ((-1.194047 51.73664 0, -1.19403 51.7...
LINESTRING (-1.212149 51.73429, -1.212271 51.73...
LINESTRING (-1.254473 51.75494, -1.254572 51.75...
LINESTRING (-1.259964 51.75829, -1.259973 51.75...
POLYGON ((-1.260947 51.75889, -1.260262 51.7577...

edonnachie · 2023-03-16T10:19:07Z

For what it's worth, I had the same problem reading in an ESRI shape file using sf v.1.0.9. Updating to 1.0.11 has fixed the problem.

dcooley added a commit to dcooley/sf that referenced this issue Feb 21, 2021

allowing multi dimensions through cpp r-spatial#1592

3ab97f6

edzer closed this as completed in 1814b02 Mar 5, 2021

edzer mentioned this issue Jun 1, 2021

st_read() fails to read mixture of XY & XYZ points generated by GPS #1683

Closed

edzer reopened this Jun 1, 2021

edzer mentioned this issue May 9, 2022

EMPTYs with mixed dimension cause error #1941

Closed

edzer mentioned this issue Nov 24, 2022

Error Reading Shapefile with st_read since version 1.09... #2046

Closed

edzer added a commit that referenced this issue Dec 23, 2022

don't stop() on mixes of XY and XYZ, but return NA ranges

f9104f6

closes #2046 #1592

edzer closed this as completed Dec 23, 2022

ccmothes mentioned this issue Jan 7, 2023

Strange bugs that need fixin' rossyndicate/nps_water_vulnerability#38

Merged

itsmevictor mentioned this issue Jan 11, 2023

z error (3 columns) with shapefile & 1.10 installation error #2081

Closed

walkerke mentioned this issue Feb 6, 2023

2014 ACS 5-year data breaking in get_acs call walkerke/tidycensus#501

Closed

edzer mentioned this issue Feb 8, 2023

Error in CPL_get_z_range(obj, 1) : z error - expecting three columns; #2095

Closed

achubaty mentioned this issue Feb 23, 2023

Error: Error: z error - expecting three columns; PredictiveEcology/LandR#91

Closed

z error when reading geojson data #1592

z error when reading geojson data #1592

Comments

yeedle commented Feb 3, 2021

edzer commented Feb 3, 2021

Uh oh!

dcooley commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcooley commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dcooley commented Feb 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

edzer commented Feb 3, 2021

Uh oh!

dcooley commented Feb 3, 2021

Uh oh!

yeedle commented Feb 3, 2021

Uh oh!

edzer commented Feb 16, 2021

Uh oh!

dcooley commented Feb 18, 2021

Uh oh!

edzer commented Feb 27, 2021

Uh oh!

edzer commented Mar 3, 2021

Uh oh!

dcooley commented Mar 3, 2021

Uh oh!

edzer commented Mar 5, 2021

Uh oh!

dcooley commented Mar 7, 2021

Uh oh!

edzer commented Mar 7, 2021 via email

Uh oh!

dcooley commented Mar 8, 2021

Uh oh!

edzer commented Mar 8, 2021

Uh oh!

dcooley commented Jun 9, 2021

Uh oh!

IvanPalm commented Jul 22, 2022

Uh oh!

barryrowlingson commented Sep 5, 2022

Uh oh!

gklarenberg commented Dec 13, 2022

Uh oh!

amb26 commented Dec 23, 2022

Uh oh!

amb26 commented Dec 23, 2022

Uh oh!

abdulrr commented Dec 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kadyb commented Dec 28, 2022

Uh oh!

abdulrr commented Dec 28, 2022

Uh oh!

itsmevictor commented Jan 11, 2023

Uh oh!

itsmevictor commented Jan 11, 2023

Uh oh!

kadyb commented Jan 11, 2023

Uh oh!

Robinlovelace commented Feb 27, 2023

Uh oh!

Robinlovelace commented Feb 27, 2023

Uh oh!

kadyb commented Feb 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Robinlovelace commented Feb 27, 2023

Uh oh!

Robinlovelace commented Feb 27, 2023

Uh oh!

edonnachie commented Mar 16, 2023

Uh oh!

dcooley commented Feb 3, 2021 •

edited

Loading

dcooley commented Feb 3, 2021 •

edited

Loading

dcooley commented Feb 3, 2021 •

edited

Loading

abdulrr commented Dec 28, 2022 •

edited

Loading

kadyb commented Feb 27, 2023 •

edited

Loading