Unable to execute algorithm run_qgis

[raff-k]

Hey,

when I execute run_qgis() with your examples like
alg <- "grass7:r.slope.aspect"
data(dem, package = "RQGIS")
slope <- run_qgis(alg, elevation = dem, slope = "slope.asc", load_output = TRUE)
or
ger <- getData(name = "GADM", country = "DEU", level = 1)
params$INPUT_LAYER <- ger
params$OUTPUT_LAYER <- file.path(tempdir(), "ger_coords.shp")
out <- run_qgis(alg = "qgis:polygoncentroids", params = params, load_output = TRUE)

I get the following error:

Error in run_qgis(alg, elevation = dem, slope = "slope.asc", load_output = TRUE) :
Unable to execute algorithm
AVer\AppData\Local\Temp\Rtmps3f6yS/elevation.tif
or
Error in run_qgis(alg = "qgis:polygoncentroids", params = params, load_output = TRUE) :
Unable to execute algorithm
AVer\AppData\Local\Temp\Rtmps3f6yS/INPUT_LAYER.shp
, respectively.

It seems to me that the temp-path is not valid, because my computer name is rAVer and not AVer. After execution I also checked the temp directory, and the files exist!
When I use hard-coded paths instead (I saved the dem on a specific location, and give the exact location also of the "slope.asc"), it works.

I use Windows 10 Pro 64-bit, R version 3.4.0 (2017-04-21), RQGIS_1.0.0 and QGIS 2.18.7.

[jannes-m]

What is the output of tempdir() on your computer? Is it really AVer\AppData\Local\Temp\Rtmps3f6yS, not C:\\Users\\rAVer\\AppData\\Local\\Temp\\Rtmps3f6yS

[raff-k]

The tempdir() output is

"C:\Users\rAVer\AppData\Local\Temp\Rtmps3f6yS"

and there are also the files inside (i.e. elevation.tif), but it seems that they cannot be loaded

[jannes-m]

so what happens if you type:
library("raster")
raster(filepath(tempdir(), "elevation.tif"))
This is basically what RQGIS is trying to do.

[vaibhavgupta13]

Sorry to be posting in this thread. I thought it would be better to raise it here rather than as a separate question as I am facing the exact same issue. The tempdir() out is:
"C:\Users\vgupta\AppData\Local\Temp\RtmpimnLLI"

But in the error it says the following:
Wrong parameter value: C:\Users�gupta\AppData\Local\Temp\RtmpimnLLI/INPUT.shp

I'm not sure what's happening between Users and gupta. I seems to eat away V!
Any thoughts?

[tim-salabim]

I second this one, we have had the same error message (wrong parameter value D:/AT/tmp/slivers.shp in our case) over the last couple of days give us headaches too. We reverted back to 0.2.0 to have it running smoothly. We traced the issue back to this function in reticulate.

[vaibhavgupta13]

@tim-salabim - What do you mean when you say you reverted to 0.2.0? RQGIS version? If yes, how can I download the RQGIS library with that particular version which works okay in R? Many thanks.

[tim-salabim]

Download the old version tar.gz from here and install locally

install.packages(path_to_RQGIS_0.2.0.tar.gz, repos = NULL, type="source")

or use the Install button in RStudio's Packages tab and set to 'Install from: Package Archive File (tar.gz)'

[vaibhavgupta13]

@tim-salabim - Thanks a lot for your input. I managed to download it in the interim but now I am being presented with another problem. The package is looking for OSGeo4W on my C drive when it doesn't exist. I have already specified the environment for QGIS using set_env("C:/Program Files/QGIS 2.14/"). Does this mean, I need to install OSGeo4W and not QGIS?

[tim-salabim]

Did you set_env("C:/Program Files/QGIS 2.14/") in run_qgis?

[vaibhavgupta13]

No, just at the beginning of my code as mentioned in my earlier comment. Does it need to be explicitly mentioned in run_qgis too?

[jannes-m]

set_env only looks for OSgeo4W if one of the following folders exist on your disk: C:/OSGEO4~1 or C:/OSGEO4~2. Maybe they are empty? Well, maybe we should check for this in set_env() as well...

Regarding you other problem, I can't really say how to help. On my machine everything works as expected. It seems to be an encoding problem. Obviously, \v cannot be interpreted, and therefore � appears (https://en.wikipedia.org/wiki/Specials_(Unicode_block)).

[jannes-m]

If you use RQGIS 0.2.0, you need to specify qgis_env in run_qgis.

qgis_env <- set_env("C:/Program Files/QGIS 2.14/")
run_qgis(alg, params, qgis_env = qgis_env)

We only started to cache the output of set_env since RQGIS 1.0.0

[jannes-m]

@raff-k When will you be the next time in Jena, so that we can have a look at this nasty problem together?

[jannes-m]

@tim-salabim Tim, I am not sure if you are really facing the same problem. Can you please provide a reproducible example?

[jannes-m]

@tim-salabim Does this code work for you (using RQGIS 1.0.1)?

library("RQGIS")
library("sf")
# create two polygons for a toy example
coords_1 <-  
  matrix(data = c(0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
         ncol = 2, byrow = TRUE)
coords_2 <-
  matrix(data = c(-0.5, -0.5, 0.5, -0.5, 0.5, 0.5, 
                  -0.5, 0.5, -0.5, -0.5),
         ncol = 2, byrow = TRUE)

# create the first polygon
poly_1 <- st_polygon(list((coords_1))) 
# convert it into a simple feature collection 
poly_1 <- st_sfc(poly_1)
poly_1 <- st_sf(geometry = poly_1)
# create a second polygon
poly_2 <- st_polygon(list((coords_2))) %>%
   st_sfc %>%
   st_sf(geometry = .)
poly_3 <- st_union(poly_1, poly_2)
poly_3$id = 1
alg <- "qgis:eliminatesliverpolygons"
tmp <- run_qgis(alg, INPUT = poly_3, MODE = 2, ATTRIBUTE = "id",
                OUTPUT = "clean.shp", load_output = TRUE)

[tim-salabim]

@jannes-m yes, this works, but when I pass a .shp file path instead of poly_3 it fails. I have tried to mimic the function calling structure from our workflow, hence the function hierarchy. Sorry my bad, when I supply a suitable path it does seem to work.

# library("RQGIS")
library("sf")
# create two polygons for a toy example
 
  coords_1 <-  
    matrix(data = c(0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
           ncol = 2, byrow = TRUE)
  coords_2 <-
    matrix(data = c(-0.5, -0.5, 0.5, -0.5, 0.5, 0.5, 
                    -0.5, 0.5, -0.5, -0.5),
           ncol = 2, byrow = TRUE)
  
  # create the first polygon
  poly_1 <- st_polygon(list((coords_1))) 
  # convert it into a simple feature collection 
  poly_1 <- st_sfc(poly_1)
  poly_1 <- st_sf(geometry = poly_1)
  # create a second polygon
  poly_2 <- st_polygon(list((coords_2))) %>%
    st_sfc %>%
    st_sf(geometry = .)
  poly_3 <- st_union(poly_1, poly_2)
  poly_3$id = 1
  

elim_sliv = function(alg, inp, mode, attr, outp, lout = FALSE, qgis_env) {
  RQGIS::run_qgis(alg = alg,
                  INPUT = inp,
                  MODE = mode,
                  ATTRIBUTE = attr,
                  OUTPUT = outp,
                  load_output = lout,
                  qgis_env = RQGIS::set_env(qgis_env))
}

run_elim_sliv = function(alg, inp, mode, attr, out_dir, lout = FALSE, qgis_env) {
 
  inpt = st_write(inp, paste0(out_dir, "slivers.shp"), delete_dsn = TRUE)
  out = paste0(out_dir, "clean.shp")
  elim_sliv(alg = alg,
            inp = paste0(out_dir, "slivers.shp"),
            mode = mode,
            attr = attr,
            outp = out,
            lout = FALSE,
            qgis_env = qgis_env)
}
            
alg <- "qgis:eliminatesliverpolygons"
inp = poly_3
mode = 2
attr = "id"
out_dir = "D:/test/"
lout = FALSE
qgis_env = "C:/Program Files/QGIS 2.18/"

run_elim_sliv(alg = alg,
              inp = inp,
              mode = mode,
              attr = attr,
              out_dir = out_dir,
              lout = lout,
              qgis_env = qgis_env)

[jannes-m]

st_write outputs NULL instead of a path, consequently you specify inpt as NULL. Maybe we should implement that INPUT arguments could not take NULL as value to prevent this. Anyways, put this line after your st_write-call, and everything should work as expected:

inpt <- paste0(out_dir, "slivers.shp")

[tim-salabim]

@jannes-m yeah sorry I realised that just now. I will report back with a suitable example in case I find one

[jannes-m]

@tim-salabim I hope you don't find one :-)

[jannes-m]

@vaibhavgupta13 What exactly does the error message tell you?
If I set a wrong path, then set_env tells me that the QGIS installation cannot be found there:

set_env("D:/Osgeo")
Error in FUN(X[[i]], ...) : 
  Folder D:/Osgeo/apps could not be found under NA Please install it.

Please follow the installation guide how to install QGIS here.

[jannes-m]

Finally, I had a machine at my disposal with which I could debug this nasty issue. Commit 484e1d3 should fix it.

[vaibhavgupta13]

@jannes-m - Thanks a lot for your response and being patient with my query. I have managed to finally get RQGIS to run on my computer. I'm very happy but there is something else that has come up. Every time I try to run find_algorithms(search_term = "([Un]ion)") in returns the file path to the temporary directory. What's happening? Thank you once again.

[jannes-m]

I think you are still using RQGIS 0.2.0. Please use our latest developer version:

devtools::install_github("jannes-m/RQGIS")

With RQGIS 0.2.0, the tempdir() was automatically returned after each function call. We fixed this later on because this behavior was confusing and also unnecessary. If find_algorithms just returns the tempdir(), then it just means that it can't find any algorithm matching your search criteria.

[vaibhavgupta13]

I just did as you suggested @jannes-m and now I'm being presented with this error

Error in initialize_python(required_module) : Installation of Python not found, Python bindings not loaded.

What else can I do from here on?

[jannes-m]

Well, this tells you that something's wrong with your QGIS Python installation. Which QGIS version are you using? And can you actually start the GUI? And do you have the Python console there on your disposal.

Please install the latest QGIS version (2.14.17 or 2.18.11). For this, please use our installation guide (http://jannes-m.github.io/RQGIS/articles/install_guide.html).

[vaibhavgupta13]

@jannes-m - Thank you once again for your response. I can confirm that I have the latest QGIS version (2.18.11). It is very much installed and I can do the analysis in QGIS with no issues. In fact, I tried doing this after I received the error message and I was able to work okay in QGIS. Having said that, I could still execute the Union function without actually using find_algorithms(search_term = "([Un]ion)") because I Had the script saved up from last week. However, I am unable to find other algorithms.

[vaibhavgupta13]

Hi @jannes-m, sorry I'm back with another issue. This time it is a rather annoying one. I got RQGIS to work the last time around. Now, lately, my old RQGIS script refuses to work. Here's the error I'm presented with:
Error in run_qgis(alg = "qgis:union", INPUT = parcel.polys, INPUT2 = council.prop, :
Unable to execute algorithm
Wrong parameter value: C:/Users/vgupta/AppData/Local/Temp/RtmpkDIhwo/INPUT.shp

Any idea what's going on here?
Thanks

[jannes-m]

Could you pls provide a reproducible example? But this works for me:

library("sf")
library("RQGIS")
coords_1 <-  
    matrix(data = c(0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
           ncol = 2, byrow = TRUE)
coords_2 <-
    matrix(data = c(-0.5, -0.5, 0.5, -0.5, 0.5, 0.5, 
                    -0.5, 0.5, -0.5, -0.5),
           ncol = 2, byrow = TRUE)
# create the first polygon
poly_1 <- st_polygon(list((coords_1))) 
# convert it into a simple feature collection 
poly_1 <- st_sfc(poly_1)
poly_1 <- st_sf(geometry = poly_1)
# create a second polygon
poly_2 <- st_polygon(list((coords_2))) %>%
  st_sfc %>%
  st_sf(geometry = .)
get_usage("qgis:union")
tmp = run_qgis("qgis:union", 
               INPUT = poly_1, INPUT2 = poly_2,
               OUTPUT = file.path(tempdir(), "union.shp"),
               load_output = TRUE)

[vaibhavgupta13]

Thanks @jannes-m. Your code seems to work okay. The code that I was using and was working until December last year is shared below:
union$INPUT <- parcel.polys
union$INPUT2 <- council.prop
union$OUTPUT <- file.path(tempdir(),"parcel_council_union.shp")
parcel.council.union <- run_qgis(alg = "qgis:union",
params = union,
load_output = TRUE)

Thank you once again :).

[jannes-m]

params = get_args_man("qgis:union")
params$INPUT = poly_1
params$INPUT2 = poly_2
params$OUTPUT = file.path(tempdir(), "union.shp")
tmp = run_qgis("qgis:union", params = params, load_output = TRUE)

also works.

[vaibhavgupta13]

I'm back once again @jannes-m . You will probably be really annoyed to hear back from me but there are some serious problems I'm facing and I'm hoping you can once again help me. Earlier, the union was working flawlessly. Today, I have executed the same code as you have provided above and it is removing data from so many of the columns used in the union! Any idea what is going on? Unfortunately, the data it is removing is quite critical to the whole process.

Again, I would like to reiterate that all this was working perfectly fine earlier!

Any help would be greatly appreciated.
Thanks.

[jannes-m]

pls provide a reproducible example

[tim-salabim]

As a guess, could it be that your column names a longer than 10 characters? I've had instances where GDAL drops the values of those columns via st_write. I think it has been addressed on the sf (R) side, but may still be problematic when using GDAL via QGIS algos.

[jannes-m]

Check out also GRASS:

get_args_man("grass7:v.overlay")
get_options('grass7:v.overlay')
out  = run_qgis("grass7:v.overlay", ainput = poly_1, binput = poly_2, 
                 operator = 1,
                 output = tempfile(fileext = ".shp"),
                 load_output = TRUE)

Or depending on the size of your polygons, give sf also a try:

sf::st_union(poly_1, poly_2)

[vaibhavgupta13]

Hi @jannes-m - It still doesn't work. As I mentioned earlier, this was working perfectly okay until a few months back. Has anything changed in the backend? sf is not suitable for my purpose because I am dealing with massive shapefiles. Any help would be appreciated.
Thanks a lot.

[pat-s]

@tim-salabim might have some data.table approaches for sf that might solve your problem.

However, as mentioned already, we need a reprex from you to be able to help.

_{Sent with GitHawk}

[vaibhavgupta13]

Here's the code I'm using:

params <- get_args_man(alg = "qgis:union")

params$INPUT <- parcel.polys
params$INPUT2 <- council.prop
params$OUTPUT <- file.path(tempdir(),"parcel_council_union.shp")
parcel.council.union <- run_qgis(alg = "qgis:union",
                                 params = params,
                                 load_output = TRUE)

I'm also attaching 2 screen dumps showing the difference in the data that is being lost.

As you can see, PLAN_NUMBE column information has fallen off! Do you recommend renaming the columns and then proceeding with the union?
Please let me know if any more information from my side might help in getting a better understanding of the issue. Thanks.

[pat-s]

@vaibhavgupta13 Please see https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example.

[raff-k]

@vaibhavgupta13, have you tried to perform the union inside a QGIS session? If the same issue appears inside QGIS, then the "bug" is related to QGIS, not RQGIS.

However, in QGIS you can investigate the log-file of the process. I.e. you can set the Keep dialog open after running an algorithm and Show extra info in Log panel parameters in Processing > Options ... > [+] General (expand). Then check the log window of the union process. Maybe you can post the log-file text?

[lbusett]

It's a rather wild guess, but you may be seeing a "variation" of this: r-spatial/sf#538

Column "PLAN_NUMBER" is longer than 10 characters, so it is automatically abbreviated on saving to shp, It becomes P_NUMBER (which you also have in the data), and data loss occurs ?

I'd try renaming (a.k.a. shortening) the columns beforehand and see what happens.

[vaibhavgupta13]

@raff-k Thanks for your comment. I can confirm it is working as desired within QGIS.
@lbusett - Thanks a lot. May be that's where the issue lies. But again the very strange thing to which I haven't received any comments on is why has it suddenly stopped working when it was working okay until at least December last year.

[vaibhavgupta13]

@lbusett - I have edited the column name to be less than 10 characters and it does seem to resolve the issue! Thanks a lot for all your help everyone.

[lbusett]

You're welcome. Concerning "what went wrong" between december and now, it could be any number of things. "Stronger suspects" IMO could be an "inadvertent" regression in the function used to write shapefiles (writeOGR ?) , or changes in GDAL ESRI driver.

[kbannarm]

I know this issue is closed, however I cannot execute algorithms.
I've tested this a few ways, most recently with code you provide above:

library("sf")
library("RQGIS")
coords_1 <-  
  matrix(data = c(0, 0, 1, 0, 1, 1, 0, 1, 0, 0),
         ncol = 2, byrow = TRUE)
coords_2 <-
  matrix(data = c(-0.5, -0.5, 0.5, -0.5, 0.5, 0.5, 
                  -0.5, 0.5, -0.5, -0.5),
         ncol = 2, byrow = TRUE)
# create the first polygon
poly_1 <- st_polygon(list((coords_1))) 
# convert it into a simple feature collection 
poly_1 <- st_sfc(poly_1)
poly_1 <- st_sf(geometry = poly_1)
# create a second polygon
poly_2 <- st_polygon(list((coords_2))) %>%
  st_sfc %>%
  st_sf(geometry = .)
get_usage("qgis:union")
tmp = run_qgis("qgis:union", 
               INPUT = poly_1, INPUT2 = poly_2,
               OUTPUT = file.path(tempdir(), "union.shp"),
               load_output = TRUE)

And I get the following error:

Error in run_qgis("qgis:union", INPUT = poly_1, INPUT2 = poly_2, OUTPUT = file.path(tempdir(), :
Unable to execute algorithm Wrong parameter value: C:/Users/bannarmk/AppData/Local/Temp/1/RtmpSIxmWj/INPUT.shp

My tempdir() is:

[1] "C:\Users\bannarmk\AppData\Local\Temp\1\RtmpSIxmWj"
and the INPUT.shp file is in it.

Any ideas?
I'm using RQGIS version 1.0.3 and QGIS version 2.18.19.

Thanks,
Katherine

[jannes-m]

Can you pls use the developer version (RQGIS 1.03.9000; hopefully this solves the issue):

install_github("jannes-m/RQGIS")

I am also using QGIS 2.18.20.

[kbannarm]

Any idea why I'm getting this error when installing the dev version?

Warning in CPL_gdal_init() : GDAL Error 1: Can't load requested DLL: C:/Program Files/QGIS 2.18/bin/gdalplugins\gdal_ECW_JP2ECW.dll 193: %1 is not a valid Win32 application.

Consequently it's defaulting to the install.packages("RQGIS") version.

[kbannarm]

scrap that - removed all dependencies previously installed and re-installed everything. Now all working :)

[jannes-m]

Excellent! Actually, this would have been the advice I would have given.