Error in cli_tools() while trying do_provision()

[DataStrategist]

I originally opened this ticket rstudio/plumber#239 , who refered me to you

So from Rstudio, I'm trying to set up plumber in a DO droplet. I have successfully authenticated my account to the R-session, but immediately after that I get the following error:

> do_provision(droplet = "boom")
Error in cli_tools() : 
ssh, scp not found on your computer
Install the missing tool(s) and try again

I have putty installed, routinely ssh from the rstudio terminal, but also the git bash on this pc has ssh ability. I also downloaded cgywin, to no avail.

How can I convince the app that I do have ssh? Perhaps it's the scp I don't have?

Thanks! SessionInfo follows:

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] plumbr_0.6.9  plumber_0.4.4

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.15          digest_0.6.15         crayon_1.3.4          aws.signature_0.3.5   R6_2.2.2             
 [6] jsonlite_1.5          magrittr_1.5          objectSignals_0.10.2  httr_1.3.1            stringi_1.1.6        
[11] curl_3.0              rstudioapi_0.7.0-9000 xml2_1.2.0            tools_3.4.2           analogsea_0.6.0      
[16] aws.s3_0.3.3          httpuv_1.3.5          yaml_2.1.14           compiler_3.4.2        base64enc_0.1-3      
[21] openssl_0.9.7   

[sckott]

thanks for opening the issue @DataStrategist

I'm not sure what exactly the issue is, it's possible it's the scp thing. I could fiddle with that internal fxn cli_tools that checks for ssh and scp, but we're working on re-making the ssh interface with the new package ssh on the ssh branch - can you install it remotes::install_github("sckott/analogsea@ssh") and try again after installing and restarting the session?

[DataStrategist]

Yup, that seemed to do something! I'm assuming you meant devtools instead of remotes on the install_github command, right?

Anyway, so that did try to do something, but now I ran into this:

I'm in London, does it perhaps only support US? I have also confirmed w/ my account, nothing has been created. But at least it does see that I have ssh! So progress!

[sckott]

https://github.com/r-lib/remotes is replacing devtools::install stuff AFAIK, but both work

You can pass on options via ... in do_provision to analogsea::droplet_create, one of which is ssh_keys which defeaults to getOption("do_ssh_keys", NULL) , so you can save as an r option your ssh key , see docs for analogsea::keys and analogsea::droplet_create

[DataStrategist]

fair enough about the ssh keys, but the error was region is not available... the ssh was just a warning. I retried with: do_provision(region='lon1') and it worked a bit further. I think perhaps some input is getting jacked up somewhere and being fed into the region? Strange that it should throw an error, it doesn't even look like the first feature on the droplet_create(). Weird!

Now I'm getting the error:

do_provision(region='lon1')
# THIS ACTION COSTS YOU MONEY!
# Provisioning a new server for which you will get a bill from DigitalOcean.
# Using default ssh keys: 
# NB: This costs $0.00744 / hour until you droplet_delete() it
# Waiting for create ..........
# New server fingerprint: 79:blalblbalblaba.
# Error in ssh::ssh_connect(user_ip, keyfile) : 
#  Authentication failed, permission denied
# In addition: Warning message:
# You have not specified any ssh_keys. This is NOT recommended. (You will receive an email with the root password in a few minutes  

So it seems like the ssh key is mandatory? Or is it an error that it thinks it's mandatory? It asked me for a password, but didn't fire an authentication window per se.

Actually: The droplet WAS created... it just failed to log me in.

[sckott]

so did you get an email with your password?

[DataStrategist]

yessir!

[sckott]

i'm not sure what happened with the region is not available error. i would have done what you did, to try a new region.

using ssh keys is not mandatory, but it's strongly recommended. ssh keys are definitely trickier on windows. If I were you I'd load up ssh package and take your server user name/IP address and ssh key and make sure it works with ssh::ssh_connect, then try to run anything on the server with ssh::ssh_exec_wait -- that's what we use internally here

[raminsu]

Ok, i am working from Rstudio desktop on windows and trying to droplet_create() and do_provision(). I succeeded in droplet_create() and after overcoming both the errors of OP (region and ssh) with help of advise as given above and the statement below:
droplet_create(ssh_keys = analogsea::as.sshkey("keynamegivenonthewebconsole"))
futher testing via ssh for the response
droplet_ssh("droplet", "whoami", user = "root", verbose = TRUE, keyfile = "thelonelyprivatekeyfileonthedesktop")

I am now stuck on this error in do_provision which is very strange to me:
Statement run:
plumber::do_provision("ughghg", example = FALSE, unstable = FALSE)

Usage:
     swapon [options] [<spec>]
Options:
 -a, --all              enable all swaps from /etc/fstab
 -d, --discard          discard freed pages before they are reused
 -e, --ifexists         silently skip devices that do not exis
 -f, --fixpgsz          reinitialize the swap space if necessary
 -h, --help             display help and exit
 -p, --priority <prio>  specify the priority of the swap device.
 -s, --summary          display summary about used swap devices and exit
 -v, --verbose          verbose mode
 -V, --version          display version and exit
The <spec> parameter:
 -L <label>             LABEL of device to be used
 -U <uuid>              UUID of device to be used
 LABEL=<label>          LABEL of device to be used
 UUID=<uuid>            UUID of device to be used
 <device>               name of device to be used
 <file>                 name of file to be used

2 C:\Users\user\AppData\Local\Temp\RtmpoXjqvT\file5402a7513a1/qRxqfdJIVp
Error in file(con, "wb") : cannot open the connection
In addition: Warning message:
In file(con, "wb") :
  cannot open file 'C:\Users\user\AppData\Local\Temp\RtmpoXjqvT\file5402a7513a1/qRxqfdJIVp': No such file or directory `_

Even tried running as administrator, no change.
Thanks
Session info

R version 3.4.2 (2017-09-28)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_India.1252  LC_CTYPE=English_India.1252   
[3] LC_MONETARY=English_India.1252 LC_NUMERIC=C                  
[5] LC_TIME=English_India.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] plumber_0.4.4        analogsea_0.6.0.9111

[sckott]

@raminsu that's an error in trying to add swap memory here https://github.com/trestletech/plumber/blob/master/R/digital-ocean.R#L62

can you use droplet_create(), then ssh in from a shell? let's make sure that works first

[raminsu]

Thanks @sckott for pointing it out. I used droplet_create() and also accessed droplet via ssh from shell. Logged in as root user and couldn't resist trying out these commands,

root@UnreasonableDwelling:~# swapon | grep "/swapfile" | wc -l
Usage:
 swapon [options] [<spec>]

Options:
 -a, --all              enable all swaps from /etc/fstab
 -d, --discard          discard freed pages before they are reused
 -e, --ifexists         silently skip devices that do not exis
 -f, --fixpgsz          reinitialize the swap space if necessary
 -h, --help             display help and exit
 -p, --priority <prio>  specify the priority of the swap device.
 -s, --summary          display summary about used swap devices and exit
 -v, --verbose          verbose mode
 -V, --version          display version and exit

The <spec> parameter:
 -L <label>             LABEL of device to be used
 -U <uuid>              UUID of device to be used
 LABEL=<label>          LABEL of device to be used
 UUID=<uuid>            UUID of device to be used
 <device>               name of device to be used
 <file>                 name of file to be used

0
root@UnreasonableDwelling:~# fallocate -l 4G /swapfile
root@UnreasonableDwelling:~# chmod 600 /swapfile
root@UnreasonableDwelling:~# mkswap /swapfile
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=fa73b382-163e-4968-b33e-ff23401fc9f4
root@UnreasonableDwelling:~# sudo swapon /swapfile
root@UnreasonableDwelling:~# sudo echo \"/swapfile none swap sw 0 0\" >> /etc/fs                                                                                        tab
root@UnreasonableDwelling:~# 

As i understand there seems to be no warning or error when run from within server. I tried to execute the do_provision() and debian_add_swap() at the end from my R session and got this error for both
Error: libssh failure at 'ssh_channel_open_session': Socket error: No error

[sckott]

thanks for this @raminsu - Are you using the version on branch ssh (installed via remotes::install_github("sckott/analogsea@ssh"))

@jeroen any idea what to make of the error:

Error: libssh failure at 'ssh_channel_open_session': Socket error: No error

[smach]

FYI I'm getting the same error message, "Error in file(con, "wb") : cannot open the connection" and then the same sort of error about can't open a file.

[raminsu]

@sckott I used the version on branch ssh via devtools::install_github("sckott/analogsea@ssh") earlier and tried it again but the error persists during do_provision("droplet") Error message: 2 C:\Users\user\AppData\Local\Temp\RtmpkRIDdA\file204460b025c7/kFuzUeFvYR Error in file(con, "wb") : cannot open the connection In addition: Warning message: In file(con, "wb") : cannot open file 'C:\Users\user\AppData\Local\Temp\RtmpkRIDdA\file204460b025c7/kFuzUeFvYR': No such file or directory . However running this command debian_add_swap("droplet") returned Setting up swapspace version 1, size = 4194300 KiB no label, UUID=d1cc7a03-e5fd-4b5c-a6df-0e50f4c00fe9 and then rerun do_provision("droplet") returns the above error again.

[sckott]

@smach can you verify that you used the ssh branch version?

[sckott]

@raminsu let's test if using package ssh works. try this and let me know what happens:

session <- "user@ip-address"
conn <- ssh::ssh_connect(host = session)
ssh_exec_wait(conn, "whoami")

replace user with your user login for the server and ip-address with the IP address for your server
and you may need to pass a path to keyfile and/or a password to passwd parameter in ssh_connect

let me know

[smach]

@sckott Yes. And in fact I tried analogsea::droplet_crate(region = "nyc1") and it worked fine. So I'm not 100% sure the problem I encountered was with analogsea as opposed to something specific in plumber::do_provision(). I tried do_provision() on the existing droplet and had trouble authenticating with my key -- I'll have to try again tonight after work. Thanks very much for all you do regarding R packages!!

[raminsu]

@sckott here is the result of trying out ssh::ssh_connect()
session <- "root@ipaddress"
conn <- ssh::ssh_connect(host = session, keyfile = "pathofkeyfile")
ssh::ssh_exec_wait(conn, "whoami")
root
[1] 0
After that i tried plumber::do_provision() and ended with same error as was mentioned earlier.

[smach]

I can't seem to connect with ssh_connect("ip.address.", keyfile = "myfile.pem"). I have a passphrase for the private key, but after I input that, I'm still being asked for a password. I can ssh to the server fine using PuTTy and the ppk version of the key with passphrase. Maybe I can't use a passphrase for the private key and ssh_connect? I'll probably go back to AWS for the plumber API, but thanks very much for trying to help.

[sckott]

@raminsu glad at least that works, now I can eliminate the possibility that ssh wasn't working

will get back to you

[sckott]

@smach bummer, will get back to you with another thing to try

[sckott]

@smach @raminsu the ssh work is now merged into master - if you could try again I would much appreciate it.

[smach]

@sckott I can now ssh into a droplet, so that's progress! do_provision() still doesn't work, though. When I did do_provision(region = "nyc1") I still got an error

Error in ssh::ssh_connect(user_ip, keyfile) : 
Authentication failed, permission denied

If I create a droplet with droplet_create(region = "nyc1") that works. And this worked:

session <- "root@the.ip.address"
conn <- ssh::ssh_connect(host = session, keyfile = "mykey.pem")
ssh::ssh_exec_wait(conn, "whoami")

But then this

do_provision(droplet = "myDropletName")

resulted in this error still

Error in ssh::ssh_connect(user_ip, keyfile) : 
  Authentication failed, permission denied

Not sure whether the problem is analogsea or plumber.

[varungautam13]

Is there a resolution to this error. I am getting the same errors, with this being the latest:

Error: ssh failed
swapon | grep "/swapfile" | wc -l > /tmp/eamCxYHhGk

[sckott]

@varungautam13 did you try with the dev version installed from github? remotes::install_github("sckott/analogsea")

[sckott]

@smach sorry to hear it's still not working. can you share your sessionInfo() so I can see what versions you have?

[varungautam13]

I have done it again with dev versions, but getting the same error
Error: ssh failed
swapon | grep "/swapfile" | wc -l > /tmp/EWGEeYMeJx

Session Info:
R version 3.5.0 (2018-04-23)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252
[2] LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods
[7] base

other attached packages:
[1] ssh_0.2 analogsea_0.6.5.9110 plumber_0.4.7

loaded via a namespace (and not attached):
[1] Rcpp_0.12.17 digest_0.6.15 crayon_1.3.4
[4] later_0.7.3 aws.signature_0.4.1 R6_2.2.2
[7] jsonlite_1.5 magrittr_1.5 httr_1.3.1
[10] stringi_1.1.7 curl_3.2 remotes_1.1.1
[13] rstudioapi_0.7 promises_1.0.1 xml2_1.2.0
[16] tools_3.5.0 aws.s3_0.3.12 httpuv_1.4.4.2
[19] yaml_2.1.19 compiler_3.5.0 base64enc_0.1-3
[22] openssl_1.0.1

[RenanOm92]

Any updates on the error that @raminsu reported?

I'm also getting this Error in file(con, "wb") : cannot open the connection on my Temp folder when executing do_provision

SSH connection is working fine and I'm using the latest dev version available on GitHub. I also executed RStudio as admin and still getting the same error. I'm using windows.

Full error: 2 C:\Users\rmoreira\AppData\Local\Temp\RtmpEHQl6M\file2d8c72313351/LfvremOUeE Error in file(con, "wb") : cannot open the connection In addition: Warning message: In file(con, "wb") : cannot open file 'C:\Users\rmoreira\AppData\Local\Temp\RtmpEHQl6M\file2d8c72313351/LfvremOUeE': No such file or directory

Thanks, mate.

[sckott]

thanks for the ping @RenanOm92 - on vacation now, will respond mid next week

[aminadibi]

Waiting for create .......
Error in cli_tools() : 
ssh, scp not found on your computer
Install the missing tool(s) and try again

Same problem here. do_provision doesn't work, even though I have Ubuntu bash installed on Windows 10 and ssh and scp both seem to be available through terminal in RStudio:

$ ssh
usage: ssh [-46AaCfGgKkMNnqsTtVvXxYy] [-B bind_interface]
           [-b bind_address] [-c cipher_spec] [-D [bind_address:]port]
           [-E log_file] [-e escape_char] [-F configfile] [-I pkcs11]
           [-i identity_file] [-J [user@]host[:port]] [-L address]
           [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]
           [-Q query_option] [-R address] [-S ctl_path] [-W host:port]
           [-w local_tun[:remote_tun]] destination [command]

$ scp
usage: scp [-346BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
           [-l limit] [-o ssh_option] [-P port] [-S program] source ... target

Using anologsea v‘0.6.5.9110’ on R 3.5.1 on Windows 10 with Ubuntu bash.

[pmirla]

Same ssh here. Any advice to fix this? Thanks

mydrop <- plumber::do_provision(region = 'nyc1')
THIS ACTION COSTS YOU MONEY!
Provisioning a new server for which you will get a bill from DigitalOcean.
Using default ssh keys: plumber
NB: This costs $0.00744 / hour until you droplet_delete() it
Waiting for create ..........
Warning: Permanently added '68.183.100.42' (ECDSA) to the list of known hosts.
root@68.183.100.42: Permission denied (publickey).
Error: ssh failed
ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/var/folders/b2/z6kcqdvn5gj7qqcy19pvw8j40000gn/T//RtmpkvcIiU/hosts root@68.183.100.42 'swapon | grep "/swapfile" | wc -l > /tmp/JqIbCtcfag'

[williamjwest]

Hello! I'm having an issue at the same place. I am able to connect to the server with the key supplied by ssh_keys.

sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS 10.14

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] analogsea_0.6.5.9110 plumber_0.4.7

loaded via a namespace (and not attached):
[1] Rcpp_1.0.0 ssh_0.3 digest_0.6.18 crayon_1.3.4 later_0.7.5 aws.signature_0.4.4
[7] R6_2.3.0 jsonlite_1.6 magrittr_1.5 httr_1.4.0 stringi_1.2.4 curl_3.2
[13] rstudioapi_0.8 promises_1.0.1 xml2_1.2.0 tools_3.5.1 aws.s3_0.3.12 httpuv_1.4.5
[19] yaml_2.2.0 compiler_3.5.1 base64enc_0.1-3 openssl_1.1

ids <- plumber::do_provision(unstable = TRUE, example = FALSE, region="nyc3",verbose=TRUE)
THIS ACTION COSTS YOU MONEY!
Provisioning a new server for which you will get a bill from DigitalOcean.
NB: This costs $0.00744 / hour until you droplet_delete() it
Waiting for create .......................
New server fingerprint: d2:97:e5:dc:de:6f:76:d3:63:55:22:2e:cd:5c:65:6c:bd:19:06:0c
Error in ssh::ssh_connect(user_ip, keyfile) :
Authentication failed, permission denied

[sergey-shchemenok]

Hello, I tried to deploy my API and get Error in file(con, "wb") : cannot open the connection

do_provision(unstable = TRUE, region = 'sgp1')
THIS ACTION COSTS YOU MONEY!
Provisioning a new server for which you will get a bill from DigitalOcean.
Adding .httr-oauth to .gitignore
Waiting for authentication in browser...
Press Esc/Ctrl + C to abort
Authentication complete.
Using default ssh keys: Caelestis
NB: This costs $0.00744 / hour until you droplet_delete() it
Waiting for create ..................
New server fingerprint: ba:91:9f:ba:88:93:f7:54:f9:2c:5d:9a:c3:ac:4d:eb:db:6d:41:06
2 C:\Users\CAELES1\AppData\Local\Temp\RtmpuYaaRl\file74c014d55ca2/JmoftZKayv
Error in file(con, "wb") : cannot open the connection
In addition: Warning message:
In file(con, "wb") :
cannot open file 'C:\Users\CAELES1\AppData\Local\Temp\RtmpuYaaRl\file74c014d55ca2/JmoftZKayv': No such file or directory

[sckott]

crap, apologies for the long delay on this, having a look now

[sckott]

@varungautam13 @sergey-shchemenok @williamjwest @pmirla @aminadibi @RenanOm92

Please try again after installing the development version of this package remotes::install_github("sckott/analogsea")(remember to restart the R instance) - the dev version relies on thessh` R package that wraps libssh2 to do ssh work instead of shelling out to ssh/scp

[sckott]

there's a few different errors happening above:

1. Permission denied (publickey) - an authentication error as you know - try installing the version here and see if that fixes it. Note that if you have a passphrase on your key you need probably want to make it so you don't have to enter your password each time, e.g., https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent

2. the cannnot open file class of errors I think is a problem in the scp operation, see Deploying Plumber on DigitalOcean rstudio/plumber#361 (comment) for discussion - there was a problem in this pkg that I fixed - but there's also some issues with plumber that I havent't been able to figure out yet - so waiting on a reply from them

[sckott]

@varungautam13 @sergey-shchemenok @williamjwest @pmirla @aminadibi @RenanOm92

Jeroen has been making some changes to the ssh package that may help here, see https://github.com/ropensci/ssh/blob/master/NEWS#L1-L5 -- Can you reinstall remotes::install_github("ropensci/analogsea") (which will install the dev version of ssh) and try again? Making sure to reaload R session.

[varungautam13]

No, I have not tried that in some time, will try now.

…

On Mon, 9 Jul, 2018, 10:13 PM Scott Chamberlain, ***@***.***> wrote: @varungautam13 <https://github.com/varungautam13> did you try with the dev version installed from github? remotes::install_github("sckott/analogsea") — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#154 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALJfBD4ZJpYAfKadQw6W2OV4yKARNqV8ks5uE4gcgaJpZM4SPtQg> .

[akando42]

This works for me.

mydrop <- plumber::do_provision(region="sfo2")

[sckott]

@troydo42 great, thx for trying and glad it works, what OS are you on? and did you try that with the CRAN ver of analogsea or the dev version?

[jeroen]

The new version is on cran so you can just do: update.packages(ask=FALSE)

[sckott]

closing now, submitting new version to cran very soon