Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stderr is not a pipe (Mac OS) #59

Closed
wlandau opened this issue Aug 14, 2020 · 23 comments · Fixed by #62
Closed

stderr is not a pipe (Mac OS) #59

wlandau opened this issue Aug 14, 2020 · 23 comments · Fixed by #62

Comments

@wlandau
Copy link

wlandau commented Aug 14, 2020

I get a strange error when I try to run PhantomJS through webdriver on Mac OS. The same code seems to run without error on Linux. Related: rstudio/shinytest#259.

> packageDescription("webdriver")$GithubSHA1
[1] "7bec71e8e2d0300a29e48ff780bf1a223fdaff76"
> webdriver::install_phantomjs()
...
> rstudioapi::restartSession()
> webdriver::run_phantomjs()
Error in process_get_error_connection(self, private) : 
  stderr is not a pipe.
Type .Last.error.trace to see where the error occured
> .Last.error.trace

 Stack trace:

 1. webdriver::run_phantomjs()
 2. base:::stop("Cannot start phantom.js, or cannot connect to it",  ...
 3. base:::strwrap(ph$read_error_lines())
 4. ph$read_error_lines()
 5. processx:::process_read_error_lines(self, private, n)
 6. processx:::process_get_error_connection(self, private)
 7. throw(new_error("stderr is not a pipe."))

 x stderr is not a pipe. 

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] png_0.1-7            ps_1.3.4             withr_2.2.0          crayon_1.3.4         packrat_0.5.0        digest_0.6.25       
 [7] showimage_1.0.0      R6_2.4.1             jsonlite_1.7.0       evaluate_0.14        httr_1.4.2           debugme_1.1.0       
[13] rlang_0.4.7          webdriver_1.0.5.9000 curl_4.3             rstudioapi_0.11      callr_3.4.3          rmarkdown_2.3       
[19] tools_4.0.2          xfun_0.16            yaml_2.2.1           compiler_4.0.2       processx_3.4.3       base64enc_0.1-3     
[25] htmltools_0.5.0.9000 knitr_1.29
@gaborcsardi
Copy link
Collaborator

I cannot reproduce this, can you pls use reprex::reprex() to create a reproducible example? Thanks!

@gaborcsardi
Copy link
Collaborator

It seems like an error happens when trying to report the original error :/, which is "Cannot start phantom.js, or cannot connect to it". So please check where phantom.js is installed, and that you can use it. You can use webdriver:::find_phantom() to find it.

@wlandau
Copy link
Author

wlandau commented Aug 14, 2020

I did not use reprex before because of tidyverse/reprex#230, but here is one.

webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.

Created on 2020-08-14 by the reprex package (v0.3.0)

webdriver:::find_phantom() shows "/Users/<CENSORED>/Library/Application Support/PhantomJS/phantomjs". I can run it in Terminal.

$ /Users/<CENSORED>/Library/Application\ Support/PhantomJS/phantomjs
phantomjs>

@shirdekel
Copy link

I also have this issue:

webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.

Created on 2020-09-01 by the reprex package (v0.3.0)

Session info
devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  ctype    en_AU.UTF-8                 
#>  tz       Australia/Sydney            
#>  date     2020-09-01                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.0)
#>  backports     1.1.8   2020-06-17 [1] CRAN (R 4.0.0)
#>  base64enc     0.1-3   2015-07-28 [1] CRAN (R 4.0.0)
#>  callr         3.4.3   2020-03-28 [1] CRAN (R 4.0.0)
#>  cli           2.0.2   2020-02-28 [1] CRAN (R 4.0.0)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.0)
#>  curl          4.3     2019-12-02 [1] CRAN (R 4.0.0)
#>  debugme       1.1.0   2017-10-22 [1] CRAN (R 4.0.2)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 4.0.0)
#>  devtools      2.3.0   2020-04-10 [1] CRAN (R 4.0.0)
#>  digest        0.6.25  2020-02-23 [1] CRAN (R 4.0.0)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.0)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.1   2020-05-13 [1] CRAN (R 4.0.0)
#>  highr         0.8     2019-03-20 [1] CRAN (R 4.0.0)
#>  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.0)
#>  httr          1.4.2   2020-07-20 [1] CRAN (R 4.0.2)
#>  jsonlite      1.7.0   2020-06-25 [1] CRAN (R 4.0.1)
#>  knitr         1.29    2020-06-23 [1] CRAN (R 4.0.2)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 4.0.0)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 4.0.0)
#>  pkgbuild      1.1.0   2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgload       1.1.0   2020-05-29 [1] CRAN (R 4.0.0)
#>  png           0.1-7   2013-12-03 [1] CRAN (R 4.0.0)
#>  prettyunits   1.1.1   2020-01-24 [1] CRAN (R 4.0.0)
#>  processx      3.4.3   2020-07-05 [1] CRAN (R 4.0.2)
#>  ps            1.3.3   2020-05-08 [1] CRAN (R 4.0.0)
#>  R6            2.4.1   2019-11-12 [1] CRAN (R 4.0.0)
#>  remotes       2.1.1   2020-02-15 [1] CRAN (R 4.0.0)
#>  rlang         0.4.7   2020-07-09 [1] CRAN (R 4.0.1)
#>  rmarkdown     2.3     2020-06-18 [1] CRAN (R 4.0.0)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 4.0.0)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  showimage     1.0.0   2018-01-24 [1] CRAN (R 4.0.2)
#>  stringi       1.4.6   2020-02-17 [1] CRAN (R 4.0.0)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  testthat      2.3.2   2020-03-02 [1] CRAN (R 4.0.0)
#>  usethis       1.6.1   2020-04-29 [1] CRAN (R 4.0.0)
#>  webdriver     1.0.5   2018-04-11 [1] CRAN (R 4.0.2)
#>  withr         2.2.0   2020-04-20 [1] CRAN (R 4.0.0)
#>  xfun          0.16    2020-07-24 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

@shirdekel
Copy link

But it's weird, because it does work sometimes. Here's a sample of 20:

webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25288.
#> 
#> $port
#> [1] 8431
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25290.
#> 
#> $port
#> [1] 7912
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25291.
#> 
#> $port
#> [1] 6088
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25292.
#> 
#> $port
#> [1] 6142
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25293.
#> 
#> $port
#> [1] 3271
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25294.
#> 
#> $port
#> [1] 4026
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25295.
#> 
#> $port
#> [1] 5722
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25297.
#> 
#> $port
#> [1] 3331
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25298.
#> 
#> $port
#> [1] 5912
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25299.
#> 
#> $port
#> [1] 5187
  webdriver::run_phantomjs()
#> $process
#> PROCESS 'phantomjs', running, pid 25300.
#> 
#> $port
#> [1] 5319
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.
  webdriver::run_phantomjs()
#> Error in process_get_error_connection(self, private): stderr is not a pipe.

Created on 2020-09-02 by the reprex package (v0.3.0)

Session info
devtools::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS Mojave 10.14.6        
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_AU.UTF-8                 
#>  ctype    en_AU.UTF-8                 
#>  tz       Australia/Sydney            
#>  date     2020-09-02                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date       lib source        
#>  assertthat    0.2.1   2019-03-21 [1] CRAN (R 4.0.0)
#>  backports     1.1.8   2020-06-17 [1] CRAN (R 4.0.0)
#>  base64enc     0.1-3   2015-07-28 [1] CRAN (R 4.0.0)
#>  callr         3.4.3   2020-03-28 [1] CRAN (R 4.0.0)
#>  cli           2.0.2   2020-02-28 [1] CRAN (R 4.0.0)
#>  crayon        1.3.4   2017-09-16 [1] CRAN (R 4.0.0)
#>  curl          4.3     2019-12-02 [1] CRAN (R 4.0.0)
#>  debugme       1.1.0   2017-10-22 [1] CRAN (R 4.0.2)
#>  desc          1.2.0   2018-05-01 [1] CRAN (R 4.0.0)
#>  devtools      2.3.0   2020-04-10 [1] CRAN (R 4.0.0)
#>  digest        0.6.25  2020-02-23 [1] CRAN (R 4.0.0)
#>  ellipsis      0.3.1   2020-05-15 [1] CRAN (R 4.0.0)
#>  evaluate      0.14    2019-05-28 [1] CRAN (R 4.0.0)
#>  fansi         0.4.1   2020-01-08 [1] CRAN (R 4.0.0)
#>  fs            1.5.0   2020-07-31 [1] CRAN (R 4.0.2)
#>  glue          1.4.1   2020-05-13 [1] CRAN (R 4.0.0)
#>  highr         0.8     2019-03-20 [1] CRAN (R 4.0.0)
#>  htmltools     0.5.0   2020-06-16 [1] CRAN (R 4.0.0)
#>  httr          1.4.2   2020-07-20 [1] CRAN (R 4.0.2)
#>  jsonlite      1.7.0   2020-06-25 [1] CRAN (R 4.0.1)
#>  knitr         1.29    2020-06-23 [1] CRAN (R 4.0.2)
#>  magrittr      1.5     2014-11-22 [1] CRAN (R 4.0.0)
#>  memoise       1.1.0   2017-04-21 [1] CRAN (R 4.0.0)
#>  pkgbuild      1.1.0   2020-07-13 [1] CRAN (R 4.0.2)
#>  pkgload       1.1.0   2020-05-29 [1] CRAN (R 4.0.0)
#>  png           0.1-7   2013-12-03 [1] CRAN (R 4.0.0)
#>  prettyunits   1.1.1   2020-01-24 [1] CRAN (R 4.0.0)
#>  processx      3.4.3   2020-07-05 [1] CRAN (R 4.0.2)
#>  ps            1.3.3   2020-05-08 [1] CRAN (R 4.0.0)
#>  R6            2.4.1   2019-11-12 [1] CRAN (R 4.0.0)
#>  remotes       2.1.1   2020-02-15 [1] CRAN (R 4.0.0)
#>  rlang         0.4.7   2020-07-09 [1] CRAN (R 4.0.1)
#>  rmarkdown     2.3     2020-06-18 [1] CRAN (R 4.0.0)
#>  rprojroot     1.3-2   2018-01-03 [1] CRAN (R 4.0.0)
#>  sessioninfo   1.1.1   2018-11-05 [1] CRAN (R 4.0.0)
#>  showimage     1.0.0   2018-01-24 [1] CRAN (R 4.0.2)
#>  stringi       1.4.6   2020-02-17 [1] CRAN (R 4.0.0)
#>  stringr       1.4.0   2019-02-10 [1] CRAN (R 4.0.0)
#>  testthat      2.3.2   2020-03-02 [1] CRAN (R 4.0.0)
#>  usethis       1.6.1   2020-04-29 [1] CRAN (R 4.0.0)
#>  webdriver     1.0.5   2018-04-11 [1] CRAN (R 4.0.2)
#>  withr         2.2.0   2020-04-20 [1] CRAN (R 4.0.0)
#>  xfun          0.16    2020-07-24 [1] CRAN (R 4.0.2)
#>  yaml          2.2.1   2020-02-01 [1] CRAN (R 4.0.0)
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

I thought the ports might be relevant, but I still got the error after rewriting the function with a fixed port that previously worked.

gaborcsardi added a commit that referenced this issue Sep 5, 2020
Will help with issues like #59.
@gaborcsardi
Copy link
Collaborator

@wlandau @shirdekel Can you try installing #60? (E.g. with remotes::install_github("rstudio/webdriver#60")). This will at least print the real output from phantom.js.

@wlandau
Copy link
Author

wlandau commented Sep 5, 2020

Here is what run_phantomjs() shows after I install #60:

Error in webdriver::run_phantomjs() : 
  Cannot start phantom.js, or cannot connect to it. stdout + stderr:
>  [INFO  - 2020-09-05T18:26:46.690Z] GhostDriver - Main - running on port 5480

@shirdekel
Copy link

I get this:

Error in webdriver::run_phantomjs() : 
  Cannot start phantom.js, or cannot connect to it. stdout + stderr:
>  

@gaborcsardi
Copy link
Collaborator

@shirdekel Does webdriver:::find_phantom() find phantom.js? Can you run it from the terminal?

@wlandau So it seems that phantom is running, actually. Do you have a local firewall that prevents connections?

@wlandau
Copy link
Author

wlandau commented Sep 8, 2020

Yes, I do have a local firewall. It does apparently allow incoming connections from phantom, but I do not know about outgoing ones. My work's installation of Mac OS does not allow me to change these settings.

@gaborcsardi
Copy link
Collaborator

@wlandau R must be able to connect to phantom, on the port that is randomly selected. If it cannot connect, then there is no way to run webdriver, I am afraid.

You can try to run phantom from the terminal:

❯ "/Users/gaborcsardi/Library/Application Support/PhantomJS/phantomjs" --webdriver=
[INFO  - 2020-09-08T12:21:39.525Z] GhostDriver - Main - running on port 8910

and then connect to http://127.0.0.1:8910 from a browser, or from R:

httr::GET("http://127.0.0.1:8910/")
Response [http://127.0.0.1:8910/]
  Date: 2020-09-08 13:22
  Status: 404
  Content-Type: text/plain
  Size: 467 B

If port 8910 works, then you can try with a port returned by webdriver:::random_port().

@wlandau
Copy link
Author

wlandau commented Sep 8, 2020

I got a 503 from httr::GET("http://127.0.0.1:8910") with phantom running on port 8910 in a separate process. If that's because of the firewall, I guess it can't be helped in my case.

> httr::GET("http://127.0.0.1:8910/")
Response [http://40.26.201.184/]
  Date: 2020-09-08 14:10
  Status: 503
  Content-Type: text/html; charset=utf-8
  Size: 2.06 kB
<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Website issue</title>
	<style type="text/css">
		* { margin:0; padding:0; }
		body { background:#dbe0e6; text-align:center; }
		#main_container  { margin:20px auto; width:500px; }
		#header { height:75px; border-bottom:1px solid #dbe0e6; }
		#content_container { float:left; margin-top:-5px; background:#ffffff; }
		#image { width:168px; padding:2px 2px 2px 2px; text-align:center; backgroun...
		#content { padding-top:15px; width:328px; float:right; text-align:left; }
...

What do you think about a slightly more verbose error message that mentions webdriver:::find_phantom() and possible firewall issues?

@gaborcsardi
Copy link
Collaborator

So is http://40.26.201.184 your proxy server? Maybe you can turn that off somehow? At least 127.0.0.1 should not go to a proxy.

Yeah, the error should be improved.

@wlandau
Copy link
Author

wlandau commented Sep 8, 2020

It could be, I am not sure. But just knowing this is a firewall or proxy issue helps a ton and lets me find workarounds. Seems to me like an improved error message is sufficient to close the issue.

@shirdekel
Copy link

shirdekel commented Sep 8, 2020

@shirdekel Does webdriver:::find_phantom() find phantom.js? Can you run it from the terminal?

Yes to both.

I followed your instructions for @wlandau and got the following:

"/Users/shirdekel/Library/Application Support/PhantomJS/phantomjs" --webdriver=
[INFO  - 2020-09-08T23:38:28.878Z] GhostDriver - Main - running on port 8910
httr::GET("http://127.0.0.1:8910/")
Response [http://127.0.0.1:8910/]
  Date: 2020-09-09 09:40
  Status: 404
  Content-Type: text/plain
  Size: 463 B

But I subsequently got the following error in terminal:

[ERROR - 2020-09-08T23:40:54.943Z] RouterReqHand - _handle.error - {"name":"Unknown Command","message":"{\"headers\":{\"Accept\":\"application/json, text/xml, application/xml, */*\",\"Accept-Encoding\":\"deflate, gzip\",\"Host\":\"127.0.0.1:8910\",\"User-Agent\":\"libcurl/7.54.0 r-curl/4.3 httr/1.4.2\"},\"httpVersion\":\"1.1\",\"method\":\"GET\",\"url\":\"/\",\"urlParsed\":{\"anchor\":\"\",\"query\":\"\",\"file\":\"\",\"directory\":\"/\",\"path\":\"/\",\"relative\":\"/\",\"port\":\"\",\"host\":\"\",\"password\":\"\",\"user\":\"\",\"userInfo\":\"\",\"authority\":\"\",\"protocol\":\"\",\"source\":\"/\",\"queryKey\":{},\"chunks\":[\"\"]}}","line":83,"sourceURL":"phantomjs://code/router_request_handler.js","stack":"_handle@phantomjs://code/router_request_handler.js:83:68"}

  phantomjs://platform/console++.js:263 in error

@gaborcsardi
Copy link
Collaborator

@shirdekel Try to install a newer/different phantomjs version. You can also install it with webdriver::install_phantomjs().

@shirdekel
Copy link

@shirdekel Try to install a newer/different phantomjs version. You can also install it with webdriver::install_phantomjs().

@gaborcsardi I did, and still am getting this error sometimes. It might be dependent on something in my R process because I have periods of time in which there is no error, and then suddenly I only get this error. After restarting R it works fine. I guess I'll keep monitoring it and update if I can identify a cause on my end.

@PhilippPro
Copy link

I have the same error and nothing worked so far, to solve it. :(

@arnaud-feldmann
Copy link

https://github.com/InseeFr/disaggR/runs/1439024221
I got the same problem on my CI, only with macos

@PhilippPro
Copy link

I got it solved. It had to do with the .Renviron file which set http-Proxys which were not suitable with phantomjs.

@wch
Copy link
Collaborator

wch commented Dec 18, 2020

@gaborcsardi So would it help to have random_port() check that a port is actually available before returning that value?

It seems like there could be two problems:

  1. The port isn't available, so phantom won't start.
  2. The port is blocked by a firewall, so even if phantom starts, the R process can't connect to it.

I guess it's not clear to me which problem people are running into here.

@gaborcsardi
Copy link
Collaborator

gaborcsardi commented Dec 18, 2020

Having to choose a port is unfortunate, the OS should provide an available one, but apparently phantom does not support that. Nevertheless a port number clash should be quite rare.

I would guess that the proxy and the firewall issues are more common. We can probably try to detect them.
But for a quick improvement a better error message would help a lot I think.

@wch
Copy link
Collaborator

wch commented Dec 18, 2020

If I understand correctly, this error happens when there's a problem starting phantomjs, which is probably due to problems opening the port:

Error in process_get_error_connection(self, private): stderr is not a pipe.

This error would also indicate a problem starting, but we don't seem to actually see anyone reporting this error:

Failed to start phantomjs. Error: ....

This error would indicate that phantomjs started, but there was a problem connecting to it, likely due to a firewall or proxy:

Cannot start phantom.js, or cannot connect to it

Is that right?

If that's the correct, then maybe we should change the second message to:

phantomjs started, but cannot connect to it on port 1234

@wch wch closed this as completed in #62 Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants