shinytest doesn't capture any outputs from learnr doc

[schloerke]

@wlandau from rstudio/learnr#237

I am having trouble using shinytest on prerendered R Markdown documents (reprex here). The example learnr tutorial works fine with rmarkdown::run(), but recordTest() seems to think I am calling runApp() from within runApp(). I get the same error when I try to deploy similar apps to my company's instance of Shiny Server.

[wlandau]

Thanks, Barret. When I got that error, I had an app.R that called learnr::run_tutorial(), so the inception turned out to make sense. That particular error went away after I treated the R Markdown document as an app inside inst.

I am actually encountering a different set of problems now. When I use shinytest on R Markdown documents, most of the testing artifacts appear to be missing. Example test script:

app <- ShinyDriver$new("../index.Rmd", seed = 0)
app$snapshotInit("open")

app$snapshot()

and 001.json:

{
  "input": {

  },
  "output": {

  },
  "export": {

  }
}

I have been meaning to post a reprex, but I have been short on time lately.

[wlandau]

I have uploaded an example of the problem I mentioned before (now described here).

[hadley]

@wlandau do you still see this problem?

[wlandau]

Just revisited https://github.com/wlandau/shinytest-learnr-reprex with rstudio/learnr@51b4fcb and fa31169 on my work's RStudio Server (R 3.6.3, RHEL 7.8). Without an app.R, the testing files from the recording are no longer blank. With an app.R, I still get the same error as before. Both tests on Mac OS still return "Error in process_get_error_connection(self, private) : stderr is not a pipe." (R 4.0.2, Mac OS 10.15.6).

[hadley]

@schloerke how about this one?

[hadley]

@wlandau it doesn't make sense to test learnr docs with an app.R, so I don't think we need to consider that case. For the pure Rmd case, could you make a reprex by using ShinyDriver directly? Something like:

library(shinytest)
app <- ShinyDriver$new("path/to/doc.Rmd")
app$snapshotInit("mytest")
app$snapshot()

[wlandau]

Sure, Hadley. The following seems to produce believable output on my work's RHEL machine.

# Ran without reprex to avoid this error: "Functions that produce HTML output found in document targeting 
content <- '---
title: "Example"
output: learnr::tutorial
runtime: shiny_prerendered
---

```{r q, echo = FALSE}
markdown_strict output."
library(learnr)
question(
  "Okay?",
  correct = "Yup.",
  incorrect = "Nope.",
  allow_retry = TRUE,
  answer("Yes", correct = TRUE),
  answer("No")
)
```
'
writeLines(content, "tutorial.Rmd")
library(shinytest)
app <- ShinyDriver$new("tutorial.Rmd")
app$snapshotInit("mytest")
app$snapshot()
list.files("tests/mytest-current")
#> [1] "001.json" "001.png" 
cat(readLines("tests/mytest-current/001.json", n = 10), sep = "\n")
#>{
#>  "input": {
#>    "tutorial-topic": null,
#>    "tutorial-visible-sections": null
#>  },
#>  "output": {
#>    "q-answer_container": {
#>      "html": "<div class=\"loading\"><strong>Loading:<\/strong> \nOkay?\n<br/><br/><br/><\/div>",
#>     "deps": [

I still get an error on my work Macbook (at ShinyDriver$new("tutorial.Rmd")) but I narrowed it down to webdriver. I will post an issue there.

> webdriver::run_phantomjs()
Error in process_get_error_connection(self, private) : 
  stderr is not a pipe.
Type .Last.error.trace to see where the error occured
> .Last.error.trace

 Stack trace:

 1. webdriver::run_phantomjs()
 2. base:::stop("Cannot start phantom.js, or cannot connect to it",  ...
 3. base:::strwrap(ph$read_error_lines())
 4. ph$read_error_lines()
 5. processx:::process_read_error_lines(self, private, n)
 6. processx:::process_get_error_connection(self, private)
 7. throw(new_error("stderr is not a pipe."))

 x stderr is not a pipe. 

[hadley]

Great, thanks for following up on this!