Checking for exercise errors (#403)

* empty_results()'s html_output needs to req()-able; otherwise, a value is never actually returned (follow up to #235)

* Refactor evaluate_exercise() and add exercise.error.checker for running a checker function on exercise evaluation errors, closes #356

* code review feedback

* knitr fig options were actually needed

* shorter tempfile pattern

* docs

* Provide an exercise.error.checker option

* Document new exercise.error.checker option

* update news

NEWS.md

@@ -6,13 +6,15 @@ learnr (development version)
 * Renamed the `exercise_submission` event to `exercise_result` and added the following fields:
   1. `id` - a randombly generated identifier that can be used to align with the associated `exercise_result` event.
   2. `time_elapsed` - the time required to run the exercise (in seconds)
-  3. `timeout_exceeded` - indicates whether the exercise was interrupted due to an exceeded timeout. May be `NA` for some platforms/evaluators if that information is not known or reported. ([#337](https://github.com/rstudio/learnr/pull/337))
+  3. `timeout_exceeded` - indicates whether the exercise was interrupted due to an exceeded timeout. May be `NA` for some platforms/evaluators if that information is not known or reported. ([#337](https://github.com/rstudio/learnr/pull/337))  
+* If a `-code-check` chunk returns feedback for an exercise submission, the result of the exercise is no longer displayed for a correct answer (only the feedback is displayed). If both the result and feedback should be displayed, all checking should be performed in a `-check` chunk (i.e., don't provide a `-code-check` chunk). ([#403](https://github.com/rstudio/learnr/pull/403))
 
 ## New features
 
 * Introduced "setup chunk chaining", which allows a setup chunk to depend on another setup chunk and so on, forming a chain of setup code that can be used for exercises via `exercise.setup`. Run `run_tutorial("setup-chunks", "learnr")` for a demo. ([#390](https://github.com/rstudio/learnr/pull/390))
 * Introduced an [experimental](https://www.tidyverse.org/lifecycle/#experimental) function `external_evaluator()` which can be used to define an exercise evaluator that runs on a remote server and is invoked via HTTP. This allows all exercise execution to be performed outside of the Shiny process hosting the learnr document. ([#345](https://github.com/rstudio/learnr/pull/345), [#354](https://github.com/rstudio/learnr/pull/354))
 * For the "forked" evaluator (the default used on Linux), add a limit to the number of forked exercises that learnr will execute in parallel. Previously, this was uncapped, which could cause a learnr process to run out of memory when an influx of traffic arrived. The default limit is 3, but it can be configured using the `tutorial.max.forked.procs` option or the `TUTORIAL_MAX_FORKED_PROCS` environment variable. ([#353](https://github.com/rstudio/learnr/pull/353))
+* Added a new `tutorial_options()`, namely `exercise.error.checker`, for customizing feedback when exercise submission code produces an evaluation error. This option accepts a function with the same arguments as `exercise.checker`. Use `gradethis::grade_learnr_error()` for a sensible default for this option. ([#403](https://github.com/rstudio/learnr/pull/403))
 * Added an event handler system, with the functions `event_register_handler()` and `one_time()`. There is also a new event `"section_viewed"`, which is triggered when a new section becomes visible. ([#398](https://github.com/rstudio/learnr/pull/398))
 * Previously, when a question submission was reset, it would be recorded as a `"question_submission"` event with the value `reset=TRUE`. Now it a separate event, `"reset_question_submission"`. ([#398](https://github.com/rstudio/learnr/pull/398))
 
@@ -22,7 +24,7 @@ learnr (development version)
 * Added a `restore` flag on `exercise_submitted` events which is `TRUE` if the exercise is being restored from a previous execution, or `FALSE` if the exercise is being run interactively.
 * Add `label` field to the `exercise_hint` event to identify for which exercise the user requested a hint. ([#377](https://github.com/rstudio/learnr/pull/377))
 * Added `include=FALSE` to setup chunks to prevent exercises from printing out messages or potential code output for those setup chunks. ([#390](https://github.com/rstudio/learnr/pull/390))
-* Added error handling when user specificies a non-existent label for `exercise.setup` option with an error message. ([#390](https://github.com/rstudio/learnr/pull/390))
+* Added error handling when user specifies a non-existent label for `exercise.setup` option with an error message. ([#390](https://github.com/rstudio/learnr/pull/390))
 * We no longer forward the checker code to browser (in html), but instead cache it. ([#390](https://github.com/rstudio/learnr/pull/390))
 * We no longer display an invisible exercise result warning automatically. Instead, authors must set the exercise chunk option `exercise.warn_invisible = TRUE` to display an invisible result warning message. ([#373](https://github.com/rstudio/learnr/pull/373))
 

R/evaluators.R

@@ -18,7 +18,7 @@ inline_evaluator <- function(expr, timelimit, ...) {
         error = function(e) {
           # TODO: could grepl the error message to determine if the error was due
           # to an exceeded timeout.
-          error_result(e$message, timeout_exceeded = NA)
+          exercise_result_error(e$message, timeout_exceeded = NA)
         }
       )
     },
@@ -111,7 +111,7 @@ setup_forked_evaluator_factory <- function(max_forked_procs){
 
           # check if it's an error and convert it to an html error if it is
           if(inherits(result, "try-error"))
-            result <<- error_result(result, timeout_exceeded = FALSE)
+            result <<- exercise_result_error(result, timeout_exceeded = FALSE)
 
           TRUE
         }
@@ -126,7 +126,7 @@ setup_forked_evaluator_factory <- function(max_forked_procs){
           running_exercises <<- running_exercises - 1
 
           # return error result
-          result <<- error_result(timeout_error_message(), timeout_exceeded = TRUE)
+          result <<- exercise_result_timeout()
           TRUE
         }
 
@@ -244,7 +244,7 @@ internal_external_evaluator <- function(
 
             print("Error submitting external exercise:")
             print(err)
-            result <<- error_result("Error submitting external exercise. Please try again later")
+            result <<- exercise_result_error("Error submitting external exercise. Please try again later")
           }
 
           curl::curl_fetch_multi(url, handle = handle, done = done_cb, fail = fail_cb, pool = pool)
@@ -280,7 +280,7 @@ internal_external_evaluator <- function(
           },
           onRejected = function(err){
             print(err)
-            result <<- error_result("Error initiating session for external requests. Please try again later")
+            result <<- exercise_result_error("Error initiating session for external requests. Please try again later")
           }
         )
       },