Vignette updates (#170)

* create intro simChef vignette

* update comprehensive simChef vignette

* add vignette for inputs/outputs cheat sheet

* restructure docs folder to accomodate multiple example experiments

Close #160

.Rbuildignore

@@ -6,7 +6,7 @@
 ^LICENSE\.md$
 ^\.github$
 ^vignettes/results$
-^vignettes/docs$
+^vignettes/example-docs$
 ^man-roxygen
 ^.*\.Rproj$
 ^\.Rproj\.user$

README.md

@@ -11,6 +11,8 @@ experiment <- create_experiment() %>%
   add_dgp(dgp1) %>%
   add_dgp(dgp2) %>%
   add_method(method1) %>%
+  add_evaluator(evaluator1) %>%
+  add_visualizer(visualizer1) %>%
   add_vary_across(
     dgp = dgp1,
     n = c(100, 1000, 10000)
@@ -21,7 +23,7 @@ experiment <- create_experiment() %>%
   )
 
 results <- experiment %>%
-  run_experiment()
+  run_experiment(n_reps = 100)
 ```
 
 ### Installation

vignettes/cheat-sheet.Rmd

@@ -0,0 +1,79 @@
+---
+title: "Cheat sheet: Inputs/outputs for user-defined functions"
+output:
+  rmarkdown::html_vignette
+vignette: >
+  %\VignetteIndexEntry{Cheat sheet: Inputs/outputs for user-defined functions}
+  %\VignetteEngine{knitr::rmarkdown}
+  %\VignetteEncoding{UTF-8}
+---
+
+```{r include = FALSE}
+knitr::opts_chunk$set(
+  collapse = TRUE,
+  comment = "#>",
+  # disable R chunk evaluation when using macOS in GH Actions
+  # see https://github.com/tidyverse/ggplot2/issues/2252#issuecomment-1006713187
+  # and https://github.com/lcolladotor/biocthis/issues/27 and
+  # https://github.com/bodkan/slendr/commit/f0f5ae18452cc9df5d151874e41b0b8fd5f29aa2#
+  eval = Sys.getenv("RUNNER_OS") != "macOS"
+)
+
+library(simChef)
+
+set.seed(12345)
+```
+
+![Overview of inputs and outputs for user-defined functions when running a `simChef` `Experiment`. Dotted boxes denote optional input arguments that are automatically passed through in a `run_experiment()` call if specified.](figures/fit_eval_viz.png){width="100%" #fig:fit-eval-viz-figure}
+
+### DGP 
+
+- **Inputs**
+  - Any number of named parameters (e.g., `dgp_param1`, `dgp_param2`, ...)
+- **Outputs**
+  - A list of named elements to be passed onto the method(s) (e.g., `dgp_out1`, `dgp_out2`, ...)
+
+### Method
+
+- **Inputs**
+  - Named parameters matching all names in the list returned by the DGP function (e.g., `dgp_out1`, `dgp_out2`, ...)
+  - Any number of additional named parameters (e.g., `method_param1`, `method_param2`, ...)
+- **Outputs**
+  - A list of named elements (e.g., `method_out1`, `method_out2`, ...)
+
+### Evaluator
+
+- **Inputs**
+  - `fit_results` (optional): output of `fit_experiment()`
+  - `vary_params` (optional): character vector of parameter names that are varied across in the experiment
+  - Any number of additional named parameters (e.g., `eval_param1`, `eval_param2`, ...)
+- **Outputs**
+  - Typically a `tibble` or `data.frame`
+
+### Visualizer
+
+- **Inputs**
+  - `fit_results` (optional): output of `fit_experiment()`
+  - `eval_results` (optional): output of `evaluate_experiment()`
+  - `vary_params` (optional): character vector of parameter names that are varied across in the experiment
+  - Any number of additional named parameters (e.g., `viz_param1`, `viz_param2`, ...)
+- **Outputs**
+  - Typically a plot (e.g., a `ggplot` or `plotly` object). Can also return tables or more generally, any R Markdown snippet (e.g., verbatim text)
+
+### Outputs of `*_experiment`
+
+- `fit_results`: output of `fit_experiment()`
+  - A tibble of results from each (replicate, DGP, Method) combination with the following columns:
+    - `.rep`: replicate ID
+    - `.dgp_name`: name of DGP
+    - `.method_name`: name of Method
+    - `vary_across` parameter columns (if applicable): value of the specified `vary_across` parameter
+    - A column corresponding to each named component in the list returned by `Method` (e.g., `method_out1`, `method_out2`, ...)
+- `eval_results`: output of `evaluate_experiment()`
+  - A named list of elements, with one element for each `Evaluator` in the `Experiment`
+    - The list names correspond to the names given to each `Evaluator` in `create_evaluator(.name = ...)`
+    - The list elements are exactly the return values from each `Evaluator` function
+- `viz_results`: output of `visualize_experiment()`
+  - A named list of elements, with one element for each `Visualizer` in the `Experiment`
+    - The list names correspond to the names given to each `Visualizer` in `create_visualizer(.name = ...)`
+    - The list elements are exactly the return values from each `Visualizer` function

vignettes/example-docs/Example Experiment/docs/dgps/Linear DGP.md

@@ -0,0 +1,23 @@
+We simulate the (test and training) covariate/design matrix $\mathbf{X} \in \mathbb{R}^{n \times p}$ from a standard normal distribution and the response vector $\mathbf{y} \in \mathbb{R}^n$ from a linear model. Specifically,
+
+\begin{align*}
+\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon},\\
+\end{align*}
+
+where
+\begin{align*}
+& \mathbf{X}_{ij} \stackrel{iid}{\sim} N\left(0, 1\right) \text{ for all } i = 1, \ldots, n \text{ and } j = 1, \ldots, p, \\
+& \boldsymbol{\epsilon}_i \stackrel{iid}{\sim} N(0, \sigma^2) \text{ for all } i = 1, \ldots, n.
+\end{align*}
+
+**Default Parameters in DGP**
+
+- Number of training samples: $n_{\text{train}} = 200$
+- Number of test samples: $n_{\text{test}} = 200$
+- Number of features: $p = 2$
+- Amount of noise: $\sigma = 1$
+- Coefficients: $\boldsymbol{\beta} = (1, 0)^\top$
+
+<span style="color: blue">
+	[In practice, documentation of DGPs should answer the questions “what” and “why”. That is, “what” is the DGP, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]
+</span>

vignettes/example-docs/Example Experiment/docs/evaluators/Prediction Accuracy.md

@@ -0,0 +1,11 @@
+Given the true responses $\mathbf{y} \in \mathbb{R}^n$ and predicted responses $\mathbf{\hat{y}} \in \mathbb{R}^n$ from various methods, we evaluate several prediction accuracy metrics, namely:
+
+- Root Mean Squared Error (RMSE): $\sqrt{\frac{1}{n} || \mathbf{y} - \mathbf{\hat{y}} ||_2^2}$
+- Mean Absolute Error (MAE): $\frac{1}{n} || \mathbf{y} - \mathbf{\hat{y}} ||_1$
+- R-squared ($R^2$): $1 - \frac{|| \mathbf{y} - \mathbf{\hat{y}} ||_2^2}{|| \mathbf{y} - \mathbf{\bar{y}} ||_2^2}$
+
+We choose to evaluate both RMSE and MAE as these can convey different messages in the presence of outliers. Further, $R^2$ provides a convenient normalization of RMSE that can often be more easily interpreted.
+
+<span style="color: blue">
+	[In practice, documentation of evaluation metrics should answer the questions “what” and “why”. That is, “what” is the metric, and “why” are we using/studying it?]
+</span>

vignettes/example-docs/Example Experiment/docs/methods/Linear Regression.md

@@ -0,0 +1,15 @@
+Given some training covariate data $\mathbf{X}$ and response vector $\mathbf{y}$, we fit a linear regression model (i.e., ordinary least squares) on $\mathbf{X}$ to predict $\mathbf{y}$ by minimizing the following objective:
+
+\begin{align*}
+\boldsymbol{\hat{\beta}} = \text{argmin}_{\boldsymbol{\beta}} || \mathbf{y} - \mathbf{X} \boldsymbol{\beta} ||_2^2.
+\end{align*}
+
+Then, to make prediction given some test data $\mathbf{X}^{\text{test}}$, we compute:
+
+\begin{align*}
+\boldsymbol{\hat{y}}^{\text{test}} = \mathbf{X}^{\text{test}} \boldsymbol{\hat{\beta}}.
+\end{align*}
+
+<span style="color: blue">
+	[In practice, documentation of methods should answer the questions “what” and “why”. That is, “what” is the method, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]
+</span>

vignettes/example-docs/Example Experiment/docs/methods/Random Forest.md

@@ -0,0 +1,9 @@
+Given some training covariate data $\mathbf{X}$ and response vector $\mathbf{y}$, we fit a random forest on $\mathbf{X}$ to predict $\mathbf{y}$. At a high-level, a random forest is an ensemble of classification or regression trees (CART) that are fitted independently of one another on bootstrapped samples of the training data. Further, each CART model is fitted by performing recursive axis-aligned binary splits, where the optimal split at each node is chosen from a random subsample of features to minimize an impurity decrease criterion. 
+
+To make predictions, CART identifies the unique leaf node containing the test data point and predicts the mean response (of training data) in that node. For a random forest, the predictions are averaged across all CARTs in the forest.
+
+For further details, we refer to [Breiman (2001)](https://link.springer.com/content/pdf/10.1023/A:1010933404324.pdf).
+
+<span style="color: blue">
+	[In practice, documentation of methods should answer the questions “what” and “why”. That is, “what” is the method, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]
+</span>

vignettes/example-docs/Example Experiment/docs/objectives.md

@@ -0,0 +1,5 @@
+The objective of this simulation experiment is to provide a basic example on how to use `simChef` and showcase the automated R Markdown-generated documentation. For the sake of illustration, this toy simulation experiment studies the performance of linear regression versus random forests under a linear data-generating process across varying noise levels.
+
+<span style="color: blue">
+	[Typically, the objective of the simulation experiment (and this blurb) will be more scientific than instructive and will warrant additional context/background and domain knowledge.]
+</span>

vignettes/example-docs/Example Experiment/docs/visualizers/Prediction Accuracy Plot.md

@@ -0,0 +1,5 @@
+We plot the prediction accuracy between the true and predicted responses, as measured via RMSE, MAE, and $R^2$, to understand how characteristics of the DGP affect various prediction methods.
+
+<span style="color: blue">
+	[In practice, documentation of the plotters should answer the questions “what” and “why”. That is, “what” is the plot, and “why” are we using/studying it?]
+</span>

vignettes/docs/dgps/Linear Gaussian DGP.md → vignettes/example-docs/Linear Regression Experiment/docs/dgps/Linear Gaussian DGP.md

@@ -16,4 +16,4 @@ In the Linear Gaussian DGP, we simulate the feature/design matrix $\mathbf{X} \i
 
 <span style="color: blue">
 	[In practice, documentation of DGPs should answer the questions “what” and “why”. That is, “what” is the DGP, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]
-</span>
+</span>

vignettes/docs/methods/OLS.md → vignettes/example-docs/Linear Regression Experiment/docs/methods/OLS.md

@@ -16,4 +16,4 @@ and then compute the two-sided p-value under the t distribution with $n - p - 1$
 
 <span style="color: blue">
 	[In practice, documentation of methods should answer the questions “what” and “why”. That is, “what” is the method, and “why” are we using/studying it? As this simulation experiment is a contrived example, we omit the “why” here.]
-</span>
+</span>

vignettes/docs/objectives.md → vignettes/example-docs/Linear Regression Experiment/docs/objectives.md

@@ -2,4 +2,4 @@ The objective of this simulation experiment is to provide a toy example on how t
 
 <span style="color: blue">
 	[Typically, the objective of the simulation experiment (and this blurb) will be more scientific than instructive and will warrant additional context/background and domain knowledge.]
-</span>
+</span>

vignettes/references.bib

@@ -0,0 +1,79 @@
+
+@article{yu-veridical-2020,
+	title = {Veridical data science},
+	volume = {117},
+	issn = {0027-8424, 1091-6490},
+	url = {http://www.pnas.org/lookup/doi/10.1073/pnas.1901326117},
+	doi = {10.1073/pnas.1901326117},
+	abstract = {Building and expanding on principles of statistics, machine learning, and scientific inquiry, we propose the predictability, computability, and stability (PCS) framework for veridical data science. Our framework, composed of both a workflow and documentation, aims to provide responsible, reliable, reproducible, and transparent results across the data science life cycle. The PCS workflow uses predictability as a reality check and considers the importance of computation in data collection/storage and algorithm design. It augments predictability and computability with an overarching stability principle. Stability expands on statistical uncertainty considerations to assess how human judgment calls impact data results through data and model/algorithm perturbations. As part of the PCS workflow, we develop PCS inference procedures, namely PCS perturbation intervals and PCS hypothesis testing, to investigate the stability of data results relative to problem formulation, data cleaning, modeling decisions, and interpretations. We illustrate PCS inference through neuroscience and genomics projects of our own and others. Moreover, we demonstrate its favorable performance over existing methods in terms of receiver operating characteristic (ROC) curves in high-dimensional, sparse linear model simulations, including a wide range of misspecified models. Finally, we propose PCS documentation based on R Markdown or Jupyter Notebook, with publicly available, reproducible codes and narratives to back up human choices made throughout an analysis. The PCS workflow and documentation are demonstrated in a genomics case study available on Zenodo.},
+	language = {en},
+	number = {8},
+	urldate = {2021-09-04},
+	journal = {Proceedings of the National Academy of Sciences},
+	author = {Yu, Bin and Kumbier, Karl},
+	month = feb,
+	year = {2020},
+	pages = {3920--3929},
+	file = {Yu and Kumbier - 2020 - Veridical data science.pdf:/home/james/Zotero/storage/2IMU9L5Q/Yu and Kumbier - 2020 - Veridical data science.pdf:application/pdf},
+}
+
+@article{lang-batchtools-2017,
+	title = {batchtools: {Tools} for {R} to work on batch systems},
+	volume = {2},
+	issn = {2475-9066},
+	shorttitle = {batchtools},
+	url = {https://joss.theoj.org/papers/10.21105/joss.00135},
+	doi = {10.21105/joss.00135},
+	abstract = {Lang et al, (2017), batchtools: Tools for R to work on batch systems, Journal of Open Source Software, 2(10), 135, doi:10.21105/joss.00135},
+	language = {en},
+	number = {10},
+	urldate = {2023-04-20},
+	journal = {Journal of Open Source Software},
+	author = {Lang, Michel and Bischl, Bernd and Surmann, Dirk},
+	month = feb,
+	year = {2017},
+	pages = {135},
+	file = {Full Text PDF:/home/james/Zotero/storage/PPA2UTP7/Lang et al. - 2017 - batchtools Tools for R to work on batch systems.pdf:application/pdf},
+}
+
+@article{wickham-welcome-2019,
+	title = {Welcome to the {Tidyverse}},
+	volume = {4},
+	issn = {2475-9066},
+	url = {https://joss.theoj.org/papers/10.21105/joss.01686},
+	doi = {10.21105/joss.01686},
+	abstract = {Wickham et al., (2019). Welcome to the Tidyverse. Journal of Open Source Software, 4(43), 1686, https://doi.org/10.21105/joss.01686},
+	language = {en},
+	number = {43},
+	urldate = {2023-04-20},
+	journal = {Journal of Open Source Software},
+	author = {Wickham, Hadley and Averick, Mara and Bryan, Jennifer and Chang, Winston and McGowan, Lucy D'Agostino and François, Romain and Grolemund, Garrett and Hayes, Alex and Henry, Lionel and Hester, Jim and Kuhn, Max and Pedersen, Thomas Lin and Miller, Evan and Bache, Stephan Milton and Müller, Kirill and Ooms, Jeroen and Robinson, David and Seidel, Dana Paige and Spinu, Vitalie and Takahashi, Kohske and Vaughan, Davis and Wilke, Claus and Woo, Kara and Yutani, Hiroaki},
+	month = nov,
+	year = {2019},
+	pages = {1686},
+	file = {Full Text PDF:/home/james/Zotero/storage/RKZ8IGRN/Wickham et al. - 2019 - Welcome to the Tidyverse.pdf:application/pdf},
+}
+
+@article{bengtsson-unifying-2021,
+	title = {A {Unifying} {Framework} for {Parallel} and {Distributed} {Processing} in {R} using {Futures}},
+	volume = {13},
+	issn = {2073-4859},
+	url = {https://journal.r-project.org/archive/2021/RJ-2021-048/index.html},
+	doi = {10.32614/RJ-2021-048},
+	abstract = {A future is a programming construct designed for concurrent and asynchronous evaluation of code, making it particularly useful for parallel processing. The future package implements the Future API for programming with futures in R. This minimal API provides sufficient constructs for implementing parallel versions of well-established, high-level map-reduce APIs. The future ecosystem supports exception handling, output and condition relaying, parallel random number generation, and automatic identification of globals lowering the threshold to parallelize code. The Future API bridges parallel frontends with parallel backends, following the philosophy that end-users are the ones who choose the parallel backend while the developer focuses on what to parallelize. A variety of backends exist, and third-party contributions meeting the specifications, which ensure that the same code works on all backends, are automatically supported. The future framework solves several problems not addressed by other parallel frameworks in R.},
+	language = {en},
+	number = {2},
+	urldate = {2023-04-20},
+	journal = {The R Journal},
+	author = {Bengtsson, Henrik},
+	year = {2021},
+	pages = {208},
+	file = {Bengtsson - 2021 - A Unifying Framework for Parallel and Distributed .pdf:/home/james/Zotero/storage/KEV9JT2I/Bengtsson - 2021 - A Unifying Framework for Parallel and Distributed .pdf:application/pdf},
+}
+
+@Manual{chang-r6-2022,
+  title = {R6: Encapsulated Classes with Reference Semantics},
+  author = {Winston Chang},
+  year = {2022},
+  note = {https://r6.r-lib.org, https://github.com/r-lib/R6/},
+}