mrc-ide · weshinsley · Jan 4, 2024 · Dec 22, 2023 · Jan 3, 2024 · Jan 3, 2024
diff --git a/DESCRIPTION b/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: hipercow
 Title: High Performance Computing
-Version: 0.2.3
+Version: 0.2.4
 Authors@R: c(person("Rich", "FitzJohn", role = c("aut", "cre"),
                     email = "rich.fitzjohn@gmail.com"),
              person("Wes", "Hinsley", role = "aut"),
@@ -23,10 +23,14 @@ Imports:
 Suggests:
     callr,
     conan2,
+    knitr,
     logwatch,
     mockery,
+    rmarkdown,
     testthat (>= 3.0.0)
 Config/testthat/edition: 3
 Remotes:
     mrc-ide/conan2,
     reside-ic/logwatch
+VignetteBuilder: knitr
+Language: en-GB
diff --git a/Makefile b/Makefile
@@ -31,4 +31,9 @@ check_all:
 clean:
 	rm -f src/*.o src/*.so src/*.gcda src/*.gcno src/*.gcov
 
-.PHONY: clean all test document install
+vignettes/%.Rmd: vignettes_src/%.Rmd
+        Rscript -e "knitr::knit('$<', '$@')"
+
+vignettes: vignettes/hello.Rmd vignettes/hipercow.Rmd vignettes/packages.Rmd vignettes/troubleshooting.Rmd
+
+.PHONY: clean all test document install vignettes
diff --git a/R/provision.R b/R/provision.R
@@ -4,7 +4,8 @@
 ##' @title Provision cluster library
 ##'
 ##' @param method The provisioning method to use, defaulting to
-##'   `NULL`, which indicates autoprovision
+##'   `NULL`, which indicates we should try and detect the best
+##'   provisioning mechanism for you.
 ##'
 ##' @param ... Arguments passed through to conan. See docs that we
 ##'   need to write still.

diff --git a/R/task.R b/R/task.R
@@ -172,7 +172,10 @@ task_eval <- function(id, envir = .GlobalEnv, verbose = FALSE, root = NULL) {
   root <- hipercow_root(root)
   t0 <- Sys.time()
   if (verbose) {
-    cli::cli_h1("hipercow running at '{root$path$root}'")
+    version <- utils::packageVersion('hipercow')
+    cli::cli_h1("hipercow {version} running at '{root$path$root}'")
+    cli::cli_alert_info("library paths:")
+    cli::cli_li(.libPaths())
     cli::cli_alert_info("id: {id}")
     cli::cli_alert_info("starting at: {t0}")
   }
@@ -258,7 +261,7 @@ task_eval <- function(id, envir = .GlobalEnv, verbose = FALSE, root = NULL) {
 ##' * `success`, `failure`, `cancelled`
 ##'
 ##' These occur in increasing order and the result of this function is
-##' the furtherst through this list.
+##' the furthest through this list.
 ##'
 ##' Later, we will introduce other types to cope with tasks that have
 ##' been retried, or that are blocked on dependencies (or have become
@@ -410,7 +413,7 @@ task_result <- function(id, root = NULL) {
 ##' Get the task log, if the task has produced one.  Tasks run by the
 ##' `windows` driver will generally produce a log.  A log might be
 ##' quite long, and you might want to print it to screen in its
-##' entirity (`task_log_show`), return it as character vector
+##' entirety (`task_log_show`), return it as character vector
 ##' (`task_log_value`).
 ##'
 ##' The function `task_log_watch` has similar semantics to

diff --git a/README.md b/README.md
@@ -6,6 +6,26 @@
 [![codecov.io](https://codecov.io/github/mrc-ide/hipercow/coverage.svg?branch=main)](https://codecov.io/github/mrc-ide/hipercow?branch=main)
 <!-- badges: end -->
 
+**NOTICE**: This will only be of use to people at DIDE, as it uses our [cluster web portal](https://mrcdata.dide.ic.ac.uk/hpc), local cluster, and local network file systems.
+
+## What is this?
+
+This is a package for interfacing with the DIDE cluster directly from R.  It is meant make jobs running on the cluster appear as if they are running locally but asynchronously.  The idea is to let the cluster appear as an extension of your own computer so you can get using it within an R project easily.
+
+This package supersedes didehpc (2015-2023).
+
+## Documentation
+
+* New to this? [The main vignette](https://mrc-ide.github.io/hipercow/articles/hipercow.html) contains full instructions and explanations about why some bits are needed.
+* Need a reminder? There is a [very quick hello world example](https://mrc-ide.github.io/hipercow/articles/hello.html) which is much shorter and may be enough to remind you.
+* Trying to install packages on the cluster? Check the [packages vignette](https://mrc-ide.github.io/hipercow/articles/packages.html) for ways of controlling this.
+* Having problems? Check the [troubleshooting guide](https://mrc-ide.github.io/hipercow/articles/troubleshooting.html).
+
+## Issues
+
+* Check the [issue tracker](https://github.com/mrc-ide/hipercow/issues) for known problems, or to create a new one
+* Use the "Cluster" channel on Teams, which Rich and Wes keep an eye on
+
 ## Installation
 
 To install `hipercow`:

diff --git a/_pkgdown.yml b/_pkgdown.yml
@@ -0,0 +1,79 @@
+url: https://mrc-ide.github.io/hipercow
+
+template:
+  bootstrap: 5
+
+articles:
+  - title: General
+    navbar: General
+    contents:
+      - hipercow
+      - hello
+      - packages
+      - troubleshooting
+      - details
+
+  - title: Advanced topics
+    navbar: Advanced topics
+    contents:
+      - administration
+
+reference:
+  - title: Get started
+    desc: >-
+      Probably the order you'll work through when starting a new
+      hipercow project; create a new root, configure it, test that it
+      works, and provision any packages you need.
+    contents:
+      - hipercow_init
+      - hipercow_configure
+      - hipercow_configuration
+      - hipercow_hello
+      - hipercow_provision
+
+  - title: Tasks
+    desc: >-
+      Create and interact with tasks
+
+  - subtitle: Creation
+    contents:
+      - task_create_explicit
+      - task_create_expr
+      - task_create_script
+      - task_create_bulk_expr
+
+  - subtitle: Interaction
+    contents:
+      - task_status
+      - task_result
+      - task_wait
+      - task_cancel
+
+  - subtitle: Logs
+    contents:
+      - task_log_show
+      - task_log_value
+      - task_log_watch
+
+  - title: Environments
+    desc: >-
+      Environments are the collections of packages and functions that
+      we construct on a cluster machine and evaluate your tasks
+      within.  These functions let you create, update and delete these
+      environments.
+    contents:
+      - starts_with("hipercow_environment")
+
+  - title: Windows support
+    desc: >-
+      Functions for use with the windows cluster
+    contents:
+      - starts_with("windows_")
+
+  - title: Advanced
+    desc: >-
+      Just leave it alone.
+    contents:
+      - hipercow_driver
+      - task_eval
+      - task_submit
diff --git a/drivers/windows/DESCRIPTION b/drivers/windows/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: hipercow.windows
 Title: DIDE HPC Support for Windows
-Version: 0.2.3
+Version: 0.2.4
 Authors@R: c(person("Rich", "FitzJohn", role = c("aut", "cre"),
                     email = "rich.fitzjohn@gmail.com"),
              person("Wes", "Hinsley", role = "aut"),

diff --git a/inst/WORDLIST b/inst/WORDLIST
@@ -0,0 +1,33 @@
+DIDE
+HPC
+ICT
+R's
+RStudio
+Rmd
+Rtools
+ZScaler
+alloc
+chmod
+cli
+cli's
+codecov
+com
+conan
+config
+didehpc
+etc
+hipercow's
+io
+lifecycle
+lockfile
+macOS
+mountname
+mountpoint
+pkgdepends
+reestablished
+renv
+rfiglet
+src
+std
+traceback
+xkcd
diff --git a/man/hipercow_provision.Rd b/man/hipercow_provision.Rd
diff --git a/man/task_log.Rd b/man/task_log.Rd
diff --git a/man/task_status.Rd b/man/task_status.Rd
diff --git a/tests/testthat/test-task.R b/tests/testthat/test-task.R
@@ -287,7 +287,7 @@ test_that("can be verbose running a task", {
     task_create_explicit(quote(sqrt(a)), export = "a", envir = env1))
   res <- evaluate_promise(
     task_eval(id, envir = env2, verbose = TRUE, root = path))
-  expect_match(res$messages, "hipercow running at", all = FALSE)
+  expect_match(res$messages, "hipercow .+ running at", all = FALSE)
   expect_match(res$messages, "id: ", all = FALSE)
   expect_match(res$messages, "starting at: ", all = FALSE)
   expect_match(res$messages, "task type: explicit", all = FALSE)
@@ -309,7 +309,7 @@ test_that("can be verbose running a failing task", {
     task_create_explicit(quote(readRDS("nofile.rds"))))
   res <- evaluate_promise(
     task_eval(id, envir = env2, verbose = TRUE, root = path))
-  expect_match(res$messages, "hipercow running at", all = FALSE)
+  expect_match(res$messages, "hipercow .+ running at", all = FALSE)
   expect_match(res$messages, "id: ", all = FALSE)
   expect_match(res$messages, "starting at: ", all = FALSE)
   expect_match(res$messages, "task type: explicit", all = FALSE)

diff --git a/vignettes/.gitignore b/vignettes/.gitignore
@@ -0,0 +1,2 @@
+*.html
+*.R
diff --git a/vignettes/administration.Rmd b/vignettes/administration.Rmd
@@ -0,0 +1,93 @@
+---
+title: "Administration"
+output: rmarkdown::html_vignette
+vignette: >
+  %\VignetteIndexEntry{Administration}
+  %\VignetteEngine{knitr::rmarkdown}
+  %\VignetteEncoding{UTF-8}
+---
+
+Are you reading the right manual?  If you are not sure, you are probably not!
+
+# Rebuilding the bootstrap library
+
+Ensure that the packages have built and are on the r-universe; this might take a little while (it rebuilds once an hour). Check on the [builds page](https://mrc-ide.r-universe.dev/builds) to see if it has updated.
+
+Install your own copy of hipercow as usual (you'll need `hipercow.windows` and `conan2` but these will be installed on use if you don't have them already).
+
+```r
+install.packages(
+  "hipercow",
+  repos = c("https://mrc-ide.r-universe.dev", "https://cloud.r-project.org"))
+```
+
+Set your working directory to anywhere on a network share (home drives are fine for this).
+
+Trigger building the bootstrap with:
+
+```r
+library(hipercow)
+hipercow_init(".")
+
+for (version in c("4.2.3", "4.3.0")) {
+  hipercow_configure("windows", r_version = version)
+  hipercow.windows:::bootstrap_update()
+}
+```
+
+# Testing a copy of hipercow on the cluster
+
+If you want to test a copy of hipercow on the cluster, you need to install a specific version of it somewhere it will be picked up.  The simplest way of doing this is to install everything into a new bootstrap environment.
+
+Create a new development bootstrap library by running (as above, from a network location):
+
+```r
+hipercow::hipercow_init(".")
+hipercow::hipercow_configure("windows", r_version = "4.3.0")
+hipercow.windows:::bootstrap_update(development = "mrc-4827")
+```
+
+Now you can use your new version, after setting the option `hipercow.development = TRUE`:
+
+```
+library(hipercow)
+hipercow_init(".", "windows", r_version = "4.3.0") # needs to match above
+options(hipercow.development = TRUE)
+id <- task_create_expr(sessionInfo())
+task_status(id)
+task_log_show(id)
+task_status(id)
+task_result(id)
+```
+
+It is assumed that you have the development version installed locally yourself, which is likely the case.
+
+# Recreating the vignettes
+
+These need to be run in a networks share so set an environment variable like
+
+```
+HIPERCOW_VIGNETTE_ROOT=/path/to/share
+```
+
+in your `.Renviron` indicating where we should work.  We'll make lots of directories here.  I'm still working on a nice way of unlocking the password automatically for non-windows use though, but things work fine within an interactive session at least.
+
+Each vignette can be built by running (ideally in a fresh session with the working directory as the package root)
+
+```
+keyring::keyring_unlock()
+knitr::knit("vignettes_src/hello.Rmd", "vignettes/hello.Rmd")
+knitr::knit("vignettes_src/hipercow.Rmd", "vignettes/hipercow.Rmd")
+knitr::knit("vignettes_src/packages.Rmd", "vignettes/packages.Rmd")
+knitr::knit("vignettes_src/troubleshooting.Rmd", "vignettes/troubleshooting.Rmd")
+```
+
+which generates a dummy Rmd file within vignettes/ for each version of the real vignette in `vignettes_src/`
+
+You may want to run `options(hipercow.development = TRUE)` to build using the development versions (see above).
+
+On the command line, `make vignettes` will run through all vignettes, but this seems to only work on some versions of make.
+
+# Installing Rtools
+
+(TODO)