intro.qmd

---
title: "Introduction"
---

The core of The Carpentries Workbench consists of three packages: 

 - [{sandpaper}]\: user interface and workflow engine
 - [{pegboard}]\: parsing and validation engine
 - [{varnish}]\: HTML templates, CSS, and JS elements

These packages are all available and released to the [Carpentries 
R-Universe][r-universe], which checks for updates to the source packages hourly.

## Building Lessons {#sec-local}

In a broad sense, this is what happens when you run `sandpaper::serve()` or
`sandpaper::build_lesson()`. The interaction between the three Workbench
packages, the lesson content, and the author can be summarised like this where
the author makes an edit:

::: {.callout-note}

### Summary Content

This content is a general picture of what happens between the packages. For a
more in-depth discussion and more detailed diagrams, please visit the [Flow 
Diagrams page](flow.html).

:::


```{mermaid}
sequenceDiagram
    autonumber
    actor Author
    participant Lesson 
    box rgb(255, 214, 216) The Workbench
    participant {sandpaper}
    participant {pegboard}
    participant {varnish}
    end

    Author ->> {sandpaper}: sandpaper::serve()
    activate Author
    {sandpaper} --) Author: website preview
    note left of {sandpaper}: monitor for changes
    Author ->> Lesson: make an edit
    deactivate Author
    Lesson -->> {sandpaper}: READ changed file(s)
    {sandpaper} -->> {pegboard}: validate Lesson
    activate {pegboard}
    note left of {sandpaper}: provision global menu elements
    {pegboard} --) Author: report accessibility 
    deactivate {pegboard}
    activate {sandpaper}
    note left of {sandpaper}: WRITE markdown
    {varnish} -->> {sandpaper}: load and apply website template
    note left of {sandpaper}: WRITE website
    {sandpaper} --) Author: website preview
    deactivate {sandpaper}
```

In terms of folder structure, the workflow runs the two-step workflow to first
render markdown files into `site/built` and then uses those files to render the
HTML, CSS, and JavaScript into `site/built`. These workflows are detailed in
[The Workflows Chapter](flow.html).


```{mermaid}
flowchart TB
    classDef default color:#383838,fill:#FFF7F1,stroke-width:1px
    classDef external color:#383838,fill:#E6EEF8,stroke-width:1px
    classDef normal color:#081457,fill:#E3E6FC,stroke-width:1px
    classDef local fill:#FFC700,stroke:#333,stroke-width:1px
    classDef remote fill:#D2BDF2, color:#201434,stroke-width:1px

    SERVE("serve()"):::normal
    BLESS("build_lesson()"):::normal

    subgraph "Core Workflow"
    BUILT["site/built"]:::local
    SITE["site/docs"]:::local
    VLESS("validate_lesson()"):::normal
    BUILDMD(["build_markdown()"]):::normal
    BUILDSITE(["build_site()"]):::normal
    end

    %%BUILT ~~~ SITE

    SERVE --> BLESS
    %% SERVE ~~~ VLESS
    %% SERVE ~~~ BUILDMD
    BLESS --> VLESS
    VLESS -.- BUILDMD
    BLESS --> BUILDMD
    BUILDMD --> BUILT
    BUILT -.- BUILDSITE
    VLESS -.- BUILDSITE
    BLESS --> BUILDSITE
    BUILDSITE --> SITE
```

::: {.callout-note}

### Resource folder names

The names of the folders inside `site/` are considered internal resources and
they can change at any time. The reason why the folder for the final website
output is called `site/docs/` is because we use the
[{pkgdown}](https://pkgdown.github.io) package to provision the website without
needing to bundle the templates inside of {sandpaper}, but we never got around
to explicitly changing the name of that folder.

:::

The `site/docs` folder contains the full website that can be safely used
offline. This is the core of the workflow and is used both locally and in a
remote setting. The only difference with the remote setting is that we use a few
Git tricks to provision the markdown cache without needing to store it in the
default branch.

## Building Lessons Remotely (e.g. on GitHub) {#sec-remote}

In the remote workflow, **we still use the same workflow as above**, except now
we use `ci_deploy()` to link the branches and folders using worktrees, which
you can think of as Git branches assigned to separate folders.

::: {.callout-note}

### Platform Independence

When we developed The Workbench, GitHub was the most widely used platform for
social coding that represented the easiest way for newcomers to contribute to
our lessons. We used this knowledge to build the workflows for the lessons on
GitHub, but we were also aware of the valid criticisms of GitHub and the dangers
of vendor lock-in. 

Thus, while the lessons are deployed using GitHub workflows and we have features
that handle pull requests and updates, **the core deployment features remain
platform-independent**. The workflows are merely instructions that we provide
for GitHub to set up the workbench and to run the individual functions. In
theory, **any platform can be configured to deploy lessons via The Workbench.**

In fact, in a pinch when GitHub workflows are not working properly, a lesson
maintainer could run `sandpaper:::ci_deploy()` to render and deploy a local
copy of the lesson.

:::



```{mermaid}
flowchart TB
    classDef default color:#383838,fill:#FFF7F1,stroke-width:1px
    classDef external color:#383838,fill:#E6EEF8,stroke-width:1px
    classDef normal color:#081457,fill:#E3E6FC,stroke-width:1px
    classDef local fill:#FFC700,stroke:#333,stroke-width:1px
    classDef remote fill:#D2BDF2,stroke:#201434,stroke-width:1px
    classDef notouch fill:#F99697,stroke:#A4050E,stroke-width:1px


    GH[("@main")]:::remote
    MDOUT[("@md-outputs")]:::notouch
    PAGES[("@gh-pages")]:::notouch
    DEPLOY(["ci_deploy()"]):::external
    CIBUILDMD(["ci_build_markdown()"]):::external
    CIBUILDSITE(["ci_build_site()"]):::external

    subgraph virtual machine
    REPO["[repo]"]:::local
    BUILT["[repo]/site/built"]:::local
    SITE["[repo]/site/docs"]:::local
    VLESS("validate_lesson()"):::normal
    BUILDMD(["build_markdown()"]):::normal
    BUILDSITE(["build_site()"]):::normal
    end


    GH ---> REPO
    GH ~~~ DEPLOY
    REPO -.- VLESS


    DEPLOY ---> VLESS
    DEPLOY ---> CIBUILDMD
    DEPLOY ---> CIBUILDSITE
    VLESS -.- BUILDMD
    CIBUILDMD ---> MDOUT
    MDOUT <-.-> BUILT
    CIBUILDMD ---> BUILDMD
    CIBUILDSITE ---> PAGES
    PAGES <-.-> SITE
    CIBUILDSITE ---> BUILDSITE
    BUILT -.- BUILDSITE
    VLESS -.- BUILDSITE
    BUILDMD --> BUILT
    BUILDSITE --> SITE
```



## Development

Development of The Workbench is overseen by Zhian N. Kamvar. New features are
added incrementally as pull requests. Pushes to the main branch are _rare_ and
discouraged. New features must have tests associated (with the exception of
{varnish}).

If you are interested, we have [documentation for the release
process](releases.html) available.



## Documentation

Reference documentation for individual functions for each package is written
alongside the function using [{roxygen2}](https://roxygen2.r-lib.org/).

This documentation is generated by `devtools::document()`

## Testing

Tests for each package live in `tests/testthat/` and follow a
`test-[file-name].R` naming convention. These are controlled by the
[{testthat}](https://testthat.r-lib.org/) package and run by `devtools::test()`.

You can find more information about testing the core packages in [Testing The
Workbench](testing.html)

## Continous Integration

The continous integration for each package tests on Ubuntu, MacOS, and Windows
systems with the last five versions of R (same as the RStudio convention). 

More information about the Continous Integration can be found in the [Continuous
Integration section](testing.html#ci) of the testing section.

--------

Coming up:

 - Testing Pull Requests (Locally and on your fork)
 - Resources for R package development
 - Adding functionality to {sandpaper}
 - Adding functionality to {pegboard}
 - Adding styling elements to {varnish}
 - Adding functionality to carpentries/actions

[{varnish}]: https://carpentries.github.io/varnish/
[{pegboard}]: https://carpentries.github.io/pegboard/
[{sandpaper}]: https://carpentries.github.io/sandpaper/
[r-universe]: https://carpentries.r-universe.dev/