This repository contains a minimal example of a data processing pipeline which
has been automated in two ways, first using GNU Make and second using
makepipe. In the former case, the
heavy lifting is being done by the Makefile; in the latter
case, by pipeline.R (or alternatively pipeline_alt.R).
The pipeline, which I have lifted directly from Jenny Bryan's STAT545 course, does four things. It:
- Obtains a large file of English words.
- Calculates a histogram of word lengths.
- Generates a figure of this histogram.
- Renders a R Markdown report in HTML.
A dependency graph for this pipeline is produced by the makepipe pipeline:
A plain text summary, saved to pipeline.md, is also produced.
The presentation/ subdirectory contains slides for a brief presentation on
pipeline automation tools delivered at the Social Research Centre in November
2021.