Skip to content

heikkil/app-pipeline-simple

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

app-pipeline-simple

Simple workflow manager written in Perl.

Features

  • Control by a command line program
  • All intermediate steps are kept
  • Run, stop, restart, rerun at will
  • Detailed logging
  • No custom language
  • YAML workflow descriptions
  • Branching workflows
  • Visualization as graphs

Philosophy

Workflow management in computational (biological) sciences is a hard problem. This Perl module is based on assumption that UNIX pipe and redirect system is closest to optimal solution with these improvements:

Enforce the storing of all intermediate steps in a file

This is for clarity, accountability and to enable arbitrarily big data sets. Pipeline can contain independent steps that remove intermediate files if so required.

Naming of each step

This is to make it possible to stop, restart, and restart at any intermediate step after adjusting pipeline parameters.

Detailed logging

To keep track of all runs of the pipeline.

A pipeline is a collection of steps that are functionally equivalent to a pipeline. In other words, execution of a pipeline equals to execution of a each ordered step within the pipeline. From that derives that the pipeline object model needs only one class that can recursively represent the whole pipeline as well as individual steps.

Installation

The working version should be installed from CPAN:

$ cpanm spipe

or

$ cpanm App::Pipeline::Simple

Getting started

Read about runtime and configuration options from the spipe online documentation:

$ spipe -h

Development

Pipeline::Simple source repository is managed with Dist::Zilla and hosted on GitHub.

About

Simple workflow manager

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages