Skip to content

Design platform for pipeline development using a set of connected bash commands in a csv

License

Notifications You must be signed in to change notification settings

theoportlock/workforce

Repository files navigation

workforce

Documentation Status

Build and run a pipeline of bash commands with python multiprocessing according to a tsv file edgelist.

Installation

Installation can be done with:

pip install workforce

Building a workforce workflow

To launch the pipeline builder, run:

workforce

To open a previously constructed pipeline, run:

workforce <PIPELINE.tsv>

Running workforce

To run a sample plan from workforce github project:

workforce -r example_plan.tsv

To run individual process(es) from the builder, select the process(es) in the order that you wish them to be excecuted and click the 'Run' button. The command line from where the builder was launched will display the standard output and error for each process.

Deleting processes from the project can be done by selecting a process and clicking the 'Delete' button

The simplest way to edit a process is to click the edge that the process is connected to. This will fill the 'Input' box with the connection details. You can then add a new node with those connections and delete the old process.

To import and use in a python shell, use the following command:

from workforce.workforce import worker
worker("<PLAN.CSV>")

The schema should be in the format of a tsv with two columns. On the left and right column is the source and target process respectively (see example).

This is tested on mac and linux but requires work for windows integration except for wsl2

About

Design platform for pipeline development using a set of connected bash commands in a csv

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages