With the steady steam of new pipelines we intend to support, there is a need for a standardised way of implementing them. The current way is based on how the first pipelines were implemented. This way results in a lot of work, a lot of code duplication and ever growing complexity.

The pipeline implementation can be divided into:

Ordering & and meta-data persisting
Starting pipelines
Persisting results
Quality control
Delivery to customers