Skip to content

Commit

Permalink
feat: add schedule cli (#25)
Browse files Browse the repository at this point in the history 
* fix(cli/pipelines.py): adding better error handling when deploying pipeline

* feat(cli/pipelines.py): adding functions to list and count schedules

* feat(cli/pipelines.py): adding schedule option to pipelines commands

* style(pipelines.py): precommit changes

* docs(cli.md): adding documentation for pipelines commands

* feat(cli/pipelines.py): adding remove function for schedules
  • Loading branch information
@odarotto
odarotto authored Mar 25, 2024
1 parent 272da62 commit 56dcd6e
Show file tree
Hide file tree
Showing 5 changed files with 475 additions and 54 deletions.
59 changes: 59 additions & 0 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# syntax=docker/dockerfile:1

# `python-base` sets up all our shared environment variables
FROM python:3.10-slim as base

# Setup Environment Variables
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PIP_NO_CACHE_DIR=true \
PIP_DISABLE_PIP_VERSION_CHECK=on \
PIP_DEFAULT_TIMEOUT=100 \
POETRY_VERSION=1.2.2 \
POETRY_HOME="/opt/poetry" \
POETRY_VIRTUALENVS_IN_PROJECT=true \
POETRY_NO_INTERACTION=1 \
PYSETUP_PATH="/opt/pysetup" \
VENV_PATH="/opt/pysetup/.venv" \
DEBIAN_FRONTEND=noninteractive \
OPENMP_ENABLED=1

# prepend poetry and venv to path
ENV PATH="$POETRY_HOME/bin:$VENV_PATH/bin:$PATH"

# Setup Shell for the Docker Image
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

# Install system dependencies
RUN set -ex \
&& apt-get update -yqq \
&& apt-get install --no-install-recommends -yqq \
curl \
ssh \
gnupg \
lsb-release \
# Cleanup
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& rm -rf /tmp/* \
&& rm -rf /var/tmp/* \
&& rm -rf /usr/share/man \
&& rm -rf /usr/share/doc \
&& rm -rf /usr/share/doc-base \
# Setup SSH
&& mkdir -p ~/.ssh \
&& touch ~/.ssh/known_hosts \
&& chmod 0600 ~/.ssh/known_hosts ~/.ssh \
&& ssh-keyscan github.com >> ~/.ssh/known_hosts \
# Install Poetry
&& pip install "poetry==$POETRY_VERSION"

CMD ["/bin/bash"]

FROM base as production_pipelines
COPY . $PYSETUP_PATH
WORKDIR $PYSETUP_PATH
RUN poetry install --without=dev
# Run the project
CMD ["workflow", "workspace", "set", "development"]
182 changes: 182 additions & 0 deletions docs/cli.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,185 @@ workflow run [PIPELINE]

!!! Note
If the `work.function` is a Click CLI command, then `workflow run` will inherit the CLI default arguments and then merge them with the `work.parameters` specified in the work object. This allows for a single CLI to be used for both interactive and non-interactive workflows.

# Pipelines CLI

## Overview

This CLI tool provides a command-line interface for interacting with the Pipelines server, enabling users to manage workflow pipelines efficiently. It supports various operations, such as deploying, listing, counting, and managing the lifecycle of pipeline configurations and schedules.

## Usage

The CLI tool offers the following commands for interacting with the workflow pipelines:

### Get server info

Get the current version of the pipelines server and info about configuration.

#### Command: `workflow pipelines version`

Example output:

```json
{
'client': {
'baseurl': 'http://localhost:8001',
'timeout': 15.0,
'token': None
},
'server': {
'version': '2.6.1'
}
}
```

### List pipelines

List all pipelines or scheduled pipelines.

#### Command: `workflow pipelines ls [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`

Example output:

```shell
Workflow Pipelines
┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┓
┃ ID ┃ Name ┃ Status ┃ Stage ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━┩
│ 65fc4dae5ffde0e8dbeebc61 │ example │ created │ 1 │
│ 65fc4eda5ffde0e8dbeebc65 │ example │ created │ 1 │
│ 65fc50065ffde0e8dbeebc69 │ example │ created │ 1 │
└──────────────────────────┴─────────┴─────────┴───────┘
```

### Count pipelines

Count pipelines configurations per collection.

#### Command: `workflow pipelines count [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`

Example output:

```shell
Workflow Pipelines
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓
┃ Name ┃ Count ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩
│ example │ 3 │
├───────────────────────────┼────────────────────┤
│ Total │ 3 │
└───────────────────────────┴────────────────────┘
```

### Deploy pipeline configurations

Deploy a workflow pipeline or schedule from a YAML file.

#### Command: `workflow pipelines deploy <FILENAME> [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`

Example output:

```shell
Workflow Pipelines
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ IDs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 65fc7450157ed72595b91a2c │
└────────────────────────────────────────────────┘
```

### Pipeline details

Get the whole payload for a pipeline configuration or schedule.

#### Command: `workflow pipelines ps <PIPELINE> <ID> [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`

Example output:

```shell
Workflow Pipelines
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Pipeline: example ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ { │
│ "id": "65fc69cf5ffde0e8dbeebcc1", │
│ "name": "example", │
│ "current_stage": 1, │
│ "status": "created", │
│ "version": "1", │
│ "creation": 1711040975.1336632, │
│ "start": null, │
│ "stop": null, │
│ "pipeline": { │
│ "runs_on": null, │
│ "services": [], │
│ "steps": [ │
│ { │
│ "name": "daily_monitoring_task", │
│ "work_id": null, │
│ "runs_on": null, │
│ "services": [], │
│ "replicate_deployments": false, │
│ "work": { │
│ "user": "test", │
│ "site": "local", │
│ "function": "guidelines.example.alpha", │
│ "parameters": { │
│ "mu0": "${{ matrix.mu0 }}", │
│ "alpha": "${{ matrix.alpha }}", │
│ "sigma0": 22.0 │
│ }, │
│ "pipeline": "daily-monitoring-task" │
│ }, │
│ "status": "created", │
│ "stage": 1, │
│ "if_condition": "", │
│ "reason": null, │
│ "matrix": null, │
│ "evaluate_on_runtime": false, │
│ "success_threshold": 1.0 │
│ } │
│ ] │
│ }, │
│ "deployments": null, │
│ "user": "test" │
│ } │
└───────────────────────────────────────────────────┘
```
### Stop a pipeline management
Kill a running pipeline.
#### Command: `workflow pipelines stop <PIPELINE> <ID> [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`
Example output:
```shell
Workflow Pipelines
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Stopped IDs ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ 65fc69cf5ffde0e8dbeebcc1 │
└────────────────────────────────────────────────┘
```
### Remove pipeline
Removes a pipeline, you can only use this command on pipelines with `status="cancelled"`
#### Command: `workflow pipelines rm <PIPELINE> <ID> [OPTIONS]`
#### Options:
* `--schedule: For interacting with the Schedule API.`
Example output:
Loading

0 comments on commit 56dcd6e

Please sign in to comment.