diff --git a/README.md b/README.md
index c6ffa21..f0dd8f0 100644
--- a/README.md
+++ b/README.md
@@ -10,17 +10,21 @@
- [argo-jupyter-scheduler](#argo-jupyter-scheduler)
- [Installation](#installation)
- [What is it?](#what-is-it)
+ - [Optional features](#optional-features)
+ - [Sending to Slack](#sending-to-slack)
- [A deeper dive](#a-deeper-dive)
- [`Job`](#job)
- [`Job Definition`](#job-definition)
- [Internals](#internals)
+ - [Output Files](#output-files)
+ - [Workflow Steps](#workflow-steps)
- [Additional thoughts](#additional-thoughts)
- [Known issues](#known-issues)
- [License](#license)
**Argo-Jupyter-Scheduler**
-Submit longing running notebooks to run without the need to keep your JupyterLab server running. And submit a notebook to run on a specified schedule.
+Submit long-running notebooks to run without the need to keep your JupyterLab server running. And submit a notebook to run on a specified schedule.
## Installation
@@ -30,11 +34,11 @@ pip install argo-jupyter-scheduler
## What is it?
-Argo-Jupyter-Scheduler is a plugin to the [Jupyter-Scheduler](https://jupyter-scheduler.readthedocs.io/en/latest/index.html) JupyterLab extension.
+Argo-Jupyter-Scheduler is a plugin to the [Jupyter-Scheduler](https://jupyter-scheduler.readthedocs.io/en/latest/index.html) JupyterLab extension.
What does that mean?
-This means this is an application that gets installed in the JupyterLab base image and runs as an extension in JupyterLab. Specifically, you will see this icon at the bottom of the JupyterLab Launcher tab:
+This means this is an application that gets installed in the JupyterLab base image and runs as an extension in JupyterLab. Specifically, you will see this icon at the bottom of the JupyterLab Launcher tab:
@@ -42,9 +46,9 @@ And this icon on the toolbar of your Jupyter Notebook:
-This also means, as a lab extension, this application is running within each user's separate JupyterLab server. The record of the notebooks you've submitted is specific to you and you only. There is no central Jupyter-Scheduler.
+This also means, as a lab extension, this application is running within each user's separate JupyterLab server. The record of the notebooks you've submitted is specific to you and you only. There is no central Jupyter-Scheduler.
-However, instead of using the base Jupyter-Scheduler, we are using **Argo-Jupyter-Scheduler**.
+However, instead of using the base Jupyter-Scheduler, we are using **Argo-Jupyter-Scheduler**.
Why?
@@ -52,6 +56,29 @@ If you want to run your Jupyter Notebook on a schedule, you need to be assured t
The solution is Argo-Jupyter-Scheduler: Jupyter-Scheduler front-end with an Argo-Workflows back-end.
+## Optional features
+
+### Sending to Slack
+
+Argo-Jupyter-Scheduler allows sending HTML output of an executed notebook to a
+Slack channel:
+
+- See the Slack API docs on how to create a bot token (starts with `xoxb`)
+- Invite your bot to a Slack channel which will be used for sending output
+- When scheduling a notebook (as described above):
+ - Select a conda environment that has `papermill` installed
+ - Add the following `Parameters`:
+ - name: `SLACK_TOKEN`, value: `xoxb-`
+ - name: `SLACK_CHANNEL`, value: `` (without leading `#`, like `scheduled-jobs`).
+
+Create job:
+
+
+
+Slack output:
+
+
+
## A deeper dive
In the Jupyter-Scheduler lab extension, you can create two things, a `Job` and a `Job Definition`.
@@ -75,7 +102,7 @@ We are also relying on the [Nebari Workflow Controller](https://github.com/nebar
A `Job-Definition` is simply a way to create to Jobs that run on a specified schedule.
-In Argo-Jupyter-Scheduler, `Job Definition` translate into a `Cron-Workflow` in Argo-Worflows. So when you create a `Job Definition`, you create a `Cron-Workflow` which in turn creates a `Workflow` to run when scheduled.
+In Argo-Jupyter-Scheduler, `Job Definition` translate into a `Cron-Workflow` in Argo-Workflows. So when you create a `Job Definition`, you create a `Cron-Workflow` which in turn creates a `Workflow` to run when scheduled.
A `Job` is to `Workflow` as `Job Definition` is to `Cron-Workflow`.
@@ -84,12 +111,70 @@ A `Job` is to `Workflow` as `Job Definition` is to `Cron-Workflow`.
Jupyter-Scheduler creates and uses a `scheduler.sqlite` database to manage and keep track of the Jobs and Job Definitions. If you can ensure this database is accessible and can be updated when the status of a job or a job definition change, then you can ensure the view the user sees from JupyterLab match is accurate.
-> By default this database is located at `~/.local/share/jupyter/scheduler.sqlite` but this is a trailet that can be modified. And since we have access to this database, we can update the database directly from the workflow itself.
-
-To acommplish this, the workflow runs in two steps. First the workflow runs the notebook, using `papermill` and the conda environment specified. And second, depending on the success of this notebook run, updates the database with this status.
-
-And when a job definition is created, a corresponding cron-workflow is created. To ensure the database is properly updated, the workflow that the cron-workflow creates has three steps. First, create a job record in the database with a status of `IN PROGRESS`. Second, run the notebook, again using `papermill` and the conda environment specified. And third, update the newly created job record with the status of the notebook run.
-
+> By default this database is located at `~/.local/share/jupyter/scheduler.sqlite` but this is a traitlet that can be modified. And since we have access to this database, we can update the database directly from the workflow itself.
+
+To accomplish this, the workflow runs in two steps. First the workflow runs the notebook, using `papermill` and the conda environment specified. And second, depending on the success of this notebook run, updates the database with this status.
+
+And when a job definition is created, a corresponding cron-workflow is created. To ensure the database is properly updated, the workflow that the cron-workflow creates has these three steps. First, create a job record in the database with a status of `IN PROGRESS`. Second, run the notebook, again using `papermill` and the conda environment specified. And third, update the newly created job record with the status of the notebook run.
+
+### Output Files
+
+In addition to `papermill`, which creates the output notebook, `jupyter
+nbconvert` is used to produce HTML output. To make these output files
+downloadable via the web UI, it's important they match the format that
+Jupyter-Scheduler expects, which is achieved by reusing `create_output_filename`
+within Argo-Jupyter-Scheduler when creating output files.
+
+The expected output filenames include timestamps that must match the start time
+of a job. For cron jobs this is tricky because the start time is set whenever
+`create-job-id` is run. All workflow steps are run in separate containers and
+the `create-job-id` container is run after the `papermill` step, which creates
+the output files.
+
+Also, the `papermill` container is defined differently because it needs to have
+access to filesystem mount points where the `papermill` and `jupyter` commands
+are located as well as access the environment variables. Due to this, the
+commands executed within the `papermill` container cannot be changed once it's
+been defined.
+
+This also means that the `papermill` container cannot have access to the job
+start time and hence cannot create filenames with the expected timestamps. To
+solve this problem, the `papermill` step always creates output files with the
+same default filenames and there is an additional `rename-files` step that runs
+after `create-job-id`, which makes sure the timestamps match the job start time.
+To pass the start time value between containers, the SQLite database is used.
+
+Finally, because `create-job-id` creates a new job every time it runs, this job
+will also have a new id. The job id is important since it's the same as the name
+of the staging directory where job output files are expected to be found by
+Jupyter-Scheduler. But the output files are created in the `papermill` step,
+which has the id of a job that defined the workflow originally when it was
+scheduled, not the current one created in `create-job-id`. To point to the
+proper location on disk, a symlink is created connecting staging job
+directories. This is also done in the `rename-files` step by looking up the job
+ids in the SQLite database.
+
+For non-cron jobs, there is no `create-job-id` step. The rest of the workflow
+steps are the same, but no database lookups are performed and no symlinks are
+created. This is not necessary because the start time is immediately available
+and the job id matches the job staging area.
+
+### Workflow Steps
+
+Here's the overview of the workflow steps:
+
+- `main` runs `papermill` and `jupyter nbconvert` to create output files
+- `create-job-id` creates a new job that can run without JupyterLab (only
+ for cron jobs)
+- `rename-files` updates timestamps of output files and adds symlinks between
+ job staging directories
+- `send-to-slack` sends HTML output to a Slack channel (only if `SLACK_TOKEN`
+ and `SLACK_CHANNEL` are provided via `Parameters` when scheduling a job)
+- `failure` or `success` sets status as "Failed" or "Completed" in the web UI.
+
+These steps are executed sequentially in separate containers. If a step fails,
+the `failure` step is called in the end. Otherwise, the `success` step is
+called.
## Additional Thoughts
diff --git a/argo_jupyter_scheduler/executor.py b/argo_jupyter_scheduler/executor.py
index 48db763..6e9e3b2 100644
--- a/argo_jupyter_scheduler/executor.py
+++ b/argo_jupyter_scheduler/executor.py
@@ -17,8 +17,12 @@
from argo_jupyter_scheduler.utils import (
WorkflowActionsEnum,
+ add_file_logger,
authenticate,
gen_cron_workflow_name,
+ gen_default_html_path,
+ gen_default_output_path,
+ gen_log_path,
gen_papermill_command_input,
gen_workflow_name,
sanitize_label,
@@ -181,9 +185,17 @@ def create_workflow(
db_url: str,
use_conda_store_env: bool = True,
):
+ input_path = staging_paths["input"]
+ log_path = gen_log_path(input_path)
+
+ # Configure logging to file first
+ add_file_logger(logger, log_path)
+
authenticate()
logger.info("creating workflow...")
+ logger.info(f"create time: {job.create_time}")
+ logger.info(f"staging paths: {staging_paths}")
labels = {
"jupyterflow-override": "true",
@@ -192,26 +204,6 @@ def create_workflow(
os.environ["PREFERRED_USERNAME"]
),
}
- cmd_args = [
- "-c",
- *gen_papermill_command_input(
- job.runtime_environment_name,
- staging_paths["input"],
- use_conda_store_env,
- ),
- ]
- envs = []
- if parameters:
- for key, value in parameters.items():
- envs.append(Env(name=key, value=value))
-
- main = Container(
- name="main",
- command=["/bin/sh"],
- args=cmd_args,
- env=envs,
- )
-
ttl_strategy = TTLStrategy(
seconds_after_completion=DEFAULT_TTL,
seconds_after_success=DEFAULT_TTL,
@@ -227,13 +219,53 @@ def create_workflow(
labels=labels,
ttl_strategy=ttl_strategy,
) as w:
+ main = main_container(
+ job, use_conda_store_env, input_path, log_path, parameters
+ )
+
with Steps(name="steps"):
Step(name="main", template=main, continue_on=ContinueOn(failed=True))
+
+ rename_files(
+ name="rename-files",
+ arguments={
+ "db_url": None,
+ "job_definition_id": None,
+ "input_path": input_path,
+ "log_path": log_path,
+ "start_time": job.create_time,
+ },
+ continue_on=ContinueOn(failed=True),
+ )
+
+ failure += " || {{steps.rename-files.status}} == Failed"
+ successful += " && {{steps.rename-files.status}} == Succeeded"
+
+ token, channel = get_slack_token_channel(parameters)
+ if token is not None and channel is not None:
+ send_to_slack(
+ name="send-to-slack",
+ arguments={
+ "db_url": None,
+ "job_definition_id": None,
+ "input_path": input_path,
+ "start_time": job.create_time,
+ "token": token,
+ "channel": channel,
+ "log_path": log_path,
+ },
+ when=successful,
+ continue_on=ContinueOn(failed=True),
+ )
+ failure += " || {{steps.send-to-slack.status}} == Failed"
+ successful += " && {{steps.send-to-slack.status}} == Succeeded"
+
update_job_status_failure(
name="failure",
arguments={"db_url": db_url, "job_id": job.job_id},
when=failure,
)
+
update_job_status_success(
name="success",
arguments={"db_url": db_url, "job_id": job.job_id},
@@ -290,7 +322,7 @@ def stop_workflow(self, job_id):
logger.info("workflow stopped")
- def _create_cwf_oject(
+ def _create_cwf_object(
self,
job: DescribeJobDefinition,
parameters: Dict[str, str],
@@ -302,9 +334,18 @@ def _create_cwf_oject(
active: bool = True,
use_conda_store_env: bool = True,
):
+ input_path = staging_paths["input"]
+ log_path = gen_log_path(input_path)
+
+ # Configure logging to file first
+ add_file_logger(logger, log_path)
+
# Argo-Workflow verbage vs Jupyter-Scheduler verbage
suspend = not active
+ logger.info(f"create time: {job.create_time}")
+ logger.info(f"staging paths: {staging_paths}")
+
labels = {
"jupyterflow-override": "true",
"jupyter-scheduler-job-definition-id": job_definition_id,
@@ -312,25 +353,7 @@ def _create_cwf_oject(
os.environ["PREFERRED_USERNAME"]
),
}
- cmd_args = [
- "-c",
- *gen_papermill_command_input(
- job.runtime_environment_name,
- staging_paths["input"],
- use_conda_store_env,
- ),
- ]
- envs = []
- if parameters:
- for key, value in parameters.items():
- envs.append(Env(name=key, value=value))
-
- main = Container(
- name="main",
- command=["/bin/sh"],
- args=cmd_args,
- env=envs,
- )
+
ttl_strategy = TTLStrategy(
seconds_after_completion=DEFAULT_TTL,
seconds_after_success=DEFAULT_TTL,
@@ -340,7 +363,7 @@ def _create_cwf_oject(
# mimics internals of the `scheduler.create_job_from_definition` method
attributes = {
**job.dict(exclude={"schedule", "timezone"}, exclude_none=True),
- "input_uri": staging_paths["input"],
+ "input_uri": input_path,
}
model = CreateJob(**attributes)
@@ -360,6 +383,10 @@ def _create_cwf_oject(
labels=labels,
ttl_strategy=ttl_strategy,
) as cwf:
+ main = main_container(
+ job, use_conda_store_env, input_path, log_path, parameters
+ )
+
with Steps(name="steps"):
create_job_record(
name="create-job-id",
@@ -369,7 +396,43 @@ def _create_cwf_oject(
"job_definition_id": job_definition_id,
},
)
+
Step(name="main", template=main, continue_on=ContinueOn(failed=True))
+
+ rename_files(
+ name="rename-files",
+ arguments={
+ "db_url": db_url,
+ "job_definition_id": job_definition_id,
+ "input_path": input_path,
+ "log_path": log_path,
+ "start_time": None,
+ },
+ continue_on=ContinueOn(failed=True),
+ )
+
+ failure += " || {{steps.rename-files.status}} == Failed"
+ successful += " && {{steps.rename-files.status}} == Succeeded"
+
+ token, channel = get_slack_token_channel(parameters)
+ if token is not None and channel is not None:
+ send_to_slack(
+ name="send-to-slack",
+ arguments={
+ "db_url": db_url,
+ "job_definition_id": job_definition_id,
+ "input_path": input_path,
+ "start_time": None,
+ "token": token,
+ "channel": channel,
+ "log_path": log_path,
+ },
+ when=successful,
+ continue_on=ContinueOn(failed=True),
+ )
+ failure += " || {{steps.send-to-slack.status}} == Failed"
+ successful += " && {{steps.send-to-slack.status}} == Succeeded"
+
update_job_status_failure(
name="failure",
arguments={
@@ -378,6 +441,7 @@ def _create_cwf_oject(
},
when=failure,
)
+
update_job_status_success(
name="success",
arguments={
@@ -404,7 +468,7 @@ def create_cron_workflow(
logger.info("creating cron workflow...")
- w = self._create_cwf_oject(
+ w = self._create_cwf_object(
job=job,
parameters=parameters,
staging_paths=staging_paths,
@@ -466,7 +530,7 @@ def update_cron_workflow(
schedule = job_definition.schedule
timezone = job_definition.timezone
- w = self._create_cwf_oject(
+ w = self._create_cwf_object(
job=job,
parameters=self.parameters,
staging_paths=staging_paths,
@@ -490,6 +554,32 @@ def update_cron_workflow(
logger.info("cron workflow updated")
+def main_container(job, use_conda_store_env, input_path, log_path, parameters):
+ envs = []
+ if parameters is not None:
+ for key, value in parameters.items():
+ envs.append(Env(name=key, value=value))
+
+ output_path = gen_default_output_path(input_path)
+ html_path = gen_default_html_path(input_path)
+
+ cmd_args = gen_papermill_command_input(
+ conda_env_name=job.runtime_environment_name,
+ input_path=input_path,
+ output_path=output_path,
+ html_path=html_path,
+ log_path=log_path,
+ use_conda_store_env=use_conda_store_env,
+ )
+
+ return Container(
+ name="main",
+ command=["/bin/sh"],
+ args=["-c", cmd_args],
+ env=envs,
+ )
+
+
@script()
def update_job_status_failure(db_url, job_id=None, job_definition_id=None):
from jupyter_scheduler.models import Status
@@ -573,3 +663,143 @@ def create_job_record(
session.add(job)
session.commit()
+
+
+@script()
+def rename_files(db_url, job_definition_id, input_path, log_path, start_time):
+ import os
+
+ from jupyter_scheduler.orm import Job, create_session
+
+ from argo_jupyter_scheduler.utils import (
+ add_file_logger,
+ gen_default_html_path,
+ gen_default_output_path,
+ gen_html_path,
+ gen_output_path,
+ setup_logger,
+ )
+
+ # Sets up logging
+ logger = setup_logger("rename_files")
+ add_file_logger(logger, log_path)
+
+ try:
+ # Gets start_time if not provided to generate file paths
+ if start_time is None:
+ db_session = create_session(db_url)
+ with db_session() as session:
+ q = (
+ session.query(Job)
+ .filter(Job.job_definition_id == job_definition_id)
+ .order_by(Job.start_time.desc())
+ .first()
+ )
+
+ # The current job id doesn't match the id in the staging area.
+ # Creates a symlink to make files downloadable via the web UI.
+ basedir = os.path.dirname(os.path.dirname(input_path))
+ old_dir = os.path.join(basedir, job_definition_id)
+ new_dir = os.path.join(basedir, q.job_id)
+ os.symlink(old_dir, new_dir)
+
+ start_time = q.start_time
+
+ old_output_path = gen_default_output_path(input_path)
+ old_html_path = gen_default_html_path(input_path)
+
+ new_output_path = gen_output_path(input_path, start_time)
+ new_html_path = gen_html_path(input_path, start_time)
+
+ # Renames files
+ os.rename(old_output_path, new_output_path)
+ os.rename(old_html_path, new_html_path)
+
+ logger.info("Successfully renamed files")
+
+ except Exception as e:
+ msg = "Failed to rename files"
+ logger.exception(msg)
+ raise Exception(msg) from e
+
+
+def get_slack_token_channel(parameters):
+ token = None
+ channel = None
+
+ if parameters is not None:
+ token = parameters.get("SLACK_TOKEN")
+ channel = parameters.get("SLACK_CHANNEL")
+
+ return token, channel
+
+
+@script()
+def send_to_slack(
+ db_url, job_definition_id, input_path, start_time, token, channel, log_path
+):
+ import json
+
+ import requests
+ from jupyter_scheduler.orm import Job, create_session
+
+ from argo_jupyter_scheduler.utils import (
+ add_file_logger,
+ gen_html_path,
+ setup_logger,
+ )
+
+ # Sets up logging
+ logger = setup_logger("send_to_slack")
+ add_file_logger(logger, log_path)
+
+ try:
+ # Gets start_time if not provided to generate file path
+ if start_time is None:
+ db_session = create_session(db_url)
+ with db_session() as session:
+ q = (
+ session.query(Job)
+ .filter(Job.job_definition_id == job_definition_id)
+ .order_by(Job.start_time.desc())
+ .first()
+ )
+
+ start_time = q.start_time
+
+ html_path = gen_html_path(input_path, start_time)
+
+ # Sends to Slack
+ url = "https://slack.com/api/files.upload"
+
+ files = {"file": (os.path.basename(html_path), open(html_path, "rb"))}
+
+ data = {
+ "initial_comment": "Attaching new file",
+ "channels": channel,
+ }
+
+ headers = {
+ "Authorization": f"Bearer {token}",
+ }
+
+ logger.info(f"Sending to Slack: file: {html_path}, channel: {channel}")
+ response = requests.post(
+ url, files=files, data=data, headers=headers, timeout=30
+ )
+ response.raise_for_status()
+
+ response = response.text
+ logger.info(f"Slack response: {response}")
+
+ if not json.loads(response).get("ok"):
+ msg = "Unexpected response when sending to Slack"
+ logger.info(msg)
+ raise Exception(msg)
+
+ logger.info("Successfully sent to Slack")
+
+ except Exception as e:
+ msg = "Failed to send to Slack"
+ logger.exception(msg)
+ raise Exception(msg) from e
diff --git a/argo_jupyter_scheduler/scheduler.py b/argo_jupyter_scheduler/scheduler.py
index 18b1e4a..5bda130 100644
--- a/argo_jupyter_scheduler/scheduler.py
+++ b/argo_jupyter_scheduler/scheduler.py
@@ -132,7 +132,25 @@ def delete_job(self, job_id: str, delete_workflow: bool = True):
if staging_paths:
path = os.path.dirname(next(iter(staging_paths.values())))
if os.path.exists(path):
- shutil.rmtree(path)
+ if os.path.islink(path):
+ realpath = os.path.realpath(path)
+ # For cron jobs, job directories in the staging dir are
+ # symlinks that point to the real staging job directory,
+ # which is created when the job is first scheduled.
+ # Since we only have access to the current job
+ # directory, which is a symlink, we first need to find
+ # the other symlinks pointing to the same real directory
+ # and remove them
+ basedir = os.path.dirname(path)
+ for f in next(os.walk(basedir))[1]:
+ f = os.path.join(basedir, f) # noqa: PLW2901
+ if os.path.islink(f) and os.path.realpath(f) == realpath:
+ os.unlink(f)
+ # Then we remove the real staging job directory, where
+ # these symlinks used to point to
+ shutil.rmtree(realpath)
+ else:
+ shutil.rmtree(path)
if delete_workflow:
p = Process(
diff --git a/argo_jupyter_scheduler/utils.py b/argo_jupyter_scheduler/utils.py
index 2136bfb..27a86ca 100644
--- a/argo_jupyter_scheduler/utils.py
+++ b/argo_jupyter_scheduler/utils.py
@@ -8,6 +8,7 @@
import urllib3
from hera.shared import global_config
+from jupyter_scheduler.utils import create_output_filename
from urllib3.exceptions import ConnectionError
CONDA_STORE_TOKEN = "CONDA_STORE_TOKEN"
@@ -39,6 +40,12 @@ def setup_logger(name):
return logger
+def add_file_logger(logger, log_path):
+ logger.setLevel(logging.DEBUG)
+ fh = logging.FileHandler(log_path)
+ logger.addHandler(fh)
+
+
logger = setup_logger(__name__)
@@ -73,9 +80,30 @@ def gen_cron_workflow_name(job_definition_id: str):
return f"job-def-{job_definition_id}"
-def gen_output_path(input_path: str):
- p = Path(input_path)
- return str(p.parent / "output.ipynb")
+def gen_default_output_path(input_path: str):
+ # The initial filename before we can get access to the timestamp. Has the
+ # "0" suffix to avoid clashing with the input filename. This value will be
+ # pretty-printed as the Unix epoch when the file is created.
+ return gen_output_path(input_path, 0)
+
+
+def gen_default_html_path(input_path: str):
+ # The initial filename before we can get access to the timestamp. Has the
+ # "0" suffix to avoid clashing with the input filename. This value will be
+ # pretty-printed as the Unix epoch when the file is created.
+ return gen_html_path(input_path, 0)
+
+
+def gen_output_path(input_path: str, start_time: int):
+ # It's important to use this exact format to make files downloadable via the
+ # web UI.
+ return create_output_filename(input_path, start_time, "ipynb")
+
+
+def gen_html_path(input_path: str, start_time: int):
+ # It's important to use this exact format to make files downloadable via the
+ # web UI.
+ return create_output_filename(input_path, start_time, "html")
def gen_log_path(input_path: str):
@@ -157,24 +185,27 @@ def _valid_env(
def gen_papermill_command_input(
- conda_env_name: str, input_path: str, use_conda_store_env: bool = True
+ conda_env_name: str,
+ input_path: str,
+ output_path: str,
+ html_path: str,
+ log_path: str,
+ use_conda_store_env: bool = True,
):
# TODO: allow overrides
kernel_name = "python3"
- output_path = gen_output_path(input_path)
- log_path = gen_log_path(input_path)
conda_env_path = gen_conda_env_path(conda_env_name, use_conda_store_env)
logger.info(f"conda_env_path: {conda_env_path}")
logger.info(f"output_path: {output_path}")
logger.info(f"log_path: {log_path}")
+ logger.info(f"html_path: {html_path}")
+
+ papermill = f"papermill -k {kernel_name} {input_path} {output_path}"
+ jupyter = f"jupyter nbconvert --to html {output_path} --output {html_path}"
- return [
- f"conda run -p {conda_env_path} papermill -k {kernel_name} {input_path} {output_path}",
- "&>",
- log_path,
- ]
+ return f'conda run -p {conda_env_path} /bin/sh -c "{{ {papermill} && {jupyter} ; }} >> {log_path} 2>&1"'
def sanitize_label(s: str):
diff --git a/assets/create-job-slack.png b/assets/create-job-slack.png
new file mode 100644
index 0000000..a199be1
Binary files /dev/null and b/assets/create-job-slack.png differ
diff --git a/assets/slack-output.png b/assets/slack-output.png
new file mode 100644
index 0000000..8c7040a
Binary files /dev/null and b/assets/slack-output.png differ