Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jupyter notebook basic example #1754

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Jupyter notebook basic example #1754

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b442fba
Jupyter notebook basic example
Mecoli1219 Oct 18, 2024
712019d
Update jupyter notebook & fix doc
Mecoli1219 Oct 28, 2024
792b27c
Update readme
Mecoli1219 Oct 28, 2024
85966d1
Move to basics
Mecoli1219 Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
289 changes: 289 additions & 0 deletions examples/basics/basics/basic_interactive_mode.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Basic Interactive Mode Usage in Jupyter Notebook Environment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Before you start, make sure your flytekit version is higher than 1.14.0. If not, please upgrade it by running the following command:\n",
"```bash\n",
"!pip install flytekit==1.14.0\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 01 Setup the Interface to Connect to the Flyte Backend\n",
"\n",
"Before starting to develop and run the tasks and workflows remotely via Jupiter Notebook, let's setup the interface via [flytekit.remote](https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.html) to interact with the Flyte backend. \n",
davidmirror-ops marked this conversation as resolved.
Show resolved Hide resolved
"\n",
"If you are running in Jupyter Notebook, it should automatically enable the interactive mode. However, you can also enable it explicitly by setting `interactive_mode_enabled` to `True`. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\">21:39:52.768159 </span><span style=\"color: #808000; text-decoration-color: #808000\">WARNING </span> remote.py:<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">273</span> - Jupyter notebook and interactive task \n",
"<span style=\"color: #7fbfbf; text-decoration-color: #7fbfbf\"> </span> support is still alpha. \n",
"</pre>\n"
],
"text/plain": [
"\u001b[2;36m21:39:52.768159\u001b[0m\u001b[2;36m \u001b[0m\u001b[33mWARNING \u001b[0m remote.py:\u001b[1;36m273\u001b[0m - Jupyter notebook and interactive task \n",
"\u001b[2;36m \u001b[0m support is still alpha. \n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from flytekit.remote import FlyteRemote\n",
"from flytekit.configuration import Config\n",
"\n",
"# The `for_sandbox` method instantiates a connection to the demo cluster.\n",
davidmirror-ops marked this conversation as resolved.
Show resolved Hide resolved
"remote = FlyteRemote(\n",
" Config.for_sandbox(), \n",
" default_project=\"flytesnacks\", \n",
" default_domain=\"development\", \n",
" interactive_mode_enabled=True, # This is not necessary if you are in a notebook.\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 02 Develop the tasks & workflows"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's create a simple task and workflow to demonstrate how to develop and run the tasks and workflows remotely via Jupiter Notebook."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from flytekit import task, workflow\n",
"\n",
"@task\n",
"def hello(name: str) -> str:\n",
" return f\"Hello {name}!\"\n",
"\n",
"@task\n",
"def world(pre: str) -> str:\n",
" return f\"{pre} Welcome to the Jupyter Notebook!\"\n",
"\n",
"@workflow\n",
"def wf(name: str) -> str:\n",
" return world(pre=hello(name=name))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"When executing the task or workflow via `FlyteRemote` interface, the related entities will be automatically registered to the Flyte backend. You can navigate to the Flyte Console via the URL to see the progress and outputs of the executions, or you can use `remote.wait()` to wait for the execution to complete. This may take a few seconds to minutes depending on the size of the task and workflow."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"http://localhost:30080/console/projects/flytesnacks/domains/development/executions/ah5cm5s482w9fl5jzz7c\n",
"{'o0': 'Hello Flyte!'}\n"
]
}
],
"source": [
"# Execute the task\n",
"exe = remote.execute(hello, inputs={\"name\": \"Flyte\"})\n",
"\n",
"# This will print the URL to the console\n",
"print(exe.execution_url)\n",
"\n",
"# Wait for the task to complete\n",
"exe = exe.wait(poll_interval=1)\n",
"\n",
"# Print the outputs\n",
"print(exe.outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are not interested in the URL to the Flyte Console and want the execution wait directly to complete, you can set `wait=True` in the `remote.execute()` method."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'o0': 'Hello world!'}\n"
]
}
],
"source": [
"# Execute the workflow and wait for it to complete\n",
"exe = remote.execute(hello, inputs={\"name\": \"world\"}, wait=True)\n",
"\n",
"# Print the outputs\n",
"print(exe.outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, we can also execute the workflows remotely and extract the outputs."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'o0': 'Hello world! Welcome to the Jupyter Notebook!'}\n"
]
}
],
"source": [
"# Execute the workflow and wait for it to complete\n",
"exe = remote.execute(wf, inputs={\"name\": \"world\"})\n",
"\n",
"# Wait for the task to complete\n",
"exe = exe.wait(poll_interval=1)\n",
"\n",
"# Print the outputs\n",
"print(exe.outputs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you repeat the execution, it will not re-register the entities to the backend, instead it will reuse the existing entities. However, if you re-execute the cells that define the tasks or workflows, it will register the updated entities to the backend when you execute them.\n",
"\n",
"Feel free to update the tasks and workflows definition and see how it works!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 03 Develop the Workflow Contains `map_task`"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`map_task` is also supported in the remote execution. You can define the `map_task` in the workflow and execute it remotely."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"from flytekit import task, workflow, map_task\n",
"from functools import partial\n",
"\n",
"@task\n",
"def fn(x: int, y: int) -> int:\n",
" return x + y + 2\n",
"\n",
"@workflow\n",
"def workflow_with_maptask(data: list[int], y: int) -> list[int]:\n",
" partial_fn = partial(fn, y=y)\n",
" return map_task(partial_fn)(x=data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's execute the workflow remotely and extract the outputs."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'o0': [7, 8, 9]}\n"
]
}
],
"source": [
"# Execute the workflow with map_task and wait for it to complete\n",
"out = remote.execute(workflow_with_maptask, inputs={\"data\": [1, 2, 3], \"y\": 4})\n",
"\n",
"# Wait for the task to complete\n",
"out = out.wait(poll_interval=1)\n",
"\n",
"# Print the outputs\n",
"print(out.outputs)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "flytekit3.12",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading