diff --git a/docs/courses/dynamic-app/README.md b/docs/courses/dynamic-app/README.md new file mode 100644 index 00000000..b24f2343 --- /dev/null +++ b/docs/courses/dynamic-app/README.md @@ -0,0 +1,184 @@ +# +

+ +

+ +

+ Website + | + Python API + | + Rust API + | + Guide + | + Discord +

+ +
+ + Build and test + + + + + + rust docs + + + PyPi Latest Release + +
+ +An extremely fast and simple **dataflow oriented robotic** framework to manage your projects and run complex **apps**, written in Rust. + +Here’s an English version of the second course, written in a clear and structured way: + +--- + +## Dynamic App: Course no.2 + +After completing Course no.1, you should feel more comfortable with `dora` and have a better understanding of how to build applications. In this course (which will be shorter than the previous one), we will make our `talker/listener` application more dynamic. + +You might have noticed that `dora` currently executes our nodes in a somewhat static way, meaning we don’t have control over the process where our node is executed—**yet**. + +`dora` has a feature that allows users to control the execution process of their nodes themselves, rather than relying on the `daemon`. + +### Application Structure + +Let’s start with the same structure from the previous course, with a `venv` already set up: + +``` +my_app/ +├── .venv/ # Directory containing the virtual environment +├── dataflow.yml # The YAML file representing the dataflow, to be executed by `dora` +└── nodes/ # Directory containing our simple nodes + ├── talker.py + └── listener.py +``` + +### The Dataflow + +Now, let’s modify the `dataflow.yml` to enable the `dynamic` function in `dora`: + +```yaml +nodes: + - id: talker + path: dynamic + outputs: + - message + + - id: listener + path: dynamic + inputs: + message: talker/message +``` + +#### Explanation: +When the `path` parameter is set to `dynamic`, `dora` will wait for you to manually start the `dynamic` nodes. + +### Python + +In our applications, it’s **essential** to specify the node name when creating a node, like so: `node = Node("talker")`. This lets `dora` know which node you want to run. + +Let’s make a small adjustment to our application: + +```python +# talker.py +from dora import Node +import pyarrow as pa + +node = Node("talker") # Providing the node name is necessary for dynamic nodes. + +text = input("Enter a message: ") +data = pa.array([text]) +node.send_output("message", data) +``` + +Now we’re finished with the code! Let's see how to run our application using `dora`. + +### Running the Application + +To run this application, you’ll need to open **two** terminal windows and activate your `venv` in each. Then, execute the following: + +```bash +# terminal 1 +source .venv/bin/activate +dora up +dora start dataflow.yml --detach +python nodes/listener.py +``` + +```bash +# terminal 2 +source .venv/bin/activate +python nodes/talker.py +``` + +In **terminal 2**, you should see the prompt `"Enter a message: "`. After entering a message, it will be displayed in **terminal 1**. + +### Full Code + +Here is the complete code for our application: + +```YAML +# dataflow.yml +nodes: + - id: talker + path: dynamic + outputs: + - message + + - id: listener + path: dynamic + inputs: + message: talker/message +``` + +```python +# talker.py +from dora import Node +import pyarrow as pa + +node = Node("talker") + +text = input("Enter a message: ") +data = pa.array([text]) +node.send_output("message", data) +``` + +```python +# listener.py +from dora import Node + +node = Node("listener") + +event = node.next() + +print(event) +``` + +--- + +This concludes our tutorial! You’ve successfully created a simple messaging app using `dora`. Happy coding! + +## Contributing + +We are passionate about supporting contributors of all levels of experience and would love to see +you get involved in the project. See the +[contributing guide](https://github.com/dora-rs/dora/blob/main/CONTRIBUTING.md) to get started. + +## Discussions + +Our main communication channels are: + +- [Our Discord server](https://discord.gg/6eMGGutkfE) +- [Our Github Project Discussion](https://github.com/orgs/dora-rs/discussions) + +Feel free to reach out on any topic, issues or ideas. + +We also have [a contributing guide](CONTRIBUTING.md). + +## License + +This project is licensed under Apache-2.0. Check out [NOTICE.md](NOTICE.md) for more information. diff --git a/docs/courses/dynamic-app/my_app/dataflow.yml b/docs/courses/dynamic-app/my_app/dataflow.yml new file mode 100644 index 00000000..cce9955d --- /dev/null +++ b/docs/courses/dynamic-app/my_app/dataflow.yml @@ -0,0 +1,10 @@ +nodes: + - id: talker + path: dynamic + outputs: + - message + + - id: listener + path: dynamic + inputs: + message: talker/message diff --git a/docs/courses/dynamic-app/my_app/nodes/listener.py b/docs/courses/dynamic-app/my_app/nodes/listener.py new file mode 100644 index 00000000..91f95f55 --- /dev/null +++ b/docs/courses/dynamic-app/my_app/nodes/listener.py @@ -0,0 +1,7 @@ +from dora import Node + +node = Node("listener") + +event = node.next() + +print(event) diff --git a/docs/courses/dynamic-app/my_app/nodes/talker.py b/docs/courses/dynamic-app/my_app/nodes/talker.py new file mode 100644 index 00000000..abd99df6 --- /dev/null +++ b/docs/courses/dynamic-app/my_app/nodes/talker.py @@ -0,0 +1,8 @@ +from dora import Node +import pyarrow as pa + +node = Node("talker") + +text = input("Enter a message: ") +data = pa.array([text]) +node.send_output("message", data) diff --git a/docs/courses/getting-started/README.md b/docs/courses/getting-started/README.md new file mode 100644 index 00000000..1dcab1ce --- /dev/null +++ b/docs/courses/getting-started/README.md @@ -0,0 +1,305 @@ +# +

+ +

+ +

+ Website + | + Python API + | + Rust API + | + Guide + | + Discord +

+ +
+ + Build and test + + + + + + rust docs + + + PyPi Latest Release + +
+ +An extremely fast and simple **dataflow oriented robotic** framework to manage your projects and run complex **apps**, written in Rust. + +## Getting Started: Course no.1 + +You've heard about `dora` and want to give it a try? You're in the right place! Let's build a simple application in Python that allows sending a message (a string) from one app (the talker) to another (the listener). + +### Installation + +If you're here, it's likely that you haven't installed the necessary tools yet. Let's fix that! You’ll need two tools: +- **`dora`**, the software (CLI) that lets you run your applications. +- A Python environment manager like **`uv`** (a tool I highly recommend). However, if you're familiar with Python and know how to create virtual environments, feel free to use the tools you're comfortable with! + +To install `dora`, follow these simple steps (run the following commands in a terminal): + +#### On macOS and Linux +```bash +curl --proto '=https' --tlsv1.2 -sSf https://raw.githubusercontent.com/dora-rs/dora/main/install.sh | bash +``` + +#### On Windows +```powershell +powershell -c "irm https://raw.githubusercontent.com/dora-rs/dora/main/install.ps1 | iex" +``` + +To install `uv`, it's equally straightforward: + +#### On macOS and Linux +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +``` + +#### On Windows +```powershell +powershell -c "irm https://astral.sh/uv/install.ps1 | iex" +``` + +### Application Structure + +Typically, we follow this structure for `dora` applications (you’ll need to create these files yourself): + +``` +my_app/ +├── dataflow.yml # The YAML file representing the dataflow, to be executed by `dora` +└── nodes/ # Directory containing our simple nodes + ├── talker.py + └── listener.py +``` + +Next, let's create a Python virtual environment (`venv`) to isolate the necessary dependencies. You can do this with `uv` like so: + +```bash +cd my_app +uv venv --python 3.12 +``` + +Now, you can install the `dora` Python API: +```bash +cd my_app +uv pip install dora-rs +``` + +At this point, the application structure should look like this: + +``` +my_app/ +├── .venv/ # Directory containing the virtual environment +├── dataflow.yml # The YAML file representing the dataflow, to be executed by `dora` +└── nodes/ # Directory containing our simple nodes + ├── talker.py + └── listener.py +``` + +### The Dataflow + +Now, we’ll write the structural description of our application in the `dataflow.yml` file. Our application will have two nodes: a `talker` and a `listener`. Here’s the content for `dataflow.yml`: + +```yaml +nodes: + - id: talker + path: nodes/talker.py + + - id: listener + path: nodes/listener.py +``` + +But how does `dora` know to use our `venv` to run the nodes? There are two options: +- **Option 1**: Start the `dora` daemon after activating the `venv`. For example: +```bash +source .venv/bin/activate +dora up +``` +- **Option 2**: Make the `dataflow` more robust: +```yaml +nodes: + - id: talker + path: shell + args: | + source .venv/bin/activate + python nodes/talker.py + + - id: listener + path: shell + args: | + source .venv/bin/activate + python nodes/listener.py +``` +#### Windows +**For windows users it's hard to make everything works correctly**, so I recommend using the first option or this `dataflow.yml`: + +```yaml +nodes: + - id: talker + path: ./.venv/Scripts/python + args: nodes/talker.py + + - id: listener + path: ./.venv/Scripts/python + args: nodes/listener.py +``` + +**Note**: you may need to execute this command `Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy Bypass -Force;` as Admin + +--- + +Now you’re almost done setting up your `dataflow.yml`. + +### Inputs/Outputs + +Nodes can communicate with each other through defined `inputs` and `outputs`. In our case, the `talker` will send a message to the `listener`. Here's how we can define the `inputs` and `outputs` for each node: + +```yaml +nodes: + - id: talker + outputs: + - message + + - id: listener + inputs: + message: talker/message +``` + +With that, we’ve completed the `dataflow.yml` setup. Now, let’s move on to programming our application using Python. + +### Python + +In each of our node files, we need to create a `Node` object to handle communication between the nodes in the application. + +```python +from dora import Node + +node = Node("talker") # Providing the node name isn't required, but it makes the code clearer. +``` + +Additionally, for `talker.py`, we need to use the `pyarrow` library to send messages, as `dora` communicates via Arrow format. Add the following: + +```python +import pyarrow as pa +``` + +Next, we’ll use two functions: `send_output` in `talker.py` and `next` in `listener.py`. + +```python +# talker.py +data = pa.array(["Hello, World!", "My name is Dora"]) +node.send_output("message", data) +``` + +```python +# listener.py +event = node.next() +print(event) +``` + +Now we’re finished with the code! Let's see how to run our application using `dora`. + +### Running the Application + +We will use the `dora start` command. However, it's essential to understand how it works. `dora start` sends a message to two processes: the `daemon` and the `coordinator`. These processes will launch and connect our nodes based on the `dataflow.yml`. + +First, launch these two processes: + +```bash +dora up +dora start dataflow.yml +``` + +Once the application starts successfully, we’ll check that `listener.py` has received the message from `talker.py`. To do this, you need the UUID of your application, which is generated every time the app is launched. After running `dora start`, you should see an output with a UUID like **0191d7cd-0dab-701e-82b8-2efbe1adf51e**. Copy this and use it to check the logs of `listener.py`: + +```bash +dora logs 0191d7cd-0dab-701e-82b8-2efbe1adf51e listener +``` + +You should then see an output like this: + +```python +{'kind': 'dora', 'id': 'message', 'value': +[ + "Hello, World!", + "My name is Dora" +], '_cleanup': , 'metadata': {}, 'type': 'INPUT'} +``` + +As you can see, the message was successfully sent from `talker.py` to `listener.py`. + +### Full Code + +Here is the complete code for our application: + +```YAML +# dataflow.yml | see above for windows user +nodes: + - id: talker + path: shell + args: | + source .venv/bin/activate + python nodes/talker.py + outputs: + - message + + - id: listener + path: shell + args: | + source .venv/bin/activate + python nodes/listener.py + inputs: + message: talker/message +``` + +```python +# talker.py +from dora import Node +import pyarrow as pa + +node = Node("talker") + +node.send_output("message", pa.array(["Hello", "World"])) +``` + +```python +# listener.py +from dora import Node + +node = Node("listener") + +event = node.next() + +print(event) +``` + +--- + +This concludes our tutorial! You’ve successfully created a simple messaging app using `dora`. Happy coding! + +## Contributing + +We are passionate about supporting contributors of all levels of experience and would love to see +you get involved in the project. See the +[contributing guide](https://github.com/dora-rs/dora/blob/main/CONTRIBUTING.md) to get started. + +## Discussions + +Our main communication channels are: + +- [Our Discord server](https://discord.gg/6eMGGutkfE) +- [Our Github Project Discussion](https://github.com/orgs/dora-rs/discussions) + +Feel free to reach out on any topic, issues or ideas. + +We also have [a contributing guide](CONTRIBUTING.md). + +## License + +This project is licensed under Apache-2.0. Check out [NOTICE.md](NOTICE.md) for more information. diff --git a/docs/courses/getting-started/my_app/dataflow.yml b/docs/courses/getting-started/my_app/dataflow.yml new file mode 100644 index 00000000..499d70af --- /dev/null +++ b/docs/courses/getting-started/my_app/dataflow.yml @@ -0,0 +1,16 @@ +nodes: + - id: talker + path: shell + args: | + . .venv/bin/activate + python nodes/talker.py + outputs: + - message + + - id: listener + path: shell + args: | + . .venv/bin/activate + python nodes/listener.py + inputs: + message: talker/message diff --git a/docs/courses/getting-started/my_app/nodes/listener.py b/docs/courses/getting-started/my_app/nodes/listener.py new file mode 100644 index 00000000..91f95f55 --- /dev/null +++ b/docs/courses/getting-started/my_app/nodes/listener.py @@ -0,0 +1,7 @@ +from dora import Node + +node = Node("listener") + +event = node.next() + +print(event) diff --git a/docs/courses/getting-started/my_app/nodes/talker.py b/docs/courses/getting-started/my_app/nodes/talker.py new file mode 100644 index 00000000..b7df3165 --- /dev/null +++ b/docs/courses/getting-started/my_app/nodes/talker.py @@ -0,0 +1,6 @@ +from dora import Node +import pyarrow as pa + +node = Node("talker") + +node.send_output("message", pa.array(["Hello", "World"])) diff --git a/node-hub/README.md b/node-hub/README.md index 9e59bc16..add706f8 100644 --- a/node-hub/README.md +++ b/node-hub/README.md @@ -1,17 +1,21 @@ ## Dora Node Hub -This hub contains useful pre-built nodes for Dora. +This hub contains useful pre-packaged nodes for Dora. # Structure The structure of the node hub is as follows (please use the same structure if you need to add a new node): +## For Python + ``` node-hub/ -└── your-node/ - ├── main.py - ├── README.mdr - └── pyproject.toml +└── my-node/ + ├── README.md + ├── pyproject.toml + └── my_node/ + ├── __init__.py + └── main.py ``` The idea is to make a `pyproject.toml` file that will install the required dependencies for the node **and** attach main @@ -28,60 +32,42 @@ And then you will need to adapt the following `pyproject.toml` file: ```toml [tool.poetry] -name = "[name of the node e.g. video-encoder, with '-' to replace spaces]" -version = "0.1" -authors = ["[Pseudo/Name] <[email]>"] -description = "Dora Node for []" +name = "my-node" +version = "0.3.6" +authors = [ + "You" +] +description = "" readme = "README.md" -packages = [ - { include = "main.py", to = "[name of the node with '_' to replace spaces]" } -] +packages = [{ include = "my_node" }] [tool.poetry.dependencies] -python = "^3.11" -dora-rs = "0.3.5" -... [add your dependencies here] ... +dora-rs = "^0.3.6" +python = "^3.7" [tool.poetry.scripts] -[name of the node with '-' to replace spaces] = "[name of the node with '_' to replace spaces].main:main" +my-node = "my_node.main:main" [build-system] -requires = ["poetry-core>=1.0.0"] +requires = ["poetry-core>=1.8.0"] build-backend = "poetry.core.masonry.api" ``` -Finally, the README.md file should explicit all inputs/outputs of the node and how to configure it in the YAML file. - -# Example - -```toml -[tool.poetry] -name = "opencv-plot" -version = "0.1" -authors = [ - "Haixuan Xavier Tao ", - "Enzo Le Van " -] -description = "Dora Node for plotting data with OpenCV" -readme = "README.md" +## For Rust -packages = [ - { include = "main.py", to = "opencv_plot" } -] - -[tool.poetry.dependencies] -python = "^3.11" -dora-rs = "^0.3.5" -opencv-python = "^4.10.0.84" +``` +node-hub/ +└── my-node/ + ├── README.md + ├── Cargo.toml + └── src/ + └── main.py +``` -[tool.poetry.scripts] -opencv-plot = "opencv_plot.main:main" +# README -[build-system] -requires = ["poetry-core>=1.0.0"] -build-backend = "poetry.core.masonry.api" -``` +The README.md file should explicit all inputs/outputs of the node and how to configure it in the YAML file. ## License