diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cf7ccbd6969e..77f6fe59cc41 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,10 +29,8 @@ There are several subpackages included that are used by Hydroflow but are more g * `multiplatform_test` provides a convenience macro for specifying and initializing tests on various platforms. -There are a few auxillary repositories as well: +There are auxillary repositories as well: -* [`hydro-project/hydroflow-template`](https://github.com/hydro-project/hydroflow-template) - provides a way to create a new Hydroflow user project, via `cargo generate hydro-project/hydroflow-template`. * [`hydro-project/rust-sitter`](https://github.com/hydro-project/rust-sitter) provides a [Tree-sitter](https://tree-sitter.github.io/tree-sitter/)-based parser generator interface, used by `hydroflow_datalog`. diff --git a/README.md b/README.md index 73bbba64fd8f..da04f8a3587f 100644 --- a/README.md +++ b/README.md @@ -40,15 +40,26 @@ For more, check out the [surface syntax section of the Hydroflow book](https://h We provide a `cargo-generate` template for you to get started from a simple working example. To install `cargo-generate`, run the following: -```bash, ignore +```bash,ignore cargo install cargo-generate ``` -Then run -```bash, ignore -cargo generate gh:hydro-project/hydroflow-template +Then run: +```bash,ignore +cargo generate gh:hydro-project/hydroflow template/hydroflow ``` -and you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice + +`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: +```bash +#shell-command-next-line +cd +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test +``` + +And you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice for adapting it to other uses. ### Enable IDE Support for Ligatures diff --git a/docs/docs/deploy/your-first-deploy.md b/docs/docs/deploy/your-first-deploy.md index c4498abaebc8..06084f58a9f4 100644 --- a/docs/docs/deploy/your-first-deploy.md +++ b/docs/docs/deploy/your-first-deploy.md @@ -13,7 +13,17 @@ First, we need to write the Hydroflow application, which will intergrate with Hy cargo install --locked cargo-generate #shell-command-next-line -cargo generate hydro-project/hydroflow-template +cargo generate gh:hydro-project/hydroflow template/hydroflow +``` + +`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: +```bash +#shell-command-next-line +cd +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test ``` We'll need to add an additional dependency for `hydroflow_cli_integration` to our `Cargo.toml`: @@ -77,13 +87,13 @@ async fn main() { let input_recv = ports .port("input") // connect to the port with a single recipient - .connect::() + .connect::() .await .into_source(); let output_send = ports .port("output") - .connect::() + .connect::() .await .into_sink(); diff --git a/docs/docs/hydroflow/concepts/index.md b/docs/docs/hydroflow/concepts/index.md index e0dcf8abec66..854e8d528b6c 100644 --- a/docs/docs/hydroflow/concepts/index.md +++ b/docs/docs/hydroflow/concepts/index.md @@ -2,12 +2,12 @@ Hydroflow is different from other distributed systems infrastructure, so you probably have questions, like: What is Hydroflow? How does it work? What is special about it, and what are the key concepts to understand? -This chapter covers those basic questions. We start simple describing Hydroflow, and build up to an understanding of +This chapter covers those basic questions. We start simple describing Hydroflow, and build up to an understanding of what makes Hydroflow uniquely powerful. But in case you want a preview of the Hydroflow goodies, here are the main themes: 1. **Distributed Correctness**: Hydroflow's type system can prevent distributed system bugs at compile time. (One example: will your code -produce the same results if you deploy on a single machine or replicate on a distributed cluster of machines?) +produce the same results if you deploy on a single machine or replicate on a distributed cluster of machines?) 2. **Local Efficiency**: Hydroflow compiles your dataflow programs into efficient, low-latency, single-threaded executables. Taken together, Hydroflow provides a high-efficiency substrate on which to build powerful languages for distributed computing. @@ -20,36 +20,36 @@ Setting fanfare aside, what *is* Hydroflow? Hydroflow is a library that can be used in any Rust program. It includes two main components: 1. A runtime library that executes low-latency, reactive dataflow programs written in Rust. (The *core API*.) -2. A domain-specific language (DSL) for specifying dataflow programs. (The Hydroflow *surface syntax*.) +2. A domain-specific language (DSL) for specifying dataflow programs. (The Hydroflow *surface syntax*.) -Hydroflow's surface syntax must be embedded in a Rust program; the Rust compiler takes that Hydroflow syntax and -compiles it into an efficient binary executable. +Hydroflow's surface syntax must be embedded in a Rust program; the Rust compiler takes that Hydroflow syntax and +compiles it into an efficient binary executable. > We call a running Hydroflow binary a *transducer*. -In typical usage, a developer writes a transducer as a single-threaded Rust program that is mostly composed of -Hydroflow surface syntax. Each transducer is typically responsible for a single +In typical usage, a developer writes a transducer as a single-threaded Rust program that is mostly composed of +Hydroflow surface syntax. Each transducer is typically responsible for a single "node" (a machine, or a core) in a distributed system composed of many such transducers, which send and receive flows of data to each other. -> Hydroflow itself does not generate distributed code. It is a library for specifying the transducers (individual nodes) that -> participate in a distributed system. +> Hydroflow itself does not generate distributed code. It is a library for specifying the transducers (individual nodes) that +> participate in a distributed system. > -> In the [Hydro Project](https://hydro.run), higher-level languages are being built on top of Hydroflow to generate -> distributed code in the form of multiple transducers. -> Meanwhile, you can use Hydroflow to write your own distributed code, by writing individual transducers that work together, +> In the [Hydro Project](https://hydro.run), higher-level languages are being built on top of Hydroflow to generate +> distributed code in the form of multiple transducers. +> Meanwhile, you can use Hydroflow to write your own distributed code, by writing individual transducers that work together, > and deploying them manually or with a tool like [Hydroplane](https://github.com/hydro-project/hydroplane). See the [Hydro Ecosystem](../ecosystem) for more on this. ### So how might a human write distributed systems with Hydroflow? -As an illustration of how you can work at the Hydroflow layer, consider the -[Chat Server example](https://github.com/hydro-project/hydroflow-template). If you run that binary +As an illustration of how you can work at the Hydroflow layer, consider the +[Chat Server example](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples/chat). If you run that binary with the command-line argument `--role server` it will start a single transducer that is responsible for a chat server: receiving membership requests and messages from clients, and forwarding messages from individual clients to all other clients. -If you run that binary with the argument `--role client` it will start a transducer that is responsible for a chat client, which -forwards chat messages from stdin to the server, and prints out messages sent by the server. As a distributed system, the chat +If you run that binary with the argument `--role client` it will start a transducer that is responsible for a chat client, which +forwards chat messages from stdin to the server, and prints out messages sent by the server. As a distributed system, the chat service would typically consist of many client transducers and a single server transducer. -Note that this is an example of an extremely simple distributed system in a "star" or "hub-and spokes" topology: the multiple client transducers are completely independent of each other, and each talks only with the central server transducer. +Note that this is an example of an extremely simple distributed system in a "star" or "hub-and spokes" topology: the multiple client transducers are completely independent of each other, and each talks only with the central server transducer.
@@ -65,8 +65,8 @@ graph TD; ```
- If we wanted something more interesting, we could consider deploying a cluster of multiple server transducers, say for fault tolerance or geo-distribution, perhaps like this: - + If we wanted something more interesting, we could consider deploying a cluster of multiple server transducers, say for fault tolerance or geo-distribution, perhaps like this: +
```mermaid @@ -84,5 +84,5 @@ graph LR; ```
- We'd need to change the server code to get servers sharing their state in a correct and reliable manner. But that's a topic for another section **TODO**; for now, let's stay focused on the basics. + We'd need to change the server code to get servers sharing their state in a correct and reliable manner. But that's a topic for another section **TODO**; for now, let's stay focused on the basics. diff --git a/docs/docs/hydroflow/quickstart/example_1_simplest.mdx b/docs/docs/hydroflow/quickstart/example_1_simplest.mdx index 3b3b75f1aee7..4a3f72c2bb34 100644 --- a/docs/docs/hydroflow/quickstart/example_1_simplest.mdx +++ b/docs/docs/hydroflow/quickstart/example_1_simplest.mdx @@ -20,10 +20,10 @@ the numbers in `0..10`. Create a clean template project: ```console #shell-command-next-line -cargo generate hydro-project/hydroflow-template -⚠️ Favorite `hydro-project/hydroflow-template` not found in config, using it as a git repository: https://github.com/hydro-project/hydroflow-template.git +cargo generate gh:hydro-project/hydroflow template/hydroflow +⚠️ Favorite `gh:hydro-project/hydroflow` not found in config, using it as a git repository: https://github.com/hydro-project/hydroflow.git 🤷 Project Name: simple -🔧 Destination: /Users/jmh/code/sussudio/simple ... +🔧 Destination: /Users/me/code/simple ... 🔧 project-name: simple ... 🔧 Generating template ... [11/11] Done: src @@ -32,7 +32,15 @@ cargo generate hydro-project/hydroflow-template ✨ Done! New project created /simple ``` -Change directory into the resulting `simple` folder or open it in your IDE. Then edit the `src/main.rs` file, replacing +After `cd`ing into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: +```bash +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test +``` + +Then edit the `src/main.rs` file, replacing *all* of its contents with the following code: {exampleCode} diff --git a/docs/docs/hydroflow/quickstart/example_7_networking.mdx b/docs/docs/hydroflow/quickstart/example_7_networking.mdx index 2d89b390a642..027804991d86 100644 --- a/docs/docs/hydroflow/quickstart/example_7_networking.mdx +++ b/docs/docs/hydroflow/quickstart/example_7_networking.mdx @@ -12,18 +12,28 @@ Hydroflow provides a robust framework for developing networked services. The bes the Hydroflow template. ```bash -cargo generate hydro-project/hydroflow-template +cargo generate gh:hydro-project/hydroflow template/hydroflow +``` + +`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: +```bash +#shell-command-next-line +cd +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test ``` The template contains a chat server & client implementation using UDP. You can start multiple instances of the client and connect to the same server. ## Things to Check -* [README](https://github.com/hydro-project/hydroflow-template) - Contains instructions on building the template and +* [README](https://github.com/hydro-project/hydroflow/tree/main/template/hydroflow#readme) - Contains instructions on building the template and running the client and server examples. -* [server.rs](https://github.com/hydro-project/hydroflow-template/blob/main/src/server.rs) - Contains the chat server +* [server.rs](https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/server.rs) - Contains the chat server implementation. The inline rust documentation explains the networking-related hydroflow operators and their usage. -* [client.rs](https://github.com/hydro-project/hydroflow-template/blob/main/src/client.rs) - Contains the chat client +* [client.rs](https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/client.rs) - Contains the chat client implementation. The inline rust documentation explains the networking-related hydroflow operators and their usage. Hydroflow also supports TCP for reliable, connection-oriented communication. Understanding the differences and use diff --git a/docs/docs/hydroflow/quickstart/setup.md b/docs/docs/hydroflow/quickstart/setup.md index f3a2c477688d..027665899ba9 100644 --- a/docs/docs/hydroflow/quickstart/setup.md +++ b/docs/docs/hydroflow/quickstart/setup.md @@ -41,6 +41,7 @@ In this book we will be using the Hydroflow template generator, which we recomme as a starting point for your Hydroflow projects. For this purpose you will need to install the `cargo-generate` tool: ```bash +#shell-command-next-line cargo install cargo-generate ``` @@ -50,26 +51,35 @@ We recommend using VS Code with the `rust-analyzer` extension (and NOT the `Rust` extension). ## Setting up a Hydroflow Project -The easiest way to get started with Hydroflow is to begin with a template project. +The easiest way to get started with Hydroflow is to begin with a template project. Create a directory where you'd like to put that project, direct your terminal there and run: ```bash -cargo generate hydro-project/hydroflow-template +#shell-command-next-line +cargo generate gh:hydro-project/hydroflow template/hydroflow +``` +You will be prompted to name your project. The `cargo generate` command will create a subdirectory +with the relevant files and folders. + +`cd` into the generated folder, ensure the correct nightly version of rust is installed: +```bash +#shell-command-next-line +cd +#shell-command-next-line +rustup update ``` -You will be prompted to name your project. The `cargo generate` command will create a subdirectory -with the relevant files and folders. As part of generating the project, the `hydroflow` library will be downloaded as a dependency. You can then open the project in VS Code or IDE of your choice, or you can simply build the template project with `cargo build`. ```bash -cd +#shell-command-next-line cargo build ``` This should return successfully. The template provides a simple working example of a Hydroflow program. As a sort of "hello, world" of distributed systems, it implements an "echo server" that -simply echoes back the messages you sent it; it also implements a client to test the server. +simply echoes back the messages you sent it; it also implements a client to test the server. We will replace the code in that example with our own, but it's a good idea to run it first to make sure everything is working. :::note @@ -104,7 +114,7 @@ Hello! This book will assume you are using the template project, but some Rust experts may want to get started with Hydroflow by cloning and working in the -repository directly. +repository directly. You should fork the repository if you want to push your changes. @@ -120,7 +130,7 @@ will provide inline type and error messages, code completion, etc. To work with the repository, it's best to start with an "example", found in the [`hydroflow/examples` folder](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples). -The simplest example is the +The simplest example is the ['hello world'](https://github.com/hydro-project/hydroflow/blob/main/hydroflow/examples/hello_world/main.rs) example; the simplest example with networking is the [`echo server`](https://github.com/hydro-project/hydroflow/blob/main/hydroflow/examples/echoserver/main.rs). @@ -129,4 +139,4 @@ The Hydroflow repository is set up as a [workspace](https://doc.rust-lang.org/bo i.e. a repo containing a bunch of separate packages, `hydroflow` is just the main one. So if you want to work in a proper separate cargo package, you can create one and add it into the [root `Cargo.toml`](https://github.com/hydro-project/hydroflow/blob/main/Cargo.toml), -much like the [provided template](https://github.com/hydro-project/hydroflow-template/blob/main/Cargo.toml). +much like the [provided template](https://github.com/hydro-project/hydroflow/tree/main/template/hydroflow#readme). diff --git a/docs/docs/hydroflow_plus/index.mdx b/docs/docs/hydroflow_plus/index.mdx index dde0e1896cac..fe0edfd2bef4 100644 --- a/docs/docs/hydroflow_plus/index.mdx +++ b/docs/docs/hydroflow_plus/index.mdx @@ -14,12 +14,15 @@ Hydroflow+ requires a particular workspace setup, as any crate that uses Hydrofl #shell-command-next-line cargo install cargo-generate #shell-command-next-line -cargo generate hydro-project/hydroflow-plus-template +cargo generate hydro-project/hydroflow template/hydroflow_plus ``` -Then, you can test the example dataflow: - +`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: ```bash #shell-command-next-line +cd +#shell-command-next-line +rustup update +#shell-command-next-line cargo test ``` diff --git a/docs/docs/hydroflow_plus/quickstart/index.mdx b/docs/docs/hydroflow_plus/quickstart/index.mdx index 361cbf7ad6f7..d366e2f61046 100644 --- a/docs/docs/hydroflow_plus/quickstart/index.mdx +++ b/docs/docs/hydroflow_plus/quickstart/index.mdx @@ -7,8 +7,16 @@ To get started with a new project, we'll use the Hydroflow+ template. The templa #shell-command-next-line cargo install cargo-generate #shell-command-next-line -cargo generate hydro-project/hydroflow-plus-template +cargo generate gh:hydro-project/hydroflow template/hydroflow_plus # Project Name: my-example-project #shell-command-next-line cd my-example-project ``` + +After `cd`ing into the generated folder, ensure the correct nightly version of rust is installed and test the generated project: +```bash +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test +``` \ No newline at end of file diff --git a/docs/docs/hydroflow_plus/quickstart/structure.mdx b/docs/docs/hydroflow_plus/quickstart/structure.mdx index 1bd078fa08e2..1c4d63ec549e 100644 --- a/docs/docs/hydroflow_plus/quickstart/structure.mdx +++ b/docs/docs/hydroflow_plus/quickstart/structure.mdx @@ -20,7 +20,17 @@ We recommend using the Hydroflow+ template to get started with a new project. Th #shell-command-next-line cargo install cargo-generate #shell-command-next-line -cargo generate hydro-project/hydroflow-plus-template +cargo generate gh:hydro-project/hydroflow template/hydroflow_plus +``` + +`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project: +```bash +#shell-command-next-line +cd +#shell-command-next-line +rustup update +#shell-command-next-line +cargo test ``` ::: diff --git a/docs/src/pages/playground.js b/docs/src/pages/playground.js index 7f146b9dd729..e1f706d3f5a1 100644 --- a/docs/src/pages/playground.js +++ b/docs/src/pages/playground.js @@ -66,7 +66,7 @@ source_iter(0..10) -> for_each(|n| println!("Howdy {}", n));`, "Chat Server": `\ -// https://github.com/hydro-project/hydroflow-template +// https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/server.rs // Define shared inbound and outbound channels outbound_chan = union() -> dest_sink_serde(outbound); inbound_chan = source_stream_serde(inbound) @@ -87,7 +87,7 @@ inbound_chan[msgs] -> [0]broadcast; clients[1] -> [1]broadcast;`, "Chat Client": `\ -// https://github.com/hydro-project/hydroflow-template +// https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/client.rs // set up channels outbound_chan = union() -> dest_sink_serde(outbound); inbound_chan = source_stream_serde(inbound) -> map(|(m, _)| m) diff --git a/template/hydroflow/README.md b/template/hydroflow/README.md index 6013a39bee71..2fb853951920 100644 --- a/template/hydroflow/README.md +++ b/template/hydroflow/README.md @@ -4,15 +4,19 @@ distributed services. It implements a simple echo server and client over UDP. ## Using the Template ```bash -cargo generate hydro-project/hydroflow-template +cargo generate gh:hydro-project/hydroflow template/hydroflow ``` -You will be prompted to name your project. Once the command completes, you can `cd` into the project and build the -template. +You will be prompted to name your project. Once the command completes, you can `cd` into the project. +Ensure the correct nightly version of rust is installed: ```bash -cd -cargo build +rustup update +``` + +Then test the project: +```bash +cargo test ``` ## Running the Template diff --git a/template/hydroflow_plus/README.md b/template/hydroflow_plus/README.md index c5cd40ea3fd4..8b1589e79f4b 100644 --- a/template/hydroflow_plus/README.md +++ b/template/hydroflow_plus/README.md @@ -2,14 +2,19 @@ This is a template for a Rust project that uses [Hydroflow+](http://github.com/hydro-project/hydroflow) for distributed services. To generate a project, run ```bash -$ cargo install cargo-generate -$ cargo generate hydro-project/hydroflow-plus-template +cargo install cargo-generate +cargo generate gh:hydro-project/hydroflow template/hydroflow_plus +cd ``` -Once the command completes, you can cd into the project and test the template. +After `cd`ing into the workspace, ensure the correct nightly version of rust is installed: ```bash -$ cd -$ cargo test +rustup update +``` + +Then test the project: +```bash +cargo test ``` ## Project Structure