Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: update cargo-generate commands, say to run rustup update to ensure nightly rust is installed #1344

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,8 @@ There are several subpackages included that are used by Hydroflow but are more g
* `multiplatform_test` provides a convenience macro for specifying and initializing tests on
various platforms.

There are a few auxillary repositories as well:
There are auxillary repositories as well:

* [`hydro-project/hydroflow-template`](https://github.com/hydro-project/hydroflow-template)
provides a way to create a new Hydroflow user project, via `cargo generate hydro-project/hydroflow-template`.
* [`hydro-project/rust-sitter`](https://github.com/hydro-project/rust-sitter) provides a
[Tree-sitter](https://tree-sitter.github.io/tree-sitter/)-based parser generator interface, used
by `hydroflow_datalog`.
Expand Down
21 changes: 16 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,26 @@ For more, check out the [surface syntax section of the Hydroflow book](https://h
We provide a `cargo-generate` template for you to get started from a simple working example.

To install `cargo-generate`, run the following:
```bash, ignore
```bash,ignore
cargo install cargo-generate
```

Then run
```bash, ignore
cargo generate gh:hydro-project/hydroflow-template
Then run:
```bash,ignore
cargo generate gh:hydro-project/hydroflow template/hydroflow
```
and you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice

`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```

And you will get a well-formed Hydroflow/Rust project to use as a starting point. It provides a simple Echo Server and Client, and advice
for adapting it to other uses.

### Enable IDE Support for Ligatures
Expand Down
16 changes: 13 additions & 3 deletions docs/docs/deploy/your-first-deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,17 @@ First, we need to write the Hydroflow application, which will intergrate with Hy
cargo install --locked cargo-generate

#shell-command-next-line
cargo generate hydro-project/hydroflow-template
cargo generate gh:hydro-project/hydroflow template/hydroflow
```

`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```

We'll need to add an additional dependency for `hydroflow_cli_integration` to our `Cargo.toml`:
Expand Down Expand Up @@ -77,13 +87,13 @@ async fn main() {
let input_recv = ports
.port("input")
// connect to the port with a single recipient
.connect::<ConnectedDirect>()
.connect::<ConnectedDirect>()
.await
.into_source();

let output_send = ports
.port("output")
.connect::<ConnectedDirect>()
.connect::<ConnectedDirect>()
.await
.into_sink();

Expand Down
40 changes: 20 additions & 20 deletions docs/docs/hydroflow/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@
Hydroflow is different from other distributed systems infrastructure, so you probably have questions, like:
What is Hydroflow? How does it work? What is special about it, and what are the key concepts to understand?

This chapter covers those basic questions. We start simple describing Hydroflow, and build up to an understanding of
This chapter covers those basic questions. We start simple describing Hydroflow, and build up to an understanding of
what makes Hydroflow uniquely powerful.

But in case you want a preview of the Hydroflow goodies, here are the main themes:
1. **Distributed Correctness**: Hydroflow's type system can prevent distributed system bugs at compile time. (One example: will your code
produce the same results if you deploy on a single machine or replicate on a distributed cluster of machines?)
produce the same results if you deploy on a single machine or replicate on a distributed cluster of machines?)
2. **Local Efficiency**: Hydroflow compiles your dataflow programs into efficient, low-latency, single-threaded executables.

Taken together, Hydroflow provides a high-efficiency substrate on which to build powerful languages for distributed computing.
Expand All @@ -20,36 +20,36 @@ Setting fanfare aside, what *is* Hydroflow?
Hydroflow is a library that can be used in any Rust program. It includes two main components:

1. A runtime library that executes low-latency, reactive dataflow programs written in Rust. (The *core API*.)
2. A domain-specific language (DSL) for specifying dataflow programs. (The Hydroflow *surface syntax*.)
2. A domain-specific language (DSL) for specifying dataflow programs. (The Hydroflow *surface syntax*.)

Hydroflow's surface syntax must be embedded in a Rust program; the Rust compiler takes that Hydroflow syntax and
compiles it into an efficient binary executable.
Hydroflow's surface syntax must be embedded in a Rust program; the Rust compiler takes that Hydroflow syntax and
compiles it into an efficient binary executable.

> We call a running Hydroflow binary a *transducer*.

In typical usage, a developer writes a transducer as a single-threaded Rust program that is mostly composed of
Hydroflow surface syntax. Each transducer is typically responsible for a single
In typical usage, a developer writes a transducer as a single-threaded Rust program that is mostly composed of
Hydroflow surface syntax. Each transducer is typically responsible for a single
"node" (a machine, or a core) in a distributed system composed of many such transducers,
which send and receive flows of data to each other.

> Hydroflow itself does not generate distributed code. It is a library for specifying the transducers (individual nodes) that
> participate in a distributed system.
> Hydroflow itself does not generate distributed code. It is a library for specifying the transducers (individual nodes) that
> participate in a distributed system.
>
> In the [Hydro Project](https://hydro.run), higher-level languages are being built on top of Hydroflow to generate
> distributed code in the form of multiple transducers.
> Meanwhile, you can use Hydroflow to write your own distributed code, by writing individual transducers that work together,
> In the [Hydro Project](https://hydro.run), higher-level languages are being built on top of Hydroflow to generate
> distributed code in the form of multiple transducers.
> Meanwhile, you can use Hydroflow to write your own distributed code, by writing individual transducers that work together,
> and deploying them manually or with a tool like [Hydroplane](https://github.com/hydro-project/hydroplane). See the [Hydro Ecosystem](../ecosystem) for more on this.

### So how might a human write distributed systems with Hydroflow?
As an illustration of how you can work at the Hydroflow layer, consider the
[Chat Server example](https://github.com/hydro-project/hydroflow-template). If you run that binary
As an illustration of how you can work at the Hydroflow layer, consider the
[Chat Server example](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples/chat). If you run that binary
with the command-line argument `--role server` it will start a single transducer that is responsible for a chat server: receiving
membership requests and messages from clients, and forwarding messages from individual clients to all other clients.
If you run that binary with the argument `--role client` it will start a transducer that is responsible for a chat client, which
forwards chat messages from stdin to the server, and prints out messages sent by the server. As a distributed system, the chat
If you run that binary with the argument `--role client` it will start a transducer that is responsible for a chat client, which
forwards chat messages from stdin to the server, and prints out messages sent by the server. As a distributed system, the chat
service would typically consist of many client transducers and a single server transducer.

Note that this is an example of an extremely simple distributed system in a "star" or "hub-and spokes" topology: the multiple client transducers are completely independent of each other, and each talks only with the central server transducer.
Note that this is an example of an extremely simple distributed system in a "star" or "hub-and spokes" topology: the multiple client transducers are completely independent of each other, and each talks only with the central server transducer.

<div align="center">

Expand All @@ -65,8 +65,8 @@ graph TD;
```
</div>

If we wanted something more interesting, we could consider deploying a cluster of multiple server transducers, say for fault tolerance or geo-distribution, perhaps like this:
If we wanted something more interesting, we could consider deploying a cluster of multiple server transducers, say for fault tolerance or geo-distribution, perhaps like this:

<div align="center">

```mermaid
Expand All @@ -84,5 +84,5 @@ graph LR;
```
</div>

We'd need to change the server code to get servers sharing their state in a correct and reliable manner. But that's a topic for another section **TODO**; for now, let's stay focused on the basics.
We'd need to change the server code to get servers sharing their state in a correct and reliable manner. But that's a topic for another section **TODO**; for now, let's stay focused on the basics.

16 changes: 12 additions & 4 deletions docs/docs/hydroflow/quickstart/example_1_simplest.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ the numbers in `0..10`.
Create a clean template project:
```console
#shell-command-next-line
cargo generate hydro-project/hydroflow-template
⚠️ Favorite `hydro-project/hydroflow-template` not found in config, using it as a git repository: https://github.com/hydro-project/hydroflow-template.git
cargo generate gh:hydro-project/hydroflow template/hydroflow
⚠️ Favorite `gh:hydro-project/hydroflow` not found in config, using it as a git repository: https://github.com/hydro-project/hydroflow.git
🤷 Project Name: simple
🔧 Destination: /Users/jmh/code/sussudio/simple ...
🔧 Destination: /Users/me/code/simple ...
🔧 project-name: simple ...
🔧 Generating template ...
[11/11] Done: src
Expand All @@ -32,7 +32,15 @@ cargo generate hydro-project/hydroflow-template
✨ Done! New project created <dir>/simple
```

Change directory into the resulting `simple` folder or open it in your IDE. Then edit the `src/main.rs` file, replacing
After `cd`ing into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```

Then edit the `src/main.rs` file, replacing
*all* of its contents with the following code:

<CodeBlock language="rust" showLineNumbers>{exampleCode}</CodeBlock>
Expand Down
18 changes: 14 additions & 4 deletions docs/docs/hydroflow/quickstart/example_7_networking.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,28 @@ Hydroflow provides a robust framework for developing networked services. The bes
the Hydroflow template.

```bash
cargo generate hydro-project/hydroflow-template
cargo generate gh:hydro-project/hydroflow template/hydroflow
```

`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```

The template contains a chat server & client implementation using UDP. You can start multiple instances of
the client and connect to the same server.

## Things to Check
* [README](https://github.com/hydro-project/hydroflow-template) - Contains instructions on building the template and
* [README](https://github.com/hydro-project/hydroflow/tree/main/template/hydroflow#readme) - Contains instructions on building the template and
running the client and server examples.
* [server.rs](https://github.com/hydro-project/hydroflow-template/blob/main/src/server.rs) - Contains the chat server
* [server.rs](https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/server.rs) - Contains the chat server
implementation. The inline rust documentation explains the networking-related hydroflow operators and their usage.
* [client.rs](https://github.com/hydro-project/hydroflow-template/blob/main/src/client.rs) - Contains the chat client
* [client.rs](https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/client.rs) - Contains the chat client
implementation. The inline rust documentation explains the networking-related hydroflow operators and their usage.

Hydroflow also supports TCP for reliable, connection-oriented communication. Understanding the differences and use
Expand Down
28 changes: 19 additions & 9 deletions docs/docs/hydroflow/quickstart/setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ In this book we will be using the Hydroflow template generator, which we recomme
as a starting point for your Hydroflow projects. For this purpose you
will need to install the `cargo-generate` tool:
```bash
#shell-command-next-line
cargo install cargo-generate
```

Expand All @@ -50,26 +51,35 @@ We recommend using VS Code with the `rust-analyzer` extension (and NOT the
`Rust` extension).

## Setting up a Hydroflow Project
The easiest way to get started with Hydroflow is to begin with a template project.
The easiest way to get started with Hydroflow is to begin with a template project.
Create a directory where you'd like to put that project, direct your terminal there and run:
```bash
cargo generate hydro-project/hydroflow-template
#shell-command-next-line
cargo generate gh:hydro-project/hydroflow template/hydroflow
```
You will be prompted to name your project. The `cargo generate` command will create a subdirectory
with the relevant files and folders.

`cd` into the generated folder, ensure the correct nightly version of rust is installed:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
```
You will be prompted to name your project. The `cargo generate` command will create a subdirectory
with the relevant files and folders.

As part of generating the project, the `hydroflow` library will be downloaded as a dependency.
You can then open the project in VS Code or IDE of your choice, or
you can simply build the template project with `cargo build`.
```bash
cd <project name>
#shell-command-next-line
cargo build
```
This should return successfully.

The template provides a simple working example of a Hydroflow program.
As a sort of "hello, world" of distributed systems, it implements an "echo server" that
simply echoes back the messages you sent it; it also implements a client to test the server.
simply echoes back the messages you sent it; it also implements a client to test the server.
We will replace the code in that example with our own, but it's a good idea to run it first to make sure everything is working.

:::note
Expand Down Expand Up @@ -104,7 +114,7 @@ Hello!

This book will assume you are using the template project, but some
Rust experts may want to get started with Hydroflow by cloning and working in the
repository directly.
repository directly.
You should fork the repository if you want to push your
changes.

Expand All @@ -120,7 +130,7 @@ will provide inline type and error messages, code completion, etc.

To work with the repository, it's best to start with an "example", found in the
[`hydroflow/examples` folder](https://github.com/hydro-project/hydroflow/tree/main/hydroflow/examples).
The simplest example is the
The simplest example is the
['hello world'](https://github.com/hydro-project/hydroflow/blob/main/hydroflow/examples/hello_world/main.rs) example;
the simplest example with networking is the
[`echo server`](https://github.com/hydro-project/hydroflow/blob/main/hydroflow/examples/echoserver/main.rs).
Expand All @@ -129,4 +139,4 @@ The Hydroflow repository is set up as a [workspace](https://doc.rust-lang.org/bo
i.e. a repo containing a bunch of separate packages, `hydroflow` is just the
main one. So if you want to work in a proper separate cargo package, you can
create one and add it into the [root `Cargo.toml`](https://github.com/hydro-project/hydroflow/blob/main/Cargo.toml),
much like the [provided template](https://github.com/hydro-project/hydroflow-template/blob/main/Cargo.toml).
much like the [provided template](https://github.com/hydro-project/hydroflow/tree/main/template/hydroflow#readme).
9 changes: 6 additions & 3 deletions docs/docs/hydroflow_plus/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,15 @@ Hydroflow+ requires a particular workspace setup, as any crate that uses Hydrofl
#shell-command-next-line
cargo install cargo-generate
#shell-command-next-line
cargo generate hydro-project/hydroflow-plus-template
cargo generate hydro-project/hydroflow template/hydroflow_plus
```

Then, you can test the example dataflow:

`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```
10 changes: 9 additions & 1 deletion docs/docs/hydroflow_plus/quickstart/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,16 @@ To get started with a new project, we'll use the Hydroflow+ template. The templa
#shell-command-next-line
cargo install cargo-generate
#shell-command-next-line
cargo generate hydro-project/hydroflow-plus-template
cargo generate gh:hydro-project/hydroflow template/hydroflow_plus
# Project Name: my-example-project
#shell-command-next-line
cd my-example-project
```

After `cd`ing into the generated folder, ensure the correct nightly version of rust is installed and test the generated project:
```bash
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```
12 changes: 11 additions & 1 deletion docs/docs/hydroflow_plus/quickstart/structure.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,17 @@ We recommend using the Hydroflow+ template to get started with a new project. Th
#shell-command-next-line
cargo install cargo-generate
#shell-command-next-line
cargo generate hydro-project/hydroflow-plus-template
cargo generate gh:hydro-project/hydroflow template/hydroflow_plus
```

`cd` into the generated folder, ensure the correct nightly version of rust is installed, and test the generated project:
```bash
#shell-command-next-line
cd <my-project>
#shell-command-next-line
rustup update
#shell-command-next-line
cargo test
```

:::
Expand Down
4 changes: 2 additions & 2 deletions docs/src/pages/playground.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ source_iter(0..10)
-> for_each(|n| println!("Howdy {}", n));`,

"Chat Server": `\
// https://github.com/hydro-project/hydroflow-template
// https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/server.rs
// Define shared inbound and outbound channels
outbound_chan = union() -> dest_sink_serde(outbound);
inbound_chan = source_stream_serde(inbound)
Expand All @@ -87,7 +87,7 @@ inbound_chan[msgs] -> [0]broadcast;
clients[1] -> [1]broadcast;`,

"Chat Client": `\
// https://github.com/hydro-project/hydroflow-template
// https://github.com/hydro-project/hydroflow/blob/main/template/hydroflow/src/client.rs
// set up channels
outbound_chan = union() -> dest_sink_serde(outbound);
inbound_chan = source_stream_serde(inbound) -> map(|(m, _)| m)
Expand Down
Loading
Loading