Skip to content

Commit

Permalink
feat(topolotree): new implementation and Hydro Deploy setup
Browse files Browse the repository at this point in the history
  • Loading branch information
Saikrishna Achalla authored and shadaj committed Sep 18, 2023
1 parent e628da5 commit e208140
Show file tree
Hide file tree
Showing 486 changed files with 15,876 additions and 4,611 deletions.
11 changes: 1 addition & 10 deletions .github/workflows/build-cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,17 +105,8 @@ jobs:
with:
working-directory: hydro_cli
target: ${{ matrix.platform.target }}
manylinux: musllinux_1_2
manylinux: auto
args: --release --out dist
- uses: uraimo/run-on-arch-action@master
name: Install built wheel
with:
arch: ${{ matrix.platform.arch }}
distro: alpine_latest
install: |
apk add py3-pip
run: |
pip3 install ${{ env.PACKAGE_NAME }} --no-index --find-links hydro_cli/dist/ --force-reinstall
- name: "Upload wheels"
uses: actions/upload-artifact@v3
with:
Expand Down
30 changes: 25 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ jobs:
uses: actions-rs/cargo@v1
with:
command: check
args: --all-targets
args: --all-targets --features python

check-wasm:
name: Check WebAssembly
Expand Down Expand Up @@ -91,7 +91,7 @@ jobs:
uses: actions-rs/cargo@v1
with:
command: check
args: -p hydroflow_lang --target wasm32-unknown-unknown --no-default-features
args: -p hydroflow_lang --target wasm32-unknown-unknown

test:
name: Test Suite
Expand Down Expand Up @@ -121,11 +121,17 @@ jobs:
toolchain: nightly
override: ${{ matrix.rust_release == 'latest-nightly' }}

- name: Run cargo test
- name: Run cargo test all targets (which does not include doctests)
uses: actions-rs/cargo@v1
with:
command: test
args: --no-fail-fast --all-targets --features python

- name: Run cargo doc tests
uses: actions-rs/cargo@v1
with:
command: test
args: --no-fail-fast
args: --no-fail-fast --doc --features python

test-wasm:
name: Test Suite (WebAssembly)
Expand Down Expand Up @@ -275,7 +281,21 @@ jobs:
uses: actions-rs/cargo@v1
with:
command: clippy
args: --all-targets -- -D warnings
args: --all-targets --features python -- -D warnings

build-website:
name: Build Website
if: ${{ needs.pre_job.outputs.should_skip != 'true' || github.event_name != 'pull_request' }}
timeout-minutes: 25
needs: pre_job
runs-on: ubuntu-latest

steps:
- name: Checkout sources
uses: actions/checkout@v3

- name: Build Website
run: bash build_docs.bash x86_64-linux-gnu-ubuntu-20.04

docs:
name: Docs (rustdoc)
Expand Down
72 changes: 50 additions & 22 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ members = [
"multiplatform_test",
"pusherator",
"relalg",
"topolotree",
"variadics",
"website_playground",
]
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
<a href="https://docs.rs/hydroflow/"><img src="https://img.shields.io/badge/docs.rs-Hydroflow-blue?style=flat-square&logo=read-the-docs&logoColor=white" alt="Docs.rs"></a>
</p>

Hydroflow is a low-latency dataflow runtime written in Rust. The [Hydro Project](https://hydro.run/docs/hydroflow/ecosystem/)
will empower developers to harness the full potential of the cloud by making distributed programs easy to specify and automatic to scale. Hydroflow serves as the lowest level in the [Hydro stack](https://hydro.run/docs/hydroflow/ecosystem/),
Hydroflow is a low-latency dataflow runtime written in Rust. The goal of the [Hydro Project](https://hydro.run)
is to empower developers to harness the full potential of the cloud by making distributed programs easy to specify and automatic to scale. Hydroflow is the lowest level in the [Hydro stack](https://hydro.run/docs/hydroflow/ecosystem/),
serving as a single-node low-latency runtime with explicit networking. This allows us to support
not just data processing pipelines, but distributed protocols (e.g. Paxos) and real-world
long-running applications as well.
Expand All @@ -17,16 +17,16 @@ Take a look at the [Hydroflow Book](https://hydro.run/docs/hydroflow/).

## The Hydroflow Surface Syntax

Hydroflow comes with a custom "surface syntax" domain-specific language which serves as a very
simple, readable IR for specifying single-node Hydroflow programs, intended to be stitched together
by the Hydro stack to create larger autoscaling distributed systems.'
Hydroflow comes with a custom "surface syntax"—a domain-specific language which serves as a very
simple, readable IR for specifying single-node Hydroflow programs. These programs are intended to be stitched together
by the Hydro stack to create larger autoscaling distributed systems.

Here's a simple example of the surface syntax. Check out the [Hydroflow Playground](https://hydro.run/playground)
for an interactive demo.
```rust
source_iter(0..10)
-> map(|n| n * n)
-> filter(|&n| n > 10)
-> filter(|n| *n > 10)
-> foo;

foo = map(|n| (n..=n+1))
Expand Down
6 changes: 1 addition & 5 deletions benches/benches/fork_join.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,11 +54,7 @@ fn benchmark_hydroflow(c: &mut Criterion) {
send1,
send2,
|_ctx, recv1, recv2, send1, send2| {
for v in recv1
.take_inner()
.into_iter()
.chain(recv2.take_inner().into_iter())
{
for v in recv1.take_inner().into_iter().chain(recv2.take_inner()) {
if v % 2 == 0 {
send1.give(Some(v));
} else {
Expand Down
Loading

0 comments on commit e208140

Please sign in to comment.