feat(hydroflow): Kube integration #1436
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…disable cloning (hydro-project#1404) Cloning was unsafe because values behind a `Rc<RefCell<...>>` in the case of tee would be entangled with the old IR. --- [//]: # (BEGIN SAPLING FOOTER) Stack created with [Sapling](https://sapling-scm.com). Best reviewed with [ReviewStack](https://reviewstack.dev/hydro-project/hydroflow/pull/1404). * hydro-project#1405 * hydro-project#1398 * __->__ hydro-project#1404
…-duplicated items (hydro-project#1412) BREAKING CHANGE: `state()` behaves slightly differently now Picked out of hydro-project#1406
…ydro-project#1411) Now, when a group only has a single active task, we skip printing a line for the group itself and instead collapse its information into the line for the inner task (recursively as necessary). This allows us to show more fine grained progress without overflowing the console.
Will move it out of the blind gossip PR next.
Co-authored-by: Shadaj Laddad <shadaj@users.noreply.github.com>
…o-project#1399) BREAKING CHANGE: changes some pub-facing names. Behaviors are the same.
…ck` (hydro-project#1423) Moved from hydro-project#1418 --------- Co-authored-by: Shadaj Laddad <shadaj@users.noreply.github.com>
…d linux localhost, fix hydro-project#1351 (hydro-project#1396)
…ult (hydro-project#1421) Now, by default streams exist at a "top-level" where there are no ticks and operators run over the entire collection. To perform iterative computations, developers must explicitly entire a tick domain (using `tick_batch`), and return to the outer domain (using `all_ticks`).
…#1431) When kernel stacks are included, the DWARF traces can become corrupted / overflown leading to flamegraphs with broken parents. We only are interested in usermode, anyways, and can measure I/O overhead through other methods.
hydro-project#1684) From hydro-project#1635. fix hydro-project#1572. BREAKING CHANGE: Renames `TransducerBuilderContext` to `ProcessBuilderContext` --------- Co-authored-by: Joe Hellerstein <jmh@berkeley.edu>
hydro-project#1614 (hydro-project#1690) Previous code also inserted `\<` into code blocks. This fixes the original issue of unescaped `<`s by ensuring all op docs have them in `code blocks`, removes the escaping.
…ject#1682) Currently, state_by uses Default::default to instantiate the backing storage. Accepting a factory function will allow the storage to be tweaked per instance of state. Example usage: pre-allocating memory for the data structures.
A pass of edits on the existing text. --------- Co-authored-by: Shadaj Laddad <shadaj@users.noreply.github.com>
Blocks some content especially on mobile
…roject#1695) Also address docs feedback for streams.
…o kube_integration_rebased
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR sets up kubernetes within Hydro Deploy. Users can create Pod hosts and deploy their programs on the machines just as they would with localhost and cloud providers. Note that we're assuming here that users will already have a Kubernetes cluster set up somewhere.
We integrate with kube.rs (which integrates with kubectl). When setting up a pod, we first create the pod and loop until it is ready to execute commands. Then, we take the compiled hydroflow binary and copy it into the pod over stdin. We use
lsofto track when files have finished being written to and finally execute the binary on the pod.We're intending this PR to be the start of auto-scaling in Hydro! Kubernetes integration will let users set procedures for vertical / horizontal auto-scaling.