Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: replace TryCP with Nomad for running scenarios #136

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,3 +32,6 @@ logs/**/*.log
scenarios/*/logs
run_summary.jsonl
summariser-report-*.json

# A symlink to the output directory of nix builds
result
35 changes: 35 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,41 @@ You can then start a second terminal and run one of the scenarios in the `scenar
```bash
RUST_LOG=info CONDUCTOR_CONFIG="CI" TRYCP_RUST_LOG="info" MIN_PEERS=2 cargo run --package trycp_write_validated -- --targets targets-ci.yaml --instances-per-target 2 --duration 60
```
#### Running Wind Tunnel Scenarios with Nomad

> [!Warning]
> This is a work in progress and currently only works with the `app_install` scenario.

##### Running Locally

You can easily test the Wind Tunnel scenarios with [Nomad](https://www.nomadproject.io) by running them locally, this requires running a Nomad agent locally
as both a client and a server.

First, enter the Nix `devShell` with `nix develop` to make sure you have all the packages install.
Alternatively, [install Nomad](https://developer.hashicorp.com/nomad/install) and Holochain locally so that both `nomad` and `hc` are in your `PATH`.

Once Nomad is installed, run the agent in `dev` mode to spin up both a server and client, do this with:
```shell
nomad agent -dev
```

Now navigate to <http://localhost:4646/ui> to view the Nomad dashboard.

Next, in a new terminal window, build the `app_install` scenario with:
```shell
nix build .#app_install
```

Once the scenario is built you can run the Nomad job with:
```shell
nomad job run nomad/run-app_install-local.nomad.hcl
```

Then, navigate to <http://localhost:4646/ui/jobs/app_install_scenario@default> where you should see one allocation (the Nomad name for an instance of the job)
this allocation should have two tasks: the `start-holochain` task and the `run_scenario` task. You can view the logs of these tasks to see the results.
The allocation should be marked as "Completed" after a few seconds.

Once you've finished testing you can kill the Nomad agent with `^C` in the first terminal running the agent.

### Published crates

Expand Down
5 changes: 4 additions & 1 deletion flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@
systems = builtins.attrNames inputs.holonix.devShells;
perSystem = { inputs', pkgs, system, config, ... }:
let
unfreePkgs = import nixpkgs { inherit system; config.allowUnfree = true; };
rustMod = flake-parts-lib.importApply ./nix/modules/rust.nix { inherit crane rust-overlay nixpkgs; };
in
{
Expand All @@ -68,8 +69,9 @@
./nix/modules/zomes.nix
];


devShells.default = pkgs.mkShell {
packages = with pkgs; [
packages = [
pkgs.influxdb2-cli
pkgs.influxdb2-server
# TODO https://docs.influxdata.com/telegraf/v1/install/#ntp
Expand All @@ -81,6 +83,7 @@
pkgs.taplo
pkgs.yamlfmt
pkgs.perl
unfreePkgs.nomad
config.rustHelper.rust
inputs'.holonix.packages.holochain
inputs'.holonix.packages.lair-keystore
Expand Down
35 changes: 35 additions & 0 deletions nomad/run-app_install-local.nomad.hcl
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
job "app_install_scenario" {
type = "batch"
group "app_install" {
task "start_holochain" {
lifecycle {
hook = "prestart"
sidecar = true
}

driver = "raw_exec"
config {
command = "bash"
args = ["-c", "hc s clean && echo 1234 | hc s --piped create && echo 1234 | hc s --piped -f 8888 run"]
}
}

task "run_scenario" {
driver = "raw_exec"
env {
RUST_LOG = "info"
}
config {
command = abspath("result/bin/app_install")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will there be a way to have this reference a common path locally and remotely so that we can use the same scripts in dev as on a real environment?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm hoping to allow this but I'm not yet sure what is the best way. I think at the very least it should be possible to provide either a path or a URL and if it's a URL then the binary gets downloaded as an artefact before being run.

args = [
"--connection-string", "ws://localhost:8888",
"--agents", "2",
"--behaviour", "minimal:1",
"--behaviour", "large:1",
"--duration", "5",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this default to a longer time when you're ready to merge this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My idea (already in the latest version) is that if no duration is provided then it is not actually set and so it will use the default provided by the scenario. This could be an issue for neverending scenarios but I think it makes the most sense and we can always kill a running scenario at set the duration if we miss it by accident.

"--no-progress"
]
}
}
}
}
Loading