[Extensibility] HTTP protocol (branch: http-extensibility) (#135)

* Initial commit * Working * Correct errors & revise for Device|Discovery v2 * Working * Should (!) work * Typo * Enrich * Refine * Fixes: #102 * Moves * Resolved * Revised for `dnsPolicy` * Dockerfiles * Added gRPC solution * Replaced * Kubernetes specs * Merge * Correct Kubernetes config filenames * Typo * Removed duplicate * Standalone broker exposes no ports * end-to-end-demo-gce.md: fixing 1 typo (#131) Hi, just removing the superfluous 's' Didier * link to codecov.io and use codecov action (#134) * Update end-to-end-demo.md (#112) * Update end-to-end-demo.md Hi there I extended the documentatio with the installation of linux-modules-extra because I had to add this package on Ubuntu 20.04 LTS (on a GCE instance): it seems that it doesn't come by default on distros / images aimed at backend servers. v4l loopback won't load if this other module is not present. Didier * Update end-to-end-demo.md fixing initial text according to revision notes in PR review * Update end-to-end-demo.md fixing style * Update end-to-end-demo.md simplifying text, based on PR review by Kate * rebase to finalize PR * Fix codecov for pull_request_target [SAME VERSION] (#138) * fix codecov for pull_request_target * add anti-pull-request-* conditional * Do not upload to codecov on PRs (#140) * end-to-end-demo-rpi4: fixing 1 typo (#143) Hello, Title says it all Didier * end-to-end demo : fixing 1 typo (#141) Hi there, The title says it all Didier * Minor e2e demo doc flow change (#139) * Minor flow change * Cleanup * Expand udev rule support (#130) * add support for SUBSYSTEMS, KERNELS, ATTRIBUTES, DRIVERS, and TAGS fields * fix formatting and add comments * add recursive search for subsystems * update udev configuration document with newly supported fields * check child device for field before looking up hierarchy * update version * document how to filter for only devices with devnodes when testing * reuse shared code * rust fmt fixes * add inline documentation for regex matching * Update end-to-end-demo.md (#113) Hi there, I added a comment regarding the importance of the version of v4l2loopback to avoid my personal initial mistake to others: I did not realize at start that the version was so important. So I went through this ticket with the dev: umlaeute/v4l2loopback#370 Didier Co-authored-by: bfjelds <bfjelds@microsoft.com> * Add sudo into udev configuration doc (#144) * Use arm7 * Reference arm7 * Reference branch on clone * `REPO="akri"` * Helm template for Agent already revised * Consistency * Use `./version.txt` for Akri version * Automate IMAGE updates * Build|Push gRPC images; Add'l consistency * Add Action to build HTTP Broker containers * I guess `http-extensibility` not `main` * Enable PRs to test changes to helm charts [SAME VERSION] (#146) * enable PRs to test changes to helm charts * handle manual job, fix push and release conditional * trigger e2e for helm changes * Updates references to `armv7` * Removed `shared` from HTTP Broker Dockerfiles * Correct version reference for image tags * Typo * Adds Golang Device|Discovery and alt-gRPC * s/println!/trace! * Consistency * Reflect relocation of apps * Simplify environment * Typo * Consistency on repo names for HTTP apps * Rename HTTP apps gRPC images * GitHub Actions to build HTTP apps containers * Configure broker security context (#148) * make udev broker security context configurable * update version * change group permissions to be default in example * fix comment with group id * Remove privileged security context from Akri Agent (#147) * Remove (redundant!?) aliasing * Remove healthchecks (`/healthz`) * Removed `akri-shared` dep * Simplified using `strategy.matrix` * Revised to use `strategy.matrix` * reference macro using fully-qualified module name * r|Microsoft|deislabs/akri * Remove unnecessary include * Add precision on arch being built * Consistent image naming and v1 * Remove `/v2/` references * Explantion of different broker implementations * Cleanup Workflows [IGNORE INTERMEDIATE BUILDS][SAME VERSION] (#155) * don't persist github creds and remove setup.sh from workflows * Update CHANGELOG for release v0.0.44 (#153) * Update CHANGELOG for release v0.0.44 * update release to better describe helm changes and supported flavors * add note about known k3s incompat * Pin CHANGELOG changes to the expected label * Fix CHANGELOG typo * Optimize build and test all supported versions (#157) * build once, test all supported versions * test step needs build step * fix copy-paste error * use startsWith instead of '==' to check kube-runtime * for download, specifying path introduces an unwanted directory * load the artifact containers into docker for Kubernetes * microk8s ctr images import is mysteriously failing with 'context deadline exceeded' * avoid k3s' crictl, add more testing versions for kubernetes & k3s * allow downgrades for kubernetes install, try to understand microk8s image import failure * swallow the return code of version checks * use persist-creds=false * disable microk8s 1.19 for now * microk8s 1.19 requires small sleep before it is ready for ctr requests * to ensure sequential running, we cannot skip the build-containers job (just its steps) (#159) * Update user guide with first release tag for Akri release chart (#160) * update user guide with first release tag for akri release chart * reference akri release chart instead of dev chart in docs * Remove pull_request_target triggers and minimize credential use [SAME VERSION] [IGNORE INTERMEDIATE BUILDS] (#161) * remove dockerhub credentials from workflows * add link to repo to intermediate containers * remove pull_request_target * only log into docker when pushing changes to ghcr * Typos * Added existing "Extensibility" preamble * Tidy * Replace "Extensibility" * attempt to merge extensibility.md i focused on the standalone to cut down on the size (the file is still 800+ lines) with a thought towards linking to some md files in http-extensibility branch for more details (on the rust grpc and golang grpc). * tighten up to reduce size combine device and discovery into one http server/pod, remove some comments from quoted code * provide a few more explanations * fix typo * Use matrix for build-http-broker-containers.yaml * Replace backup md file with extensibility-http-grpc.md Create document for deeper dive into gRPC broker/client code. This can be referenced from extensibility.md. * update extensibility.md to be consistent 'HTTP' * create links to http-extensibility branch * after walkthrough, make a couple fixes * use different curl container busyboxplus seems to have a dns problem that manifests on k3s * remove workflows for http-extensibility * remove new workflows in extensibility branch Co-authored-by: Didier Durand <durand.didier@gmail.com> Co-authored-by: bfjelds <bfjelds@microsoft.com> Co-authored-by: Jiri Appl <jiria@microsoft.com> Co-authored-by: Kate Goldenring <kate.goldenring@microsoft.com>
project-akri · Jan 7, 2021 · c7279e7 · c7279e7
1 parent 008db56
commit c7279e7
Show file tree

Hide file tree

Showing 35 changed files with 1,854 additions and 1 deletion.
diff --git a/Cross.toml b/Cross.toml
@@ -11,4 +11,4 @@ image = "ghcr.io/deislabs/akri/rust-crossbuild:x86_64-unknown-linux-gnu-0.1.16-0
 image = "ghcr.io/deislabs/akri/rust-crossbuild:armv7-unknown-linux-gnueabihf-0.1.16-0.0.7"
 
 [target.aarch64-unknown-linux-gnu]
-image = "ghcr.io/deislabs/akri/rust-crossbuild:aarch64-unknown-linux-gnu-0.1.16-0.0.7"
+image = "ghcr.io/deislabs/akri/rust-crossbuild:aarch64-unknown-linux-gnu-0.1.16-0.0.7"
diff --git a/agent/src/protocols/http/discovery_handler.rs b/agent/src/protocols/http/discovery_handler.rs
@@ -0,0 +1,79 @@
+use super::super::{DiscoveryHandler, DiscoveryResult};
+
+use akri_shared::akri::configuration::HTTPDiscoveryHandlerConfig;
+use async_trait::async_trait;
+use failure::Error;
+use reqwest::get;
+use std::collections::HashMap;
+
+const BROKER_NAME: &str = "AKRI_HTTP";
+const DEVICE_ENDPOINT: &str = "AKRI_HTTP_DEVICE_ENDPOINT";
+
+pub struct HTTPDiscoveryHandler {
+    discovery_handler_config: HTTPDiscoveryHandlerConfig,
+}
+impl HTTPDiscoveryHandler {
+    pub fn new(discovery_handler_config: &HTTPDiscoveryHandlerConfig) -> Self {
+        trace!("[http:new] Entered");
+        HTTPDiscoveryHandler {
+            discovery_handler_config: discovery_handler_config.clone(),
+        }
+    }
+}
+#[async_trait]
+
+impl DiscoveryHandler for HTTPDiscoveryHandler {
+    async fn discover(&self) -> Result<Vec<DiscoveryResult>, failure::Error> {
+        trace!("[http:discover] Entered");
+
+        let url = self.discovery_handler_config.discovery_endpoint.clone();
+        trace!("[http:discover] url: {}", &url);
+
+        match get(&url).await {
+            Ok(resp) => {
+                trace!(
+                    "[http:discover] Connected to discovery endpoint: {:?} => {:?}",
+                    &url,
+                    &resp
+                );
+
+                // Reponse is a newline separated list of devices (host:port) or empty
+                let device_list = &resp.text().await?;
+
+                let result = device_list
+                    .lines()
+                    .map(|endpoint| {
+                        trace!("[http:discover:map] Creating DiscoverResult: {}", endpoint);
+                        trace!(
+                            "[http:discover] props.inserting: {}, {}",
+                            BROKER_NAME,
+                            DEVICE_ENDPOINT,
+                        );
+                        let mut props = HashMap::new();
+                        props.insert(BROKER_NAME.to_string(), "http".to_string());
+                        props.insert(DEVICE_ENDPOINT.to_string(), endpoint.to_string());
+                        DiscoveryResult::new(endpoint, props, true)
+                    })
+                    .collect::<Vec<DiscoveryResult>>();
+                trace!("[protocol:http] Result: {:?}", &result);
+                Ok(result)
+            }
+            Err(err) => {
+                trace!(
+                    "[http:discover] Failed to connect to discovery endpoint: {}",
+                    &url
+                );
+                trace!("[http:discover] Error: {}", err);
+
+                Err(failure::format_err!(
+                    "Failed to connect to discovery endpoint results: {:?}",
+                    err
+                ))
+            }
+        }
+    }
+    fn are_shared(&self) -> Result<bool, Error> {
+        trace!("[http:are_shared] Entered");
+        Ok(true)
+    }
+}
diff --git a/agent/src/protocols/http/mod.rs b/agent/src/protocols/http/mod.rs
@@ -0,0 +1,2 @@
+mod discovery_handler;
+pub use self::discovery_handler::HTTPDiscoveryHandler;