[Extensibility] HTTP protocol (branch: http-extensibility) (#135)

* Initial commit * Working * Correct errors & revise for Device|Discovery v2 * Working * Should (!) work * Typo * Enrich * Refine * Fixes: #102 * Moves * Resolved * Revised for `dnsPolicy` * Dockerfiles * Added gRPC solution * Replaced * Kubernetes specs * Merge * Correct Kubernetes config filenames * Typo * Removed duplicate * Standalone broker exposes no ports * end-to-end-demo-gce.md: fixing 1 typo (#131) Hi, just removing the superfluous 's' Didier * link to codecov.io and use codecov action (#134) * Update end-to-end-demo.md (#112) * Update end-to-end-demo.md Hi there I extended the documentatio with the installation of linux-modules-extra because I had to add this package on Ubuntu 20.04 LTS (on a GCE instance): it seems that it doesn't come by default on distros / images aimed at backend servers. v4l loopback won't load if this other module is not present. Didier * Update end-to-end-demo.md fixing initial text according to revision notes in PR review * Update end-to-end-demo.md fixing style * Update end-to-end-demo.md simplifying text, based on PR review by Kate * rebase to finalize PR * Fix codecov for pull_request_target [SAME VERSION] (#138) * fix codecov for pull_request_target * add anti-pull-request-* conditional * Do not upload to codecov on PRs (#140) * end-to-end-demo-rpi4: fixing 1 typo (#143) Hello, Title says it all Didier * end-to-end demo : fixing 1 typo (#141) Hi there, The title says it all Didier * Minor e2e demo doc flow change (#139) * Minor flow change * Cleanup * Expand udev rule support (#130) * add support for SUBSYSTEMS, KERNELS, ATTRIBUTES, DRIVERS, and TAGS fields * fix formatting and add comments * add recursive search for subsystems * update udev configuration document with newly supported fields * check child device for field before looking up hierarchy * update version * document how to filter for only devices with devnodes when testing * reuse shared code * rust fmt fixes * add inline documentation for regex matching * Update end-to-end-demo.md (#113) Hi there, I added a comment regarding the importance of the version of v4l2loopback to avoid my personal initial mistake to others: I did not realize at start that the version was so important. So I went through this ticket with the dev: umlaeute/v4l2loopback#370 Didier Co-authored-by: bfjelds <bfjelds@microsoft.com> * Add sudo into udev configuration doc (#144) * Use arm7 * Reference arm7 * Reference branch on clone * `REPO="akri"` * Helm template for Agent already revised * Consistency * Use `./version.txt` for Akri version * Automate IMAGE updates * Build|Push gRPC images; Add'l consistency * Add Action to build HTTP Broker containers * I guess `http-extensibility` not `main` * Enable PRs to test changes to helm charts [SAME VERSION] (#146) * enable PRs to test changes to helm charts * handle manual job, fix push and release conditional * trigger e2e for helm changes * Updates references to `armv7` * Removed `shared` from HTTP Broker Dockerfiles * Correct version reference for image tags * Typo * Adds Golang Device|Discovery and alt-gRPC * s/println!/trace! * Consistency * Reflect relocation of apps * Simplify environment * Typo * Consistency on repo names for HTTP apps * Rename HTTP apps gRPC images * GitHub Actions to build HTTP apps containers * Configure broker security context (#148) * make udev broker security context configurable * update version * change group permissions to be default in example * fix comment with group id * Remove privileged security context from Akri Agent (#147) * Remove (redundant!?) aliasing * Remove healthchecks (`/healthz`) * Removed `akri-shared` dep * Simplified using `strategy.matrix` * Revised to use `strategy.matrix` * reference macro using fully-qualified module name * r|Microsoft|deislabs/akri * Remove unnecessary include * Add precision on arch being built * Consistent image naming and v1 * Remove `/v2/` references * Explantion of different broker implementations * Cleanup Workflows [IGNORE INTERMEDIATE BUILDS][SAME VERSION] (#155) * don't persist github creds and remove setup.sh from workflows * Update CHANGELOG for release v0.0.44 (#153) * Update CHANGELOG for release v0.0.44 * update release to better describe helm changes and supported flavors * add note about known k3s incompat * Pin CHANGELOG changes to the expected label * Fix CHANGELOG typo * Optimize build and test all supported versions (#157) * build once, test all supported versions * test step needs build step * fix copy-paste error * use startsWith instead of '==' to check kube-runtime * for download, specifying path introduces an unwanted directory * load the artifact containers into docker for Kubernetes * microk8s ctr images import is mysteriously failing with 'context deadline exceeded' * avoid k3s' crictl, add more testing versions for kubernetes & k3s * allow downgrades for kubernetes install, try to understand microk8s image import failure * swallow the return code of version checks * use persist-creds=false * disable microk8s 1.19 for now * microk8s 1.19 requires small sleep before it is ready for ctr requests * to ensure sequential running, we cannot skip the build-containers job (just its steps) (#159) * Update user guide with first release tag for Akri release chart (#160) * update user guide with first release tag for akri release chart * reference akri release chart instead of dev chart in docs * Remove pull_request_target triggers and minimize credential use [SAME VERSION] [IGNORE INTERMEDIATE BUILDS] (#161) * remove dockerhub credentials from workflows * add link to repo to intermediate containers * remove pull_request_target * only log into docker when pushing changes to ghcr * Typos * Added existing "Extensibility" preamble * Tidy * Replace "Extensibility" * attempt to merge extensibility.md i focused on the standalone to cut down on the size (the file is still 800+ lines) with a thought towards linking to some md files in http-extensibility branch for more details (on the rust grpc and golang grpc). * tighten up to reduce size combine device and discovery into one http server/pod, remove some comments from quoted code * provide a few more explanations * fix typo * Use matrix for build-http-broker-containers.yaml * Replace backup md file with extensibility-http-grpc.md Create document for deeper dive into gRPC broker/client code. This can be referenced from extensibility.md. * update extensibility.md to be consistent 'HTTP' * create links to http-extensibility branch * after walkthrough, make a couple fixes * use different curl container busyboxplus seems to have a dns problem that manifests on k3s * remove workflows for http-extensibility * remove new workflows in extensibility branch Co-authored-by: Didier Durand <durand.didier@gmail.com> Co-authored-by: bfjelds <bfjelds@microsoft.com> Co-authored-by: Jiri Appl <jiria@microsoft.com> Co-authored-by: Kate Goldenring <kate.goldenring@microsoft.com>
project-akri · Dec 9, 2020 · 241a39f · 241a39f
1 parent 3c5f721
commit 241a39f
Show file tree

Hide file tree

Showing 87 changed files with 3,675 additions and 1,218 deletions.
diff --git a/.github/actions/build-component-multi-arch/action.yml b/.github/actions/build-component-multi-arch/action.yml
@@ -20,13 +20,7 @@ inputs:
     description: Component prefix used by Makefile
     required: true
   github_event_name:
-    description: Specify the github event name (push, pull_request_target, release, etc)
-    required: true
-  dockerhub_username:
-    description: Dockerhub name
-    required: true
-  dockerhub_password:
-    description: Dockerhub password
+    description: Specify the github event name (push, pull_request, release, etc)
     required: true
 runs:
   using: 'node12'

diff --git a/.github/actions/build-component-multi-arch/main.js b/.github/actions/build-component-multi-arch/main.js
@@ -30,9 +30,6 @@ async function shell_cmd(cmd) {
         const latest_label = `latest${dev_suffix}`;
         console.log(`Use labels: versioned=${versioned_label} latest=${latest_label}`);
 
-        console.log(`Log into dockerhub to avoid throttled anonymous dockerhub pulls user=${core.getInput('dockerhub_username')}`);
-        await shell_cmd(`echo "${core.getInput('dockerhub_password')}" | docker login -u ${core.getInput('dockerhub_username')} --password-stdin`);
-
         console.log(`Login into Container Registry user=${core.getInput('container_registry_username')} repo=${core.getInput('container_registry_base_url')}`);
         await shell_cmd(`echo "${core.getInput('container_registry_password')}" | docker login -u ${core.getInput('container_registry_username')} --password-stdin ${core.getInput('container_registry_base_url')}`);
 

diff --git a/.github/actions/build-component-per-arch/action.yml b/.github/actions/build-component-per-arch/action.yml
@@ -26,7 +26,7 @@ inputs:
     description: Specify whether rust is being built
     required: true
   github_event_name:
-    description: Specify the github event name (push, pull_request_target, release, etc)
+    description: Specify the github event name (push, pull_request, release, etc)
     required: true
   github_ref:
     description: Specify the github ref
@@ -37,12 +37,6 @@ inputs:
   github_merged:
     description: Specify whether a PR has been merged
     required: true
-  dockerhub_username:
-    description: Dockerhub name
-    required: true
-  dockerhub_password:
-    description: Dockerhub password
-    required: true
 runs:
   using: 'node12'
   main: 'main.js'
diff --git a/.github/actions/build-component-per-arch/main.js b/.github/actions/build-component-per-arch/main.js
@@ -25,9 +25,6 @@ async function shell_cmd(cmd) {
     try {
         console.log(`Start main.js`)
 
-        console.log(`Log into dockerhub to avoid throttled anonymous dockerhub pulls user=${core.getInput('dockerhub_username')}`);
-        await shell_cmd(`echo "${core.getInput('dockerhub_password')}" | docker login -u ${core.getInput('dockerhub_username')} --password-stdin`);
-
         console.log(`Use multiarch/qemu-user-static to configure cross-plat`);
         await shell_cmd('docker run --rm --privileged multiarch/qemu-user-static --reset -p yes');
 
@@ -67,9 +64,6 @@ async function shell_cmd(cmd) {
         }
         console.log(`Makefile build target suffix: ${makefile_target_suffix}`)
 
-        console.log(`Login into Container Registry user=${core.getInput('container_registry_username')} repo=${core.getInput('container_registry_base_url')}`);
-        await shell_cmd(`echo "${core.getInput('container_registry_password')}" | docker login -u ${core.getInput('container_registry_username')} --password-stdin ${core.getInput('container_registry_base_url')}`);
-
         if (core.getInput('build_rust') == '1') {
             console.log(`Install Rust`)
             child_process.execSync(`curl https://sh.rustup.rs | sh -s -- -y --default-toolchain=1.41.0`);
@@ -102,6 +96,9 @@ async function shell_cmd(cmd) {
         await shell_cmd(`docker run ${image_name} find container-images-legal-notice.md | wc -l | grep -v 0`)
 
         if (push_containers == "1") {
+            console.log(`Login into Container Registry user=${core.getInput('container_registry_username')} repo=${core.getInput('container_registry_base_url')}`);
+            await shell_cmd(`echo "${core.getInput('container_registry_password')}" | docker login -u ${core.getInput('container_registry_username')} --password-stdin ${core.getInput('container_registry_base_url')}`);
+
             console.log(`Push the versioned container: make ${core.getInput('makefile_component_name')}-docker-per-arch-${makefile_target_suffix}`)
             process.env.LABEL_PREFIX = `${versioned_label}`
             await exec.exec(`make ${core.getInput('makefile_component_name')}-docker-per-arch-${makefile_target_suffix}`)

diff --git a/.github/actions/build-intermediate/action.yml b/.github/actions/build-intermediate/action.yml
@@ -23,7 +23,7 @@ inputs:
     description: Platform to build (amd64|arm64|arm32)
     required: true
   github_event_name:
-    description: Specify the github event name (push, pull_request_target, release, etc)
+    description: Specify the github event name (push, pull_request, release, etc)
     required: true
   github_ref:
     description: Specify the github ref
@@ -34,12 +34,6 @@ inputs:
   github_merged:
     description: Specify whether a PR has been merged
     required: true
-  dockerhub_username:
-    description: Dockerhub name
-    required: true
-  dockerhub_password:
-    description: Dockerhub password
-    required: true
 runs:
   using: 'node12'
   main: 'main.js'
diff --git a/.github/actions/build-intermediate/main.js b/.github/actions/build-intermediate/main.js
@@ -25,9 +25,6 @@ async function shell_cmd(cmd) {
     try {
         console.log(`Start main.js`)
 
-        console.log(`Log into dockerhub to avoid throttled anonymous dockerhub pulls user=${core.getInput('dockerhub_username')}`);
-        await shell_cmd(`echo "${core.getInput('dockerhub_password')}" | docker login -u ${core.getInput('dockerhub_username')} --password-stdin`);
-
         console.log(`Use multiarch/qemu-user-static to configure cross-plat`);
         child_process.execSync('docker run --rm --privileged multiarch/qemu-user-static --reset -p yes');
 
@@ -62,15 +59,15 @@ async function shell_cmd(cmd) {
         }
         console.log(`Makefile build target suffix: ${makefile_target_suffix}`)
 
-        console.log(`Login into Container Registry user=${core.getInput('container_registry_username')} repo=${core.getInput('container_registry_base_url')}`);
-        await shell_cmd(`echo "${core.getInput('container_registry_password')}" | docker login -u ${core.getInput('container_registry_username')} --password-stdin ${core.getInput('container_registry_base_url')}`);
-
         process.env.PREFIX = `${core.getInput('container_prefix')}`
 
         console.log(`Build the versioned container: make ${core.getInput('makefile_component_name')}-build-${makefile_target_suffix}`)
         await exec.exec(`make ${core.getInput('makefile_component_name')}-build-${makefile_target_suffix}`)
 
         if (push_containers == "1") {
+            console.log(`Login into Container Registry user=${core.getInput('container_registry_username')} repo=${core.getInput('container_registry_base_url')}`);
+            await shell_cmd(`echo "${core.getInput('container_registry_password')}" | docker login -u ${core.getInput('container_registry_username')} --password-stdin ${core.getInput('container_registry_base_url')}`);
+
             console.log(`Push the versioned container: make ${core.getInput('makefile_component_name')}-docker-per-arch-${makefile_target_suffix}`)
             await exec.exec(`make ${core.getInput('makefile_component_name')}-docker-per-arch-${makefile_target_suffix}`)
         } else {

diff --git a/.github/workflows/build-agent-container.yml b/.github/workflows/build-agent-container.yml
@@ -10,20 +10,6 @@ on:
     - build/containers/Dockerfile.agent
     - agent/**
     - shared/**
-    - build/setup.sh
-    - version.txt
-    - build/akri-containers.mk
-    - Makefile
-  pull_request_target:
-    branches: [ main ]
-    paths:
-    - .github/actions/build-component-per-arch/**
-    - .github/actions/build-component-multi-arch/**
-    - .github/workflows/build-agent-container.yml
-    - build/containers/Dockerfile.agent
-    - agent/**
-    - shared/**
-    - build/setup.sh
     - version.txt
     - build/akri-containers.mk
     - Makefile
@@ -36,7 +22,6 @@ on:
     - build/containers/Dockerfile.agent
     - agent/**
     - shared/**
-    - build/setup.sh
     - version.txt
     - build/akri-containers.mk
     - Makefile
@@ -51,11 +36,6 @@ env:
 jobs:
 
   per-arch:
-    # Run workflow pull_request if it is NOT a fork, as pull_request_target if it IS a fork
-    if: >-
-      ( github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.fork == true ) ||
-      ( github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false ) ||
-      ( !(startsWith(github.event_name, 'pull_request')) )
     runs-on: ubuntu-latest
     timeout-minutes: 40
     strategy:
@@ -66,18 +46,10 @@ jobs:
           - amd64
 
     steps:
-    - name: Checkout the merged commit from PR and base branch
-      uses: actions/checkout@v2
-      if: github.event_name == 'pull_request_target'
-      with:
-        # pull_request_target is run in the context of the base repository
-        # of the pull request, so the default ref is master branch and
-        # ref should be manually set to the head of the PR
-        ref: refs/pull/${{ github.event.pull_request.number }}/head
-
     - name: Checkout the head commit of the branch
-      if: ${{ github.event_name != 'pull_request_target' }}
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -100,8 +72,6 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}
         platform: ${{ matrix.arch }}
         build_rust: "1"
@@ -115,6 +85,8 @@ jobs:
     steps:
     - name: Checkout repo
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -134,6 +106,4 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}
diff --git a/.github/workflows/build-controller-container.yml b/.github/workflows/build-controller-container.yml
@@ -10,7 +10,6 @@ on:
     - build/containers/Dockerfile.controller
     - controller/**
     - shared/**
-    - build/setup.sh
     - version.txt
     - build/akri-containers.mk
     - Makefile
@@ -23,20 +22,6 @@ on:
     - build/containers/Dockerfile.controller
     - controller/**
     - shared/**
-    - build/setup.sh
-    - version.txt
-    - build/akri-containers.mk
-    - Makefile
-  pull_request_target:
-    branches: [ main ]
-    paths:
-    - .github/actions/build-component-per-arch/**
-    - .github/actions/build-component-multi-arch/**
-    - .github/workflows/build-controller-container.yml
-    - build/containers/Dockerfile.controller
-    - controller/**
-    - shared/**
-    - build/setup.sh
     - version.txt
     - build/akri-containers.mk
     - Makefile
@@ -51,11 +36,6 @@ env:
 jobs:
 
   per-arch:
-    # Run workflow pull_request if it is NOT a fork, as pull_request_target if it IS a fork
-    if: >-
-      ( github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.fork == true ) ||
-      ( github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false ) ||
-      ( !(startsWith(github.event_name, 'pull_request')) )
     runs-on: ubuntu-latest
     timeout-minutes: 40
     strategy:
@@ -65,18 +45,10 @@ jobs:
           - arm32v7
           - amd64
     steps:
-    - name: Checkout the merged commit from PR and base branch
-      uses: actions/checkout@v2
-      if: github.event_name == 'pull_request_target'
-      with:
-        # pull_request_target is run in the context of the base repository
-        # of the pull request, so the default ref is master branch and
-        # ref should be manually set to the head of the PR
-        ref: refs/pull/${{ github.event.pull_request.number }}/head
-
     - name: Checkout the head commit of the branch
-      if: ${{ github.event_name != 'pull_request_target' }}
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -99,8 +71,6 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}
         platform: ${{ matrix.arch }}
         build_rust: "1"
@@ -114,6 +84,8 @@ jobs:
     steps:
     - name: Checkout repo
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -133,6 +105,4 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}
diff --git a/.github/workflows/build-onvif-video-broker-container.yml b/.github/workflows/build-onvif-video-broker-container.yml
@@ -23,17 +23,6 @@ on:
     - version.txt
     - build/akri-containers.mk
     - Makefile
-  pull_request_target:
-    branches: [ main ]
-    paths:
-    - .github/actions/build-component-per-arch/**
-    - .github/actions/build-component-multi-arch/**
-    - .github/workflows/build-onvif-video-broker-container.yml
-    - build/containers/Dockerfile.onvif-video-broker
-    - samples/brokers/onvif-video-broker/**
-    - version.txt
-    - build/akri-containers.mk
-    - Makefile
   release:
     types:
       - published
@@ -45,11 +34,6 @@ env:
 jobs:
 
   per-arch:
-    # Run workflow pull_request if it is NOT a fork, as pull_request_target if it IS a fork
-    if: >-
-      ( github.event_name == 'pull_request_target' && github.event.pull_request.head.repo.fork == true ) ||
-      ( github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork == false ) ||
-      ( !(startsWith(github.event_name, 'pull_request')) )
     runs-on: ubuntu-latest
     timeout-minutes: 20
     strategy:
@@ -60,18 +44,10 @@ jobs:
           - amd64
 
     steps:
-    - name: Checkout the merged commit from PR and base branch
-      uses: actions/checkout@v2
-      if: github.event_name == 'pull_request_target'
-      with:
-        # pull_request_target is run in the context of the base repository
-        # of the pull request, so the default ref is master branch and
-        # ref should be manually set to the head of the PR
-        ref: refs/pull/${{ github.event.pull_request.number }}/head
-
     - name: Checkout the head commit of the branch
-      if: ${{ github.event_name != 'pull_request_target' }}
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -94,8 +70,6 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}
         platform: ${{ matrix.arch }}
         build_rust: "0"
@@ -109,6 +83,8 @@ jobs:
     steps:
     - name: Checkout repo
       uses: actions/checkout@v2
+      with:
+        persist-credentials: false
 
     - name: Prepare To Install
       uses: actions/setup-node@v1
@@ -128,6 +104,4 @@ jobs:
         container_registry_base_url: ghcr.io
         container_registry_username: ${{ secrets.crUsername }}
         container_registry_password: ${{ secrets.crPassword }}
-        dockerhub_username: ${{ secrets.dhUsername }}
-        dockerhub_password: ${{ secrets.dhPassword }}
         makefile_component_name: ${{ env.MAKEFILE_COMPONENT }}