Skip to content

Commit

Permalink
Merge branch 'main' into hotfix-return-types-in-funcs
Browse files Browse the repository at this point in the history
  • Loading branch information
samayer12 authored Dec 13, 2024
2 parents a22e053 + 34f4e50 commit f85718d
Show file tree
Hide file tree
Showing 65 changed files with 2,149 additions and 731 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,17 +44,17 @@ jobs:

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@aa578102511db1f4524ed59b8cc2bae4f6e88195 # v3.27.6
uses: github/codeql-action/init@8a93837afdf1873301a68d777844b43e98cd4313 # v3.27.8
with:
languages: ${{ matrix.language }}

# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@aa578102511db1f4524ed59b8cc2bae4f6e88195 # v3.27.6
uses: github/codeql-action/autobuild@8a93837afdf1873301a68d777844b43e98cd4313 # v3.27.8

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@aa578102511db1f4524ed59b8cc2bae4f6e88195 # v3.27.6
uses: github/codeql-action/analyze@8a93837afdf1873301a68d777844b43e98cd4313 # v3.27.8
with:
category: "/language:${{matrix.language}}"

95 changes: 95 additions & 0 deletions .github/workflows/deploy-helm.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: Deploy Test - Helm

permissions: read-all
on:
workflow_dispatch:
push:
branches: ["main"]
pull_request:
branches: ["main"]

jobs:
helm:
name: deploy test
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
with:
egress-policy: audit

- name: Set up Kubernetes
uses: azure/setup-kubectl@3e0aec4d80787158d308d7b364cb1b702e7feb7f # v4.0.0
with:
version: 'latest'

- name: "install k3d"
run: "curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash"
shell: bash

- name: clone pepr
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: defenseunicorns/pepr
path: pepr

- name: setup node
uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af # v4.1.0
with:
node-version: 22
cache: "npm"
cache-dependency-path: pepr

- name: "set env: PEPR"
run: echo "PEPR=${GITHUB_WORKSPACE}/pepr" >> "$GITHUB_ENV"

- name: install pepr deps
run: |
cd "$PEPR"
npm ci
- name: build pepr package + image
run: |
cd "$PEPR"
npm run build:image
- name: "set env: MOD_NAME"
run: |
echo "MOD_NAME=pepr-test-helm" >> "$GITHUB_ENV"
- name: "set env: MOD_PATH"
run: |
echo "MOD_PATH=${PEPR}/${MOD_NAME}" >> "$GITHUB_ENV"
- name: init pepr module
run: |
cd "$PEPR"
npx pepr init --name "$MOD_NAME" --description "$MOD_NAME" --skip-post-init --confirm
sed -i 's/uuid": ".*",/uuid": "'$MOD_NAME'",/g' "$MOD_PATH/package.json"
- name: build pepr module
run: |
cd "$MOD_PATH"
npm install "${PEPR}/pepr-0.0.0-development.tgz"
npx pepr build --custom-image pepr:dev
- name: "set env: CLUSTER"
run: echo "CLUSTER=$MOD_NAME" >> "$GITHUB_ENV"

- name: prep test cluster
run: |
k3d cluster create "$CLUSTER"
k3d image import pepr:dev --cluster "$CLUSTER"
- name: "set env: KUBECONFIG"
run: echo "KUBECONFIG=$(k3d kubeconfig write "$CLUSTER")" >> "$GITHUB_ENV"

- name: deploy pepr module
run: |
cd "$MOD_PATH"
helm install "$MOD_NAME" "./dist/${MOD_NAME}-chart" --kubeconfig "$KUBECONFIG"
- name: Check Deployment Readiness
timeout-minutes: 5
run: |
${PEPR}/.github/workflows/scripts/check-deployment-readiness.sh pepr-$MOD_NAME
110 changes: 110 additions & 0 deletions .github/workflows/deploy-zarf.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
name: Deploy Test - Zarf

permissions: read-all
on:
workflow_dispatch:
push:
branches: ["main"]
pull_request:
branches: ["main"]

env:
MOD_NAME: pepr-test-zarf

jobs:
zarf:
name: deploy test
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
with:
egress-policy: audit

- name: Set up Kubernetes
uses: azure/setup-kubectl@3e0aec4d80787158d308d7b364cb1b702e7feb7f # v4.0.0
with:
version: 'latest'

- name: "install k3d"
run: "curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash"
shell: bash

- name: Install The Latest Release Version of Zarf
uses: defenseunicorns/setup-zarf@10e539efed02f75ec39eb8823e22a5c795f492ae #v1.0.1
with:
download-init-package: true

- name: clone pepr
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
repository: defenseunicorns/pepr
path: pepr

- name: setup node
uses: actions/setup-node@39370e3970a6d050c480ffad4ff0ed4d3fdee5af # v4.1.0
with:
node-version: 22
cache: "npm"
cache-dependency-path: pepr

- name: "set env: PEPR"
run: echo "PEPR=${GITHUB_WORKSPACE}/pepr" >> "$GITHUB_ENV"

- name: Install Pepr Dependencies
run: |
cd "$PEPR"
npm ci
- name: Build Pepr Package + Image
run: |
cd "$PEPR"
npm run build:image
- name: "set env: MOD_PATH"
run: |
echo "MOD_PATH=${PEPR}/${MOD_NAME}" >> "$GITHUB_ENV"
- name: Init Pepr Module
run: |
cd "$PEPR"
npx pepr init --name "$MOD_NAME" --description "$MOD_NAME" --skip-post-init --confirm
sed -i 's/uuid": ".*",/uuid": "'$MOD_NAME'",/g' "$MOD_PATH/package.json"
- name: Build Pepr Module
run: |
cd "$MOD_PATH"
npm install "${PEPR}/pepr-0.0.0-development.tgz"
npx pepr build --custom-image pepr:dev
- name: "set env: CLUSTER"
run: echo "CLUSTER=$MOD_NAME" >> "$GITHUB_ENV"

- name: Prepare Test Cluster
run: |
k3d cluster create "$CLUSTER"
k3d image import pepr:dev --cluster "$CLUSTER"
- name: "set env: KUBECONFIG"
run: echo "KUBECONFIG=$(k3d kubeconfig write "$CLUSTER")" >> "$GITHUB_ENV"


- name: Initialize Zarf
run: |
cd "$MOD_PATH"
zarf init --confirm
- name: Package Pepr Module with Zarf
run: |
cd "$MOD_PATH"
zarf package create --confirm "dist/"
- name: Deploy Pepr Module with Zarf
run: |
cd "$MOD_PATH"
zarf package deploy --confirm zarf-package-pepr-pepr-test-zarf-amd64-0.0.1.tar.zst
- name: Check Deployment Readiness
timeout-minutes: 5
run: |
${PEPR}/.github/workflows/scripts/check-deployment-readiness.sh pepr-$MOD_NAME
2 changes: 1 addition & 1 deletion .github/workflows/node.js.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
- run: npm ci
- run: npm run test:unit
- name: Upload coverage to Codecov
uses: codecov/codecov-action@015f24e6818733317a2da2edd6290ab26238649a # v5.0.7
uses: codecov/codecov-action@7f8b4b4bde536c465e797be725718b88c5d95e0e # v5.1.1
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_ORG_TOKEN }}
journey:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/scorecard.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,6 @@ jobs:

# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@aa578102511db1f4524ed59b8cc2bae4f6e88195 # v2.2.4
uses: github/codeql-action/upload-sarif@8a93837afdf1873301a68d777844b43e98cd4313 # v2.2.4
with:
sarif_file: results.sarif
48 changes: 48 additions & 0 deletions .github/workflows/scripts/check-deployment-readiness.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash

set -euo pipefail

check_deployment_readiness() {
local deployment_name=$1
local namespace=$2
local expected_ready_replicas=$3
local timeout=${4:-300} # Timeout in seconds (default: 5 minutes)
local interval=${5:-5} # Interval between checks in seconds
local elapsed=0

echo "$(date '+%Y-%m-%d %H:%M:%S') - Checking readiness for deployment '$deployment_name' in namespace '$namespace'..."
echo "$(date '+%Y-%m-%d %H:%M:%S') - Using timeout: ${timeout}s, interval: ${interval}s"

while [ "$elapsed" -lt "$timeout" ]; do
ready_replicas=$(kubectl get deploy "$deployment_name" -n "$namespace" -o jsonpath='{.status.readyReplicas}' 2>/dev/null || echo "0")
ready_replicas=${ready_replicas:-0} # Default to 0 if null

if [ "$ready_replicas" == "$expected_ready_replicas" ]; then
echo "$(date '+%Y-%m-%d %H:%M:%S') - Deployment '$deployment_name' is ready with $ready_replicas replicas."
return 0
fi

echo "$(date '+%Y-%m-%d %H:%M:%S') - Waiting for deployment '$deployment_name' to be ready. Ready replicas: ${ready_replicas:-0}/${expected_ready_replicas}."
kubectl get deploy -n "$namespace"
sleep "$interval"
elapsed=$((elapsed + interval))
done

echo "$(date '+%Y-%m-%d %H:%M:%S') - Timeout reached while waiting for deployment '$deployment_name' to be ready."
return 1
}

# Define success criteria
expected_pepr_replicas=2
expected_watcher_replicas=1
module_name=${1:-}
namespace=${2:-pepr-system} # Default to 'pepr-system' if null

if [ -z "$module_name" ]; then
echo "Error: Module name MUST be provided as the first argument."
exit 1
fi

check_deployment_readiness "$module_name" "$namespace" $expected_pepr_replicas || exit 1 # Check readiness for the first deployment

check_deployment_readiness "$module_name-watcher" "$namespace" $expected_watcher_replicas || exit 1 # Check readiness for the watcher deployment
2 changes: 1 addition & 1 deletion .github/workflows/secret-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,6 @@ jobs:
with:
fetch-depth: 0
- name: Default Secret Scanning
uses: trufflesecurity/trufflehog@35943b41905eb1195f021955da17c233ed555e24 # main
uses: trufflesecurity/trufflehog@6ceb49097f21249369f015c4d571173e9252f04d # main
with:
extra_args: --debug --no-verification # Warn on potential violations
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ node_modules/
stats.html
.vscode
insecure-tls*
pepr-test-module
pepr-test-*
pepr-upgrade-test
*.tar
*.tgz
Expand Down
35 changes: 35 additions & 0 deletions docs/090_roadmap/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,38 @@ _2024 Roadmap_
- Load test Pepr/KFC to identify bottlenecks and areas of improvement.
- Ensure that Pepr/KFC can handle a large number of resources and events over a sustained period of time (nightly).

_2025 Roadmap_
## Phase 1: Code Quality - Experimentation

- **Q1**:
- **Turn on eslint enforcement and configure settings and see no warnings**:
- Eliminate circular dependencies, complexity, return statements, etc.
- **Metric and Performance Baselining**:
- Establish a baseline for performance and resource utilization metrics. Use this data to make informed decisions about the direction of the project in terms of Deno2
- **OTEL Preparation**:
- Come up with a plan to implement Open Telemetry. Specifically distributed tracing, metrics, logs and events. Use this data to make debugging easier from a UDS Core prespective. There will be documentation work on how to use an OTEL collector with a Pepr Module.
- **Nightly Release**:
- Establish a nightly release process. This will help us to catch bugs early and ensure that the project is always in a releasable state.

## Phase 2: Durable Storage for Metrics and Performance Tests / Transactional Pepr Store

- **Q2**:
- **Professional Dashboard displaying metrics and performance tests originating from CI**:
- **Determine if a Transactional PeprStore makes sense**:
- Sus out details involved with having a transactional Pepr Store. What are the implications of this? What are the benefits? What are the drawbacks? What are the use-cases? What are the technologies that can be used to implement this?
- **Experimentation with Deno2**:
- Experiment with Deno2 through Dash Days and see if it can be used in the project. Look into the performance improvements and new features that Deno2 brings to the table.


## Phase 3: TBD

- **Q3**:
- **Deno2 Implementation**:
- If determined to be advisable, move forward with migrating the project to Deno2 (starting with the kubernetes-fluent-client..?). This phase will focus on adapting the codebase, conducting extensive testing, and creating comprehensive documentation to ensure a seamless transition.
- **Transactional PeprStore Implementation**:
- Begin integrating transactional functionality into PeprStore. The implementation will emphasize robust testing and clear documentation to support fast and reliable data operations in a transactional manner.

## Phase 4: TDB

- **Q4**:

Loading

0 comments on commit f85718d

Please sign in to comment.