refactor: minor lint fixes & improve docs for TLD other than default (#…

…111) ### 📘 Documentation - Include detail on providing `tld` in the configuration, allowing `eu` and other TLDs to be used. - Mention `tilt up` in the initial setup config as viable option. ### 🤖 CI & Build - Improve mage tasks with secret setup and tear down for better development support and troubleshooting. - Bump go version in release pipeline to use `1.21` as can include standard library security improvements. - Remove failing error condition on `mage job:rebuild` to better allow default setup without running local builds, such as just using the published docker image. This supports easier demo/test usage by support. ### 🔨 Refactor - Improve logging with error wrapping and remove deprecated Go `ioutil` usage. ### ⬆️ Dependencies - Bump tooling such as changie, release, trunk, more security scanners. - Other dependency bumps such as `golang.org/x/net`. fixes AB#526486
DelineaXPM · Sep 5, 2023 · 2d05721 · 2d05721
1 parent 59aba72
commit 2d05721
Show file tree

Hide file tree

Showing 30 changed files with 471 additions and 62 deletions.
diff --git a/.aqua/aqua.yaml b/.aqua/aqua.yaml
@@ -6,14 +6,14 @@ checksum:
   require_checksum: false
 registries:
   - type: standard
-    ref: v4.39.0 # renovate: depName=aquaproj/aqua-registry
+    ref: v4.44.1 # renovate: depName=aquaproj/aqua-registry
   - name: local
     type: local
     path: registry.yaml
 packages:
   - name: miniscruff/changie@v1.12.0
     tags: ['release']
-  - name: golang/go@go1.20.3
+  - name: golang/go@go1.21.0
     tags: ['first', 'release', 'test', 'scan', 'lint']
   - name: direnv/direnv@v2.32.2
   - name: magefile/mage@v1.14.0
@@ -32,7 +32,6 @@ packages:
   - name: kubernetes/kubectl
     version: v1.25.2
   - name: helm/helm@v3.11.1
-  - name: DelineaXPM/dsv-cli@v1.40.1
   - name: kubernetes/minikube@v1.31.2
     tags: ['ci']
   - name: stern/stern@v1.23.0
@@ -43,3 +42,5 @@ packages:
     version: v1.4.2
     registry: local
     tags: ['goinstall']
+  - name: DelineaXPM/dsv-cli@v1.40.5
+  - name: gitleaks/gitleaks@v8.18.0
diff --git a/.changes/unreleased/⬆️ Dependencies-20230817-161010.yaml b/.changes/unreleased/⬆️ Dependencies-20230817-161010.yaml
diff --git a/.changes/v1.2.1.md b/.changes/v1.2.1.md
@@ -0,0 +1,22 @@
+## v1.2.1 - 2023-09-05
+
+### 📘 Documentation
+
+- Include detail on providing `tld` in the configuration, allowing `eu` and other TLDs to be used.
+- Mention `tilt up` in the initial setup config as viable option.
+
+### 🤖 CI & Build
+
+- Improve mage tasks with secret setup and tear down for better development support and troubleshooting.
+- Bump go version in release pipeline to use `1.21` as can include standard library security improvements.
+- Remove failing error condition on `mage job:rebuild` to better allow default setup without running local builds, such as just using the published docker image.
+  This supports easier demo/test usage by support.
+
+### 🔨 Refactor
+
+- Improve logging with error wrapping and remove deprecated Go `ioutil` usage.
+
+### ⬆️ Dependencies
+
+- Bump tooling such as changie, release, trunk, more security scanners.
+- Other dependency bumps such as `golang.org/x/net`.
diff --git a/.envrc b/.envrc
@@ -2,6 +2,24 @@
 export DIRENV_WARN_TIMEOUT='10s'
 export DIRENV_LOG_FORMAT=""
 
+INFO_COLOR="\033[1;30;40m"
+RESET_COLOR="\033[0m"
+WARNING_COLOR="\033[33m"
+END_WARNING_COLOR="\033[0m"
+IMPORTANT_COLOR="\033[104;30m"
+WORKING_COLOR="\033[94m"
+BACKGROUND_GREEN="\033[94m"
+RESET_BACKGROUND="\033[0;49m"
+
+# variable for setting terminal output with blue background with black text
+BACKGROUND_BLUE="\033[44;30m"
+
+# variable for setting terminal output with light green background with black text and bold
+BACKGROUND_LIGHT_GREEN="\033[1;102;30m"
+
+# variable for setting terminal output with light yellow background and black text that is bold
+BACKGROUND_LIGHT_YELLOW="\033[1;103;30m"
+
 source_env "$HOME"
 source_env_if_exists ./env/.envrc
 export PATH="${GOPATH}/bin:${PATH}"
@@ -24,3 +42,25 @@ alias tilt="$(aqua which tilt)"
 alias mages="$(aqua which mage-select)"
 
 eval $(minikube docker-env)
+
+if [[ -f ".env" ]]; then
+  dotenv_if_exists ".env"
+else
+  printf "${BACKGROUND_LIGHT_YELLOW}LOCAL ENVIRONMENT CONFIGURATION${RESET_COLOR}\n"
+  printf "${WARNING_COLOR}👉 .env not found\n"
+  printf "${WARNING_COLOR}\tSuggested fixes:\n"
+  printf "${WARNING_COLOR}\t\t1. create ${BACKGROUND_LIGHT_GREEN}.env${RESET_COLOR}\n"
+  printf "${WARNING_COLOR}\t\t2. ensure ${RESET_COLOR}${BACKGROUND_LIGHT_GREEN}.env${RESET_COLOR}${WARNING_COLOR} contains the following variables: \n\n"
+fi
+# Check if DSV_PROFILE_NAME is set
+if [[ -z "${DSV_PROFILE_NAME}" ]]; then
+  printf "\t\t\t- ${WARNING_COLOR}👉 DSV_PROFILE_NAME not set${RESET_COLOR}\n"
+else
+  printf "✔️ ${INFO_COLOR}DSV_PROFILE_NAME set${RESET_COLOR}\n"
+fi
+# Check if DSV_TENANT_NAME is set
+if [[ -z "${DSV_TENANT_NAME}" ]]; then
+  printf "\t\t\t- ${WARNING_COLOR}👉 DSV_TENANT_NAME not set${RESET_COLOR}\n"
+else
+  printf "✔️ ${INFO_COLOR}DSV_TENANT_NAME set${RESET_COLOR}\n"
+fi
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
@@ -2,10 +2,10 @@
 name: lint
 on:
   # Run CI against all pushes (direct commits, also merged PRs), Pull Requests
-  push:
-    branches:
-      - main
-      - stable-*
+  # push:
+  #   branches:
+  #     - main
+  #     - stable-*
   pull_request:
   # Run CI once per day (at 06:00 UTC)
   # This ensures that even if there haven't been commits that we are still testing against latest version of ansible-test for each ansible-base version

diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -20,7 +20,9 @@ jobs:
       - name: Setup Golang with cache
         uses: magnetikonline/action-golang-cache@797f193169d3c8ba6f60d90f50ecdadd2583fbd8 # tag=v3
         with:
-          go-version-file: go.mod
+          go-version: ~1.21
+        # with:
+        #   go-version-file: go.mod
 
       - uses: aquaproj/aqua-installer@61e2563dfe7674cbf74fe6ec212e444198a3bb00 # tag=v2.0.2
         with:

diff --git a/.trunk/.gitignore b/.trunk/.gitignore
@@ -2,7 +2,7 @@
 *logs
 *actions
 *notifications
+*tools
 plugins
 user_trunk.yaml
 user.yaml
-tools
diff --git a/.trunk/trunk.yaml b/.trunk/trunk.yaml
@@ -2,7 +2,7 @@ version: 0.1
 plugins:
   sources:
     - id: trunk
-      ref: v1.0.0
+      ref: v1.2.1
       uri: https://github.com/trunk-io/plugins
 actions:
   enabled:
@@ -30,7 +30,7 @@ actions:
     - id: helm-lint
       # runtime: go
       description: run helm lint on pre-push
-      run: 'helm lint --quiet --strict charts/*'
+      run: 'helm lint --quiet charts/*'
       triggers:
         - git_hooks: [pre-push]
 runtimes:
@@ -39,7 +39,7 @@ runtimes:
     - node@18.12.1
     - python@3.10.8
 cli:
-  version: 1.13.0
+  version: 1.15.0
 lint:
   threshold:
     - linters: [gitleaks]
@@ -48,16 +48,22 @@ lint:
     - cspell
     - gofmt
   enabled:
+    - checkov@2.4.9
+    - gokart@0.5.1
+    - osv-scanner@1.3.6
+    - terrascan@1.18.3
+    - trivy@0.44.1
+    - trufflehog@3.54.3
     - gofumpt@0.5.0
-    - renovate@36.49.0
+    - renovate@36.81.0
     - golangci-lint@SYSTEM
     - git-diff-check
     - taplo@0.8.1
-    - markdownlint@0.35.0
-    - prettier@3.0.2
+    - markdownlint@0.36.0
+    - prettier@3.0.3
     - actionlint@1.6.25
     - hadolint@2.12.0
-    - gitleaks@8.17.0
+    - gitleaks@8.18.0
     - shellcheck@0.9.0
     - shfmt@3.6.0
     - yamllint@1.32.0
@@ -69,7 +75,7 @@ lint:
     - actionlint@1.6.25
     - gitleaks@8.15.3
     - hadolint@2.12.0
-    - markdownlint@0.35.0
+    - markdownlint@0.36.0
     - shellcheck@0.9.0
     - shfmt@3.6.0
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -6,6 +6,22 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
 and is generated by [Changie](https://github.com/miniscruff/changie).
 
+## v1.2.1 - 2023-09-05
+
+### 🤖 CI & Build
+
+- Improve mage tasks with secret setup and tear down for better development support and troubleshooting.
+
+- Bump go version in release pipeline to use `1.21` as can include standard library security improvements.
+
+### 🔨 Refactor
+
+- Improve logging with error wrapping and remove deprecated Go `ioutil` usage.
+
+### ⬆️ Dependencies
+
+- Bump changie installation, goreleaser.
+
 ## v1.2.0 - 2023-04-27
 
 ### 🤖 CI & Build

diff --git a/Tiltfile b/Tiltfile
@@ -175,7 +175,7 @@ local_resource(
   deps=['.cache/'],
   resource_deps=[
     "job:init",
-    "job:rebuildimages"
+    # "job:rebuildimages"
   ],
   auto_init=False,
   labels=["deploy"],
@@ -217,4 +217,4 @@ local_resource(
   labels=["setup"],
 )
 
-# k8s_resource('injector', resource_deps='minikube:init', pod_readiness='ignore')
+# k8s_resource('injector', resource_deps='minikube:init', pod_readiness='ignore')
diff --git a/charts/dsv-injector/Chart.yaml b/charts/dsv-injector/Chart.yaml
@@ -8,7 +8,7 @@ description: |
   NAMESPACE='dsv'
   CREDENTIALS_JSON_FILE='.cache/credentials.json' # replace with your credentials file
   IMAGE_REPOSITORY='docker.io/delineaxpm/dsv-k8s'
-  IMAGE_TAG='v1.2.0' # use latest for test, but pin to a specific version for production
+  IMAGE_TAG='v1.2.1' # use latest for test, but pin to a specific version for production
 
   helm install \
       --namespace $NAMESPACE \
@@ -29,7 +29,7 @@ keywords:
   - secrets
   - vault
 type: application
-version: v1.2.0
+version: v1.2.1
 appVersion: latest
 maintainers:
   - name: Sheldon Hull

diff --git a/charts/dsv-injector/values.yaml b/charts/dsv-injector/values.yaml
@@ -29,7 +29,7 @@ image:
   # -- Overrides the image tag whose default is the chart appVersion.
   # Local Testing: Use `latest`.
   # @default -- the current app version/chart version
-  tag: v1.2.0
+  tag: v1.2.1
   # -- Entrypoint is the path to the binary. Since the container image could contain multiple binaries, this makes sure it's correctly mapped to the binary.
   entrypoint: /app/dsv-injector
 

diff --git a/charts/dsv-syncer/Chart.yaml b/charts/dsv-syncer/Chart.yaml
@@ -7,7 +7,7 @@ description: |
   ```shell
   NAMESPACE='dsv'
   IMAGE_REPOSITORY='docker.io/delineaxpm/dsv-k8s'
-  IMAGE_TAG='v1.2.0' # use latest for test, but pin to a specific version for production
+  IMAGE_TAG='v1.2.1' # use latest for test, but pin to a specific version for production
 
   helm install \
       --namespace $NAMESPACE \
@@ -26,7 +26,7 @@ keywords:
   - secrets
   - vault
 type: application
-version: v1.2.0
+version: v1.2.1
 appVersion: latest
 maintainers:
   - name: Sheldon Hull

diff --git a/charts/dsv-syncer/templates/syncer-cronjob.yaml b/charts/dsv-syncer/templates/syncer-cronjob.yaml
@@ -43,4 +43,4 @@ spec:
           volumes:
             - name: credentials
               secret:
-                secretName: {{ .Values.dsvInjectorCredentialsSecretName }}
+                secretName: {{ .Values.dsvInjectorCredentialsSecretName }}
diff --git a/charts/dsv-syncer/values.yaml b/charts/dsv-syncer/values.yaml
@@ -18,7 +18,7 @@ image:
   # -- Overrides the image tag whose default is the chart appVersion.
   # Local Testing: Use `latest`.
   # @default -- the current app version/chart version
-  tag: v1.2.0
+  tag: v1.2.1
   # -- Entrypoint is the path to the binary. Since the container image could contain multiple binaries, this makes sure it's correctly mapped to the binary.
   entrypoint: /app/dsv-syncer
 

diff --git a/docs/configure.md b/docs/configure.md
@@ -3,7 +3,24 @@
 This focuses on the DSV configuration required to use with Kubernetes.
 This applies to both local testing Kubernetes and your own seperate cluster.
 
-## JSON Credentials for Helm Install
+## Help Getting Started
+
+Run `mage dsv:setupdsv` to create the required DSV configuration for testing.
+This requires you to have already run `dsv init` in the project and runs against the profile you specified in `.env`.
+You should ensure `direnv allow` has been run and the `.env` file is loaded.
+Your `zsh` terminal should warn you if you didn't create the `.env` file.
+
+The order:
+
+- `mage dsv:setupdsv`
+- `mage dsv:createsecret`
+- `mage dsv:convertClientToCredentials`
+
+To tear down and recreate with new secret, just run `mage dsv:destroy`
+
+## Manually Creating (Prior Method Before Automation)
+
+### JSON Credentials for Helm Install
 
 The configuration requires a JSON formatted list of Client Credential and Tenant mappings.
 
@@ -34,7 +51,7 @@ You can place your temporary config in `.cache/credentials.json` as this is igno
 
 ### Update Manifests
 
-This would be referenced by a Kubernetes secret with annontations like:
+This would be referenced by a Kubernetes secret with annotations like:
 
 ```yaml
 ---
@@ -44,7 +61,7 @@ metadata:
   name: user-domain-pass
   annotations:
     dsv.delinea.com/credentials: app1
-    dsv.delinea.com/set-secret: 'k8s:sync:test'
+    dsv.delinea.com/set-secret: 'tests:dsv-k8s'
 ```
 
 If using the provided examples, you can edit: `.cache/manifests` and adjust the secrets to map.
@@ -60,7 +77,7 @@ Create the role that will allow creating a client for programmatic access
 
 ```shell
 dsv role create --name 'k8s' --desc 'test profile for k8s'
-dsv secret create --path 'k8s:sync:test' --data '{"password": "admin","username": "admin"}'
+dsv secret create --path 'tests:dsv-k8s' --data '{"password": "admin","username": "admin"}'
 ```
 
 Create a policy that allows the local user to read the secret, modify this to the correct user/group mapping:

diff --git a/docs/setup-developer.md b/docs/setup-developer.md
@@ -15,6 +15,11 @@
 
 As always, the source of truth is `mage` so if the task names in the doc don't work, check the CLI for the proper commands.
 
+## Optional
+
+If you are using codespaces, most of the tooling should be ready out of the box as long as you open `zsh` terminal.
+Run `tilt up` and then you can invoke much of this (including watch the logs stream) from the terminal.
+
 ## Reference
 
 - Optional: [devcontainer/codespaces](devcontainer.md)

diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
@@ -1,5 +1,22 @@
 # Troubleshooting
 
+## Supporting Alternative TLD
+
+If you are using an alternative TLD, such as `https://{mytenant}.secretsvaultcloud.eu`, try adding the tld to the credentials file.
+
+```json
+{
+  "default": {
+    "credentials": {
+      "clientId": "",
+      "clientSecret": ""
+    },
+    "tenant": "mytenant",
+    "tld": "eu"
+  }
+}
+```
+
 ## Obtaining Logs
 
 For both customers and development, stern allows easier debugging by providing a stream of the logs for both syncer & injector in one workflow.