-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: sanitizing store keys - store path v2 #918
Conversation
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This definitely seems functional, I do have a few comments on things that might be painful from the core side with this path:
- In the past I have relied on a
k describe
of the peprstore resource(s) to see what my currently store k/v pairs are. With the keys base64 encoded that's no longer the most transparent experience. I think that's okay since it's not dissimilar from the experience viewing something like a secret, but it may be nice to consider a way to view the store easily (whether that's just a documented kubectl command, k9s plugin, or a newnpx pepr <store-viewer>
type thing? - The migration script is definitely usable, and we could probably find a way to make the upgrade completely seamless (i.e. not require any end user to do more than just a
zarf deploy
). I don't know how much effort this would be on this side of things but it would definitely be a more pleasant experience if pepr completely handled the migration for us. When I was playing around with this issue early on I tried some basic migration tactics in our code like when callinggetItem
first check if the old format key existed (non-base64 in this case). If it did, perform a "migration" by deleting and setting the new key, before using the value. Unsure if something similar would fit well in pepr. - Related to ^ but not directly tied to these changes, it might be nice to explore adding a
Store.migrate(oldKey, newKey)
type function for end users if they change their code in what keys they use. Might just be a lack of knowledge on my part but in exploring changing the store keys it didn't feel like the most seamless experience.
2 & 3 are good ideas too. I just have a lot of open questions around how to guarantee we will get the capability name right without user intervention in the code as it is today because the capability name is tightly coupled to the module. However, i definitely think we can come up with something together to see how we can make this upgrade a seamless experience without the user even knowing it happened. Lets chat when you are freed up on current work and we will come up with a game plan. For now I am going to put this PR on old and do more investigation and research. |
Now thinking back on this, what if we:
What are you thoughts on that? We could also roll out the store view feature at the same time |
## Description Extends existing Store functionality to v2 to enable storing keys like URLs, and provides a seamless hand-off migration path for users while introducing additional tooling to view the store resource. - [x] How to enable the store to set special characters as keys - [x] Address how to view the Store CR key/values in a user friendly way - [x] Hands off end user migration path is decided on - [ ] Stakeholder feedback - [ ] UDS Core processes `UDSExcemptions` on startup so we want to leave that process unbothered (We think this should probably be fine but need to test. Other solutions pre-starup would involve `initContainer`). If this were a problem i think it could be solved by moving [`startExemptionWatch`](https://github.com/defenseunicorns/uds-core/blob/2e1c53e939b99794c8e6994f20282974bd139917/pepr.ts#L17) to the [`Store.onReady`](https://github.com/defenseunicorns/uds-core/blob/2e1c53e939b99794c8e6994f20282974bd139917/pepr.ts#L38) callback. IMO this checkbox is part of stakeholder feedback. Also, i don't think this Store change will affect the startExemptionWatch function bc they are not using the actual Pepr Store in that instance but populating a map[string]interface{} type of store object. ## Related Issue Fixes #924 <!-- or --> Relates to #918 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [x] Other (security config, docs update, etc) ## Checklist before merging - [x] Test, docs, adr added or updated as needed - [x] [Contributor Guide Steps](https://docs.pepr.dev/main/contribute/#submitting-a-pull-request) followed --------- Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com> Co-authored-by: Barrett <81570928+btlghrants@users.noreply.github.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
New dependencies detected. Learn more about Socket for GitHub ↗︎
|
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not much left to say after the in-depth, in-person walk-through -- seems to do what it says on the tin. 👍
this runner is bjorked |
Signed-off-by: Case Wylie <cmwylie19@defenseunicorns.com>
The GitHub Runners are failing for node/npm Run npm ci
npm ci
shell: /usr/bin/bash -e {0}
npm error Exit handler never called!
npm error This is an error with npm itself. Please report this error at:
npm error <https://github.com/npm/cli/issues>
npm error A complete log of this run can be found in: /home/runner/.npm/_logs/[2](https://github.com/defenseunicorns/pepr/actions/runs/9980001407/job/27580881374#step:4:2)024-07-17T19_05_0[3](https://github.com/defenseunicorns/pepr/actions/runs/9980001407/job/27580881374#step:4:3)_312Z-debug-0.log Relevant Issues: Relevant news:
We need to talk about changing the matrix test from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple minor suggestions & one Q.
Co-authored-by: Barrett <81570928+btlghrants@users.noreply.github.com>
Co-authored-by: Barrett <81570928+btlghrants@users.noreply.github.com>
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---|---|---| | [defenseunicorns/uds-common](https://togithub.com/defenseunicorns/uds-common) | | minor | `v0.8.0` -> `v0.11.1` | [![age](https://developer.mend.io/api/mc/badges/age/github-tags/defenseunicorns%2fuds-common/v0.11.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/github-tags/defenseunicorns%2fuds-common/v0.11.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/github-tags/defenseunicorns%2fuds-common/v0.8.0/v0.11.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/github-tags/defenseunicorns%2fuds-common/v0.8.0/v0.11.1?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [pepr](https://togithub.com/defenseunicorns/pepr) | dependencies | minor | [`0.32.7` -> `0.33.0`](https://renovatebot.com/diffs/npm/pepr/0.32.7/0.33.0) | [![age](https://developer.mend.io/api/mc/badges/age/npm/pepr/0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/pepr/0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/pepr/0.32.7/0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/pepr/0.32.7/0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [registry1.dso.mil/ironbank/opensource/defenseunicorns/pepr/controller](https://togithub.com/defenseunicorns/pepr) ([source](https://repo1.dso.mil/dsop/opensource/defenseunicorns/pepr/controller)) | | minor | `v0.32.7` -> `v0.33.0` | [![age](https://developer.mend.io/api/mc/badges/age/docker/registry1.dso.mil%2fironbank%2fopensource%2fdefenseunicorns%2fpepr%2fcontroller/v0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/docker/registry1.dso.mil%2fironbank%2fopensource%2fdefenseunicorns%2fpepr%2fcontroller/v0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/docker/registry1.dso.mil%2fironbank%2fopensource%2fdefenseunicorns%2fpepr%2fcontroller/v0.32.7/v0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/docker/registry1.dso.mil%2fironbank%2fopensource%2fdefenseunicorns%2fpepr%2fcontroller/v0.32.7/v0.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/) | | [ts-jest](https://kulshekhar.github.io/ts-jest) ([source](https://togithub.com/kulshekhar/ts-jest)) | devDependencies | patch | [`29.2.2` -> `29.2.4`](https://renovatebot.com/diffs/npm/ts-jest/29.2.2/29.2.4) | [![age](https://developer.mend.io/api/mc/badges/age/npm/ts-jest/29.2.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://developer.mend.io/api/mc/badges/adoption/npm/ts-jest/29.2.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://developer.mend.io/api/mc/badges/compatibility/npm/ts-jest/29.2.2/29.2.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://developer.mend.io/api/mc/badges/confidence/npm/ts-jest/29.2.2/29.2.4?slim=true)](https://docs.renovatebot.com/merge-confidence/) | --- ### Release Notes <details> <summary>defenseunicorns/uds-common (defenseunicorns/uds-common)</summary> ### [`v0.11.1`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.11.1) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.11.0...v0.11.1) ##### Bug Fixes - renovate ghcr host docker type ([#​201](https://togithub.com/defenseunicorns/uds-common/issues/201)) ([9c298e0](https://togithub.com/defenseunicorns/uds-common/commit/9c298e08417ce928dbbf4356c23182f8b1a62ffb)) - renovate typo token/password ([#​202](https://togithub.com/defenseunicorns/uds-common/issues/202)) ([5d7ea03](https://togithub.com/defenseunicorns/uds-common/commit/5d7ea03815929a662c529b2078bdf895d8f3ac1b)) - update renovate creds ([#​200](https://togithub.com/defenseunicorns/uds-common/issues/200)) ([1c6eb24](https://togithub.com/defenseunicorns/uds-common/commit/1c6eb24f37b4059589a70c9addeffb80895d450b)) ##### Miscellaneous - add renovate support for org ghcr packages ([#​199](https://togithub.com/defenseunicorns/uds-common/issues/199)) ([2c5de9c](https://togithub.com/defenseunicorns/uds-common/commit/2c5de9cc41cad9d1e02faf39c0cad364933f335f)) - **deps:** update uds common support dependencies ([#​195](https://togithub.com/defenseunicorns/uds-common/issues/195)) ([04b6409](https://togithub.com/defenseunicorns/uds-common/commit/04b64091ba0528463713f66d8167572a533e0c3d)) - fix codeowners ([#​196](https://togithub.com/defenseunicorns/uds-common/issues/196)) ([856ef22](https://togithub.com/defenseunicorns/uds-common/commit/856ef221b39e65070e966942b42e79d408f59b76)) ### [`v0.11.0`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.11.0) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.10.0...v0.11.0) ##### Features - add support for uds-core snapshots ([#​193](https://togithub.com/defenseunicorns/uds-common/issues/193)) ([7a39915](https://togithub.com/defenseunicorns/uds-common/commit/7a39915ceff7a1a9e319846042ab74390fda6f2b)) ##### Miscellaneous - **deps:** update uds common support dependencies ([#​187](https://togithub.com/defenseunicorns/uds-common/issues/187)) ([a0bbfb0](https://togithub.com/defenseunicorns/uds-common/commit/a0bbfb043e670a175fbdc44585e2bbb5b695acf7)) ### [`v0.10.0`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.10.0) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.9.0...v0.10.0) ##### Features - add task for determining target repo based on flavor ([#​188](https://togithub.com/defenseunicorns/uds-common/issues/188)) ([6810324](https://togithub.com/defenseunicorns/uds-common/commit/681032402a315c8db80975571242ed8db73e78bf)) ### [`v0.9.0`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.9.0) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.8.2...v0.9.0) ##### ⚠ BREAKING CHANGES - update doug ci credential for new identity config req ##### Bug Fixes - update doug ci credential for new identity config req ([71340f7](https://togithub.com/defenseunicorns/uds-common/commit/71340f7d4fc0cd8fd6c44335b54e0b12769965d1)) ### [`v0.8.2`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.8.2) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.8.1...v0.8.2) ##### Miscellaneous - add additional install step to playwright install ([#​183](https://togithub.com/defenseunicorns/uds-common/issues/183)) ([41855e4](https://togithub.com/defenseunicorns/uds-common/commit/41855e42bd73c67109ed42935f1e67ab7305ddda)) - **deps:** update uds common support dependencies ([#​179](https://togithub.com/defenseunicorns/uds-common/issues/179)) ([e1a0d5a](https://togithub.com/defenseunicorns/uds-common/commit/e1a0d5acba2c0cc083af6ac2823d9cf068008453)) - fix the Zarf package renovate regex to the correct versionTemplate ([#​181](https://togithub.com/defenseunicorns/uds-common/issues/181)) ([272b502](https://togithub.com/defenseunicorns/uds-common/commit/272b502fa2f36b3703f9cdcbdbfb579ce437a0d7)) ### [`v0.8.1`](https://togithub.com/defenseunicorns/uds-common/releases/tag/v0.8.1) [Compare Source](https://togithub.com/defenseunicorns/uds-common/compare/v0.8.0...v0.8.1) ##### Miscellaneous - add cgr identity assume to setup action ([#​180](https://togithub.com/defenseunicorns/uds-common/issues/180)) ([2ec74fb](https://togithub.com/defenseunicorns/uds-common/commit/2ec74fbe496c5cdcc88cd3f424951f11271fe5d6)) - fix version matching for UDS packages ([#​176](https://togithub.com/defenseunicorns/uds-common/issues/176)) ([e068b6a](https://togithub.com/defenseunicorns/uds-common/commit/e068b6a255cc856e313485826a2140a3977c6b03)) </details> <details> <summary>defenseunicorns/pepr (pepr)</summary> ### [`v0.33.0`](https://togithub.com/defenseunicorns/pepr/releases/tag/v0.33.0) [Compare Source](https://togithub.com/defenseunicorns/pepr/compare/v0.32.7...v0.33.0) #### Features - feat: sanitizing store keys - store path v2 by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/918](https://togithub.com/defenseunicorns/pepr/pull/918) - feat: merge package env vars by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/963](https://togithub.com/defenseunicorns/pepr/pull/963) #### What's Changed - chore: update docs around punycode warning by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/960](https://togithub.com/defenseunicorns/pepr/pull/960) - chore: enhance debugging docs by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/945](https://togithub.com/defenseunicorns/pepr/pull/945) - chore: update store adr by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/958](https://togithub.com/defenseunicorns/pepr/pull/958) - chore: add date to license by [@​cmwylie19](https://togithub.com/cmwylie19) in [https://github.com/defenseunicorns/pepr/pull/957](https://togithub.com/defenseunicorns/pepr/pull/957) - chore: bump actions/dependency-review-action from 4.3.3 to 4.3.4 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/952](https://togithub.com/defenseunicorns/pepr/pull/952) - chore: bump fast-check from 3.19.0 to 3.20.0 in the development-dependencies group by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/956](https://togithub.com/defenseunicorns/pepr/pull/956) - chore: bump github/codeql-action from 3.25.11 to 3.25.12 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/955](https://togithub.com/defenseunicorns/pepr/pull/955) - chore: bump anchore/scan-action from 4.0.0 to 4.1.0 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/962](https://togithub.com/defenseunicorns/pepr/pull/962) - chore: bump pino from 9.2.0 to 9.3.1 in the production-dependencies group by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/961](https://togithub.com/defenseunicorns/pepr/pull/961) - chore: bump chainguard/node-lts from `ea8ec8f` to `490bcd9` by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/965](https://togithub.com/defenseunicorns/pepr/pull/965) - chore: bump [@​types/node](https://togithub.com/types/node) from 18.19.39 to 18.19.40 in the development-dependencies group by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/964](https://togithub.com/defenseunicorns/pepr/pull/964) - chore: bump docker/login-action from 3.2.0 to 3.3.0 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/973](https://togithub.com/defenseunicorns/pepr/pull/973) - chore: bump chainguard/node-lts from `490bcd9` to `4db5b44` by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/967](https://togithub.com/defenseunicorns/pepr/pull/967) - chore: bump docker/setup-buildx-action from 3.4.0 to 3.5.0 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/972](https://togithub.com/defenseunicorns/pepr/pull/972) - chore: bump step-security/harden-runner from 2.8.1 to 2.9.0 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/969](https://togithub.com/defenseunicorns/pepr/pull/969) - chore: bump github/codeql-action from 3.25.12 to 3.25.13 by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/971](https://togithub.com/defenseunicorns/pepr/pull/971) - chore: bump kubernetes-fluent-client from 2.6.4 to 2.6.5 in the production-dependencies group by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/976](https://togithub.com/defenseunicorns/pepr/pull/976) - chore: bump the development-dependencies group across 1 directory with 3 updates by [@​dependabot](https://togithub.com/dependabot) in [https://github.com/defenseunicorns/pepr/pull/974](https://togithub.com/defenseunicorns/pepr/pull/974) **Full Changelog**: defenseunicorns/pepr@v0.32.7...v0.33.0 </details> <details> <summary>kulshekhar/ts-jest (ts-jest)</summary> ### [`v29.2.4`](https://togithub.com/kulshekhar/ts-jest/blob/HEAD/CHANGELOG.md#2924-2024-08-01) [Compare Source](https://togithub.com/kulshekhar/ts-jest/compare/v29.2.3...v29.2.4) ### [`v29.2.3`](https://togithub.com/kulshekhar/ts-jest/blob/HEAD/CHANGELOG.md#2923-2024-07-18) [Compare Source](https://togithub.com/kulshekhar/ts-jest/compare/v29.2.2...v29.2.3) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR was generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View the [repository job log](https://developer.mend.io/github/defenseunicorns/uds-core). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40MzEuNCIsInVwZGF0ZWRJblZlciI6IjM4LjE4LjE3IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Micah Nagel <micah.nagel@defenseunicorns.com>
Description
This PR Migrates the Store to a v2 data path, migrates keys from previous version before a read of the CR, patches them to v2, then starts a watch. It also adds fuzzing, property-based, more unit and journey tests.
Related Issue
Fixes #915
Relates to PR defenseunicorns/pepr-excellent-examples#46 updating the e2e test to expect the keys base64 encoded
Type of change
Checklist before merging