Test GHCR integration with "in upstream" branch instead of fork with write perms dropping #665
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Interesting, this time got a failure with exit code 139, memory violation/exhaustion signal: Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c wget ${WGET_EXTRA_ARGS} -O /etc/apk/keys/adoptium.rsa.pub \"${TEMURIN_APK_KEY_URL}\" && echo \"${TEMURIN_APK_REPO_URL}\" >> /etc/apk/repositories && apk add ${APK_EXTRA_ARGS} --no-cache ${TEMURIN_APK_VERSION} && mkdir -p /opt/fedramp/oscaljs && mkdir -p /opt/fedramp/constraints && (cd /opt/fedramp/oscaljs && npm install oscal@${OSCAL_JS_VERSION})" did not complete successfully: exit code: 139Rerunning and crossing my fingers this time! |
aj-stein-gsa
force-pushed
the
chore/656-test-ghcr-integration-without-fork
branch
from
982c577
to
f65cee7
Compare
Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored.
For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build.
Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions.
This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry.
For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now.
Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile.
See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731
The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731
We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly.
It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README.
aj-stein-gsa
force-pushed
the
chore/656-test-ghcr-integration-without-fork
branch
from
f65cee7
to
8701cab
Compare
|
/cc @david-waltermire I had one minor issue with tag formatting and the build failure is coming from Cloud Pages (with my GitHub account and Oauth I cannot figure out how to rid myself of that, I get access denied). Can you let me know if you are OK with this and squashing. Docs churn is next on the agenda for me in prep for Wednesday. |
If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms.
It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested
- Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing.
I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID.
For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it.
Just removing it may not have done the trick.
|
Alright, there has not been a lot of feedback and most of the kinks are ironed out and a development workflow and release pipeline has a foundation now, so I will merge it and take follow-on feedback from community through the pilot. |
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 24, 2024
…write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 25, 2024
…write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick.
brian-ruf
pushed a commit
to brian-ruf/fedramp-automation
that referenced
this pull request
Nov 8, 2024
…write perms dropping (GSA#665) * Add initial OCI spec for container for GSA#655 * Now add FR constraint files for GSA#655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for GSA#656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for GSA#656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for GSA#656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for GSA#656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for GSA#656 Just removing it may not have done the trick.
This was referenced Nov 13, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Committer Notes
This is a continuation of #656.
All Submissions:
For now, see #656 for details, may copy-paste a little later when done.