Build docker image through technical walkthroughs and quicker local development #656
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aj-stein-gsa
force-pushed
the
655-build-a-comprehensive-container-with-required-tools
branch
4 times, most recently
from
e86560b
to
f356a90
Compare
aj-stein
force-pushed
the
655-build-a-comprehensive-container-with-required-tools
branch
from
687b679
to
b7d92af
Compare
aj-stein-gsa
force-pushed
the
655-build-a-comprehensive-container-with-required-tools
branch
from
1967e76
to
376ff78
Compare
Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored.
For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build.
Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions.
aj-stein-gsa
force-pushed
the
655-build-a-comprehensive-container-with-required-tools
branch
from
3d4ba98
to
0e2d970
Compare
aj-stein-gsa
force-pushed
the
655-build-a-comprehensive-container-with-required-tools
branch
from
978630a
to
7eb56c5
Compare
This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry.
For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now.
|
I am tacking the container image build on the |
Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile.
See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731
The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731
We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly.
|
Figuring out organization permissions to write to GitHub containers to github.com/GSA is turning into a fun exercise that I had hoped to take 15 minutes, not an hour on and off. More to follow! |
|
Perhaps the fact that this initial PR is adding any It may be time to merge into a github.com/GSA/fedramp-automation branch and point that back to the feature branch to consider. |
aj-stein-gsa
changed the base branch from
feature/external-constraints
to
chore/656-test-ghcr-integration-without-fork
aj-stein-gsa
merged commit 982c577
into
GSA:chore/656-test-ghcr-integration-without-fork
2 of 3 checks passed
|
More to follow with another PR to check the per comments with the branch in URL below with a follow-on PR to test. |
aj-stein-gsa
added a commit
to aj-stein-gsa/fedramp-automation
that referenced
this pull request
Sep 11, 2024
It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested
aj-stein-gsa
added a commit
to aj-stein-gsa/fedramp-automation
that referenced
this pull request
Sep 11, 2024
- Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing.
aj-stein-gsa
added a commit
to aj-stein-gsa/fedramp-automation
that referenced
this pull request
Sep 11, 2024
I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 11, 2024
For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 11, 2024
Just removing it may not have done the trick.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 11, 2024
…write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 24, 2024
…write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick.
aj-stein-gsa
added a commit
that referenced
this pull request
Sep 25, 2024
…write perms dropping (#665) * Add initial OCI spec for container for #655 * Now add FR constraint files for #655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for #656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for #656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for #656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for #656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for #656 Just removing it may not have done the trick.
brian-ruf
pushed a commit
to brian-ruf/fedramp-automation
that referenced
this pull request
Nov 8, 2024
…write perms dropping (GSA#665) * Add initial OCI spec for container for GSA#655 * Now add FR constraint files for GSA#655 * Woops, fix typo in clone path for fd_data_dl scratch container * Constraints in /opt/fedramp sudir, make it WORKDIR * Switch to Alpine Maven scratch image not Debian * Switch to Node for final image, install oscaljs * Add checkout data to final image * Fix missed parameterization of git image * Add non-default OCI image build target for make * Verify GPG signaure of oscal-cli build * Add clean target for OCI image builds * Allow for TLS bypass and proxy in Makefile Disable cert-checking for the local version that is built on laptops for GSA staff who make use of a VPN/proxy solution that intercept all TLS communication for security monitoring. This includes not just Docker, but also the containers as they build an image. Since production images will be made in GitHub Actions without the Makefile, these directives will be ignored. * Do not do slow git clone, use local COPY instead For speed, ease of access, and leave commit metadata from the container ID linked to the commit hash itself, just copy from the outside context of the image build. * Add publish target to Makefile with useful tags Also try docker push to GHCR to start before moving on the "in pipeline" build with GitHub Actions. * Fix repeat docker commands for correct tag-n-push * Correct the org.opencontainers.image.source label * Actions: perms for writing packages (ghcr.io) * Actions: follow GH tutorial, more perms added * Actions: build, sign, push, attest and OCI image This workflow change is the first attempt at building, pushing, and signing the validation-tools image to push to the ghcr.io registry. * Actions: ref_name for image tags problematic For both PRs and non-PR branches, that seems to cause problems for tags that we ought to avoid for now. * Actions: use action correctly, no manual labels * Actions: remove metadata from Dockerfile, use GHA * Actions: woops, forgot explicit checkout path Our GHA CI/CD checks out to `./git-content`, `.` by default so the action directive looking for context did not find the Dockerfile. * Actions: check if least privilege perms block push See more details in this reply and the larger context from others who cannot push a built container to ghcr.io. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: scratch that, `write-all` blocked by org The github.com/GSA organization still blocks the write to an org-level package in very permissive move. Tips from the discussions posts did not help here. https://github.com/orgs/community/discussions/57724#discussioncomment-7779731 * Actions: add metadata action SHA options We need to force SHA1 long (not seven-digit short version to avoid collisions), remove both `sha-` prefix and remove suffix explicitly. * Actions, sigh, really remove `sha256` prefix again It seems that didn't stick the last time, so I will try this config again and follow the official custom hash label strategy from the action example from the official README. * Support MVP platforms, arm64 and amd64 If not we will only support modern Apple computers with modern M1 chips, not Intel environments for PC and older Macs. We need broad support for these top platforms. * Explicit platform option for buildx too for GSA#656 It seems this may be needed because I still get similar but different warnings on multi-platform docker builds when using on macOS on an Apple laptop with a M1 processor and amd64 processor for personal computers with Windows and Linux operating systems respectively. > WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v3) and no specific platform was requested * Pin metadata action and update configs for GSA#656 - Had a slightly wrong version of docker/metadata-action that could not use annotations properly, hence no annotations on image. - Use annotations instead of custom override labels with that action. - Update docker/build-push-registry action to retrieve those labels as well. - Change subject name for attestation to end with `-attestation` suffix to make the GHCR registry entries less confusing. * Woops, attestation subject === image name for GSA#656 I re-read the dogs. Attestations will be uploaded to Sigstore but I will not busy up the registry with them every moment as it will make it even more confusing for novice users and advanced developers what data they are looking for by content-addressable git commit hash ID. * Explanatory comments on Dockerfile lint for GSA#656 For future analysis or assessment, I am leaving information in the Dockerfile as comments to address warning output in docker build and push flagging a potential finding re secrets based on variable names. ```sh 4 warnings found (use docker --debug to expand): - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "OSCAL_CLI_GPG_KEY") (line 20) - SecretsUsedInArgOrEnv: Do not use ARG or ENV instructions for sensitive data (ARG "TEMURIN_APK_KEY_URL") (line 45) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 17) - FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 43) ``` They are IDs to secrets, not actually secrets, now I have documented it. * Attestations need explicit reg push off for GSA#656 Just removing it may not have done the trick.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Committer Notes
Closes #655.
All Submissions:
Have you squashed any non-relevant commits and commit messages? [instructions]Will do so on review/approve/merge cycle.If applicable, have all FedRAMP Documents Related to OSCAL Adoption affected by the changes in this issue have been updated.?Will follow-up in the cleanup docs issue per Update or create necessary installation documentation #659.By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.