Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Content processing CI/CD is flaky #643

Open
1 of 12 tasks
aj-stein-gsa opened this issue Aug 29, 2024 · 1 comment
Open
1 of 12 tasks

Content processing CI/CD is flaky #643

aj-stein-gsa opened this issue Aug 29, 2024 · 1 comment
Assignees
@aj-stein-gsa
Labels
bug Something isn't working

Comments

@aj-stein-gsa
Copy link
Contributor

This relates to ...

  • the FedRAMP OSCAL Registry
  • the FedRAMP OSCAL baselines
  • the Guide to OSCAL-based FedRAMP Content
  • the Guide to OSCAL-based FedRAMP System Security Plans (SSP)
  • the Guide to OSCAL-based FedRAMP Security Assessment Plans (SAP)
  • the Guide to OSCAL-based FedRAMP Security Assessment Results (SAR)
  • the Guide to OSCAL-based FedRAMP Plan of Action and Milestones (POA&M)
  • the FedRAMP SSP OSCAL Template (JSON or XML Format)
  • the FedRAMP SAP OSCAL Template (JSON or XML Format)
  • the FedRAMP SAR OSCAL Template (JSON or XML Format)
  • the FedRAMP POA&M OSCAL Template (JSON or XML Format)
  • the FedRAMP OSCAL Validations

What happened?

It seems the content generation and testing CI in GHA is flaky looking at #641 failing on an unrelated change and clicking back through the master branch from time to time.

Recent GHA job output leads me to one hypothesis, that is some parts of how to run Docker Compose has changed and the old way is no longer supported.

In the last few years, Docker Compose had a single wrapper command executable, docker-compose and then moved to docker compose (observe the space). It would seem that GitHub discussion chatter confirms that was obsoleted.

This would mean local tests work, but not GHA. We need to investigate 1) if this is the only root case; 2) confirm/deny other root causes; 3) fix some/none/all and improve stability of all tests on PR and post-merge.

Relevant log output

##[debug]Evaluating condition for step: 'Build content generator'
##[debug]Evaluating: success()
##[debug]Evaluating success:
##[debug]=> true
##[debug]Result: true
##[debug]Starting: Build content generator
##[debug]Loading inputs
##[debug]Loading env
Run make init-content
##[debug]/usr/bin/bash -e /home/runner/work/_temp/4b01530c-f811-45dd-82e9-e2bd38a89b21.sh
Building Docker image for OSCAL content generation...
docker-compose -f vendor/oscal/build/docker-compose.yml -f src/content/docker-compose.yml build
make: docker-compose: Command not found
make: *** [src/content/module.mk:16: init-content] Error 127
Error: Process completed with exit code 2.
##[debug]Finishing: Build content generator

How do we replicate this issue?

  1. Create a PR with any change.
  2. Observe somewhat consistent failure of the content-artifacts.yml workflow.

Where, exactly?

GitHub Actions, so .github/workflows.

Other relevant details

No response
@aj-stein-gsa aj-stein-gsa added the bug Something isn't working label Aug 29, 2024
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
From docker-compose to docker compose for GSA#643
f3cf3dd 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
From docker-compose to docker compose for GSA#643
81f7965 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
@aj-stein-gsa aj-stein-gsa mentioned this issue Sep 10, 2024
Merged
6 tasks
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
Update obsolete actions/checkout actions for GSA#643
d6658f6 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
Remove version from compose files for GSA#643
92a8552 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
From docker-compose to docker compose for GSA#643
9555679 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
Update obsolete actions/checkout actions for GSA#643
f4e1f6a 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa
Remove version from compose files for GSA#643
e4461b7 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
aj-stein-gsa added a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa
From docker-compose to docker compose for #643
d5b9ddc 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
aj-stein-gsa added a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa
Update obsolete actions/checkout actions for #643
75b8a0a 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
aj-stein-gsa added a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa
Remove version from compose files for #643
6096faa 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
Rene2mt pushed a commit to Rene2mt/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
From docker-compose to docker compose for GSA#643
20ed9b4 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
Rene2mt pushed a commit to Rene2mt/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
Update obsolete actions/checkout actions for GSA#643
c8d5185 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
Rene2mt pushed a commit to Rene2mt/fedramp-automation that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
Remove version from compose files for GSA#643
e9f5a44 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
Rene2mt pushed a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
From docker-compose to docker compose for #643
dfd6ea6 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
Rene2mt pushed a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
Update obsolete actions/checkout actions for #643
2c18797 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
Rene2mt pushed a commit that referenced this issue Sep 10, 2024
@aj-stein-gsa @Rene2mt
Remove version from compose files for #643
9356a91 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
@aj-stein-gsa aj-stein-gsa moved this from 🆕 New to 👀 In review in FedRAMP Automation Sep 18, 2024
@aj-stein-gsa
Copy link
Contributor Author

I actually "completed the work," but it is not merged up to master yet so I will leave this open and mark "in progress" until it is merged there and complete.

@aj-stein-gsa aj-stein-gsa self-assigned this Sep 18, 2024
@aj-stein-gsa aj-stein-gsa mentioned this issue Sep 23, 2024
Merged
7 tasks
aj-stein-gsa added a commit to Rene2mt/fedramp-automation that referenced this issue Sep 23, 2024
@aj-stein-gsa
From docker-compose to docker compose for GSA#643
fa47836 
More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185
aj-stein-gsa added a commit that referenced this issue Sep 23, 2024
@Rene2mt @aj-stein-gsa
Fix odp ID for SI-03 (issue#695) (#714)
8c539cf 
* Fix odp ID for SI-03 (issue#695)

* From `docker-compose` to `docker compose` for #643

More details about GHA runner compatibility with old versus new docker
compose commands in this discussion post.

https://github.com/orgs/community/discussions/27185

---------

Co-authored-by: A.J. Stein <aj@gsa.gov>
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 24, 2024
@aj-stein-gsa
Update obsolete actions/checkout actions for GSA#643
b4ad4f7 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
aj-stein-gsa added a commit to aj-stein-gsa/fedramp-automation that referenced this issue Sep 24, 2024
@aj-stein-gsa
Remove version from compose files for GSA#643
3a0d415 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
brian-ruf pushed a commit to brian-ruf/fedramp-automation that referenced this issue Nov 8, 2024
@aj-stein-gsa @brian-ruf
Update obsolete actions/checkout actions for GSA#643
5f148ce 
Not all, but many, actions were pinned to old versions by tag and to an
explicit SHA1 hash from the repo that are old enough to cause GHA
error notices. Upgraded from these to current SHA1 hash for the most
(692973e3d937129bcbf40652eb9f2f61becf3332) for recent v4.1.7 release.

> The following actions uses node12 which is deprecated and will be forced to run on node16: actions/checkout@v2. For more info: https://github.blog/changelog/2023-06-13-github-actions-all-actions-will-run-on-node16-instead-of-node12-by-default/

> The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v2. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/

Source: https://github.com/GSA/fedramp-automation/actions/runs/10783353551
brian-ruf pushed a commit to brian-ruf/fedramp-automation that referenced this issue Nov 8, 2024
@aj-stein-gsa @brian-ruf
Remove version from compose files for GSA#643
d230726 
This directive is now obsolete, causes more noise in the GHA workflow
run logs.

https://forums.docker.com/t/docker-compose-yml-version-is-obsolete/141313
This was referenced Nov 13, 2024
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aj-stein-gsa aj-stein-gsa

Labels
bug Something isn't working
Projects
FedRAMP Automation
Status: 🚢 Ready to Ship
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

643-content-processing-cicd-is-flaky aj-stein-gsa/fedramp-automation
1 participant
@aj-stein-gsa