Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): GHA - container image build & push #841

Merged
merged 5 commits into from
Mar 10, 2022

Conversation

kskewes-sf
Copy link
Contributor

@kskewes-sf kskewes-sf commented Mar 1, 2022

shared:

  • GHA doesn't have a repository name env var like clouddriver so
    add a step to extract the name from spinnaker/clouddriver
  • Change build step to create installation distribution for use in
    following container image build steps.
  • add Alpine/slim container build step
  • add Ubuntu container build step

when open PR has commit pushed to fork's branch:

  • build only and don't push
    • latest{-slim|ubuntu}
    • {git-SHA}-{DATE-TIME}{-slim|ubuntu}
  • validates gradle build, unit tests pass and Dockerfile's build

when commit pushed to specific branches (eg: merge PR):

  • build and push with branch tags:
    • master -> master-latest-unvalidated-{slim|ubuntu}
    • master -> master-{git-SHA}-{DATE-TIME}-unvalidated-{slim|ubuntu}
    • release-* -> {release-*}-latest-unvalidated-{slim-ubuntu}
    • release-* -> {release-*}-{git-SHA}-{DATE-TIME}-unvalidated-{slim-ubuntu}
  • DONE: change version-* to release-* to do action on merge to release
    branches.

when push tag x.y.z (i.e: Release):

  • build and push with version tag.
    • RELEASE_VERSION -> {RELEASE_VERSION}-unvalidated-{slim-ubuntu}
    • RELEASE_VERSION* -> {RELEASE_VERSION}-{git-SHA}-{DATE-TIME}-unvalidated-{slim-ubuntu}
  • DONE: clarify if this is still necessary or should be revised to be
    inline with release-x.y.z - follow same pattern as Debian packages.

DONE: confirmed we need/want {date} in image tag name per previous CI. - Identifies distinct builds created by non-idempotent actions.

.github/workflows/pr.yml Outdated Show resolved Hide resolved
@kskewes-sf kskewes-sf force-pushed the container-build-gha branch 2 times, most recently from 2a1f464 to e47dd63 Compare March 3, 2022 00:54
shared:
- GHA doesn't have a repository name env var like `clouddriver` so add a step
  to extract the name from `spinnaker/clouddriver`
- Change build step to create installation distribution for use in
  following container image build steps.
- add Alpine/slim container build step
- add Ubuntu container build step

when open PR has commit pushed to fork's branch:
- build only and don't push
- validate unit tests pass and Dockerfile's build

when commit pushed to specific branches (eg: merge PR):
- build and push with branch tags:
  - `master` -> `spinnaker-master-latest-unvalidated-{slim|ubuntu}`
  - `version-*` ->
  `spinnaker-{version-*}-latest-unvalidated-{slim-ubuntu}`
- TODO: change `version-*` to `release-*` to do action on merge to release
  branches?

when push tag x.y.z (i.e: Release):
- build and push with version tag.
- TODO: clarify if this is still necessary or should be revised to be
  inline with `release-x.y.z`

TODO: confirm if we need/want {date} in image tag name per previous CI.
We are not generating a new patch version here, just building and
pushing latest container image for the release.

Also `version-*` branches don't exist so we can remove it from GHA.
Add git commit SHA and date time to all container tag variations in all build
& push flows

Container registries list images by date pushed so it can be quicker to
find the latest image using this sorting versus looking for and comparing date
strings within the tag.

However due to the use of non-idempotent commands like `apk update` in
`Dockerfile`'s the same git SHA could result in different container
images (with different container image SHA).
@kskewes-sf kskewes-sf force-pushed the container-build-gha branch from 04b1a30 to d75942b Compare March 6, 2022 22:45
`spinnaker` is already in the container repository name so the prefix
`spinnaker-` doesn't add any meaning to our container tags so we can
remove it simplifying the tag.

When we are adding a git commit SHA and DATE TIME the string "-latest"
doesn't add any meaning so we can remove it, simplifying the tag.
@kskewes-sf kskewes-sf force-pushed the container-build-gha branch from f165294 to 7355a0a Compare March 6, 2022 22:58
@kskewes-sf kskewes-sf marked this pull request as draft March 7, 2022 01:46
We need to build and push them first as "unvalidated".
After integration tests have run successfully we will add a new set of
tags with "validated" instead and also without, eg: `rosco:1.x.y`
@kskewes-sf kskewes-sf marked this pull request as ready for review March 10, 2022 02:56
@dbyron-sf dbyron-sf merged commit 5d4375a into spinnaker:master Mar 10, 2022
dbyron-sf added a commit to dbyron-sf/orca that referenced this pull request Mar 10, 2022
dbyron-sf added a commit to dbyron-sf/fiat that referenced this pull request Mar 10, 2022
dbyron-sf added a commit to spinnaker/fiat that referenced this pull request Mar 10, 2022
dbyron-sf added a commit to spinnaker/orca that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/clouddriver that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/echo that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/fiat that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/front50 that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/gate that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/halyard that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/igor that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/keel that referenced this pull request Mar 10, 2022
dbyron-sf pushed a commit to spinnaker/igor that referenced this pull request Mar 10, 2022
dbyron-sf pushed a commit to spinnaker/halyard that referenced this pull request Mar 10, 2022
kskewes-sf added a commit to kskewes-sf/clouddriver that referenced this pull request Mar 10, 2022
dbyron-sf pushed a commit to spinnaker/gate that referenced this pull request Mar 10, 2022
to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
dbyron-sf pushed a commit to spinnaker/front50 that referenced this pull request Mar 10, 2022
to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
dbyron-sf pushed a commit to spinnaker/echo that referenced this pull request Mar 10, 2022
to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
dbyron-sf pushed a commit to spinnaker/clouddriver that referenced this pull request Mar 10, 2022
dbyron-sf pushed a commit to spinnaker/keel that referenced this pull request Mar 15, 2022
to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
kskewes-sf added a commit to kskewes-sf/kayenta that referenced this pull request Mar 16, 2022
to Google Artifact Registry

see: spinnaker/rosco#841 and follow up PR's all bundled here.
dbyron-sf pushed a commit to spinnaker/kayenta that referenced this pull request Mar 16, 2022
to Google Artifact Registry

see: spinnaker/rosco#841 and follow up PR's all bundled here.
kskewes-sf added a commit to kskewes-sf/deck that referenced this pull request Mar 22, 2022
to Google Artifact Registry

see: spinnaker/rosco#841 and follow up PR's all bundled here.
dbyron-sf pushed a commit to spinnaker/deck that referenced this pull request Mar 22, 2022
* chore(ci): Merge GHA steps to single step

Before adding container build (and push) steps on master/release-*/PR/tag
simplify jobs down and confirm there is no unreasonable increase in CI job
duration.

* chore(ci): Update mergify job match and make consistent

- `Deck CI` was renamed to `build` as per the other services.
- It was difficult to diff compare this file to the other services so
  some keys were slightly rearranged to minimise the diff. No functional
  impact.

* chore(ci): GHA - container image and apt package build & push

to Google Artifact Registry

see: spinnaker/rosco#841 and follow up PR's all bundled here.

* chore(ci): GHA - gradle build different to java service

- there is no `installDist` target. `build` target generates files we
  can copy into container in `Dockerfile`.
- switch bintray plugin for (Google) artifact publish plugin
- call artifact publish plugin directly rather than via a "publish" task
mergify bot pushed a commit to spinnaker/deck that referenced this pull request Mar 22, 2022
* chore(ci): Merge GHA steps to single step

Before adding container build (and push) steps on master/release-*/PR/tag
simplify jobs down and confirm there is no unreasonable increase in CI job
duration.

* chore(ci): Update mergify job match and make consistent

- `Deck CI` was renamed to `build` as per the other services.
- It was difficult to diff compare this file to the other services so
  some keys were slightly rearranged to minimise the diff. No functional
  impact.

* chore(ci): GHA - container image and apt package build & push

to Google Artifact Registry

see: spinnaker/rosco#841 and follow up PR's all bundled here.

* chore(ci): GHA - gradle build different to java service

- there is no `installDist` target. `build` target generates files we
  can copy into container in `Dockerfile`.
- switch bintray plugin for (Google) artifact publish plugin
- call artifact publish plugin directly rather than via a "publish" task

(cherry picked from commit f5fe6c2)

# Conflicts:
#	.github/workflows/ci.yml
#	.mergify.yml
rsh09 pushed a commit to OpsMx/gate that referenced this pull request Mar 31, 2022
* fix(oauth): remove circular dependency on ExternalAuthTokenFilter bean in OAuth2SsoConfig when oauth2 is enabled (spinnaker#1492)

Previously in spring 2.2.5, if oauth2 is enabled there is no circular
dependency on ExternalAuthTokenFilter bean in OAuth2SsoConfig.

In spring 2.2.13, if oauth2 is enabled there is a circular dependency
error on ExternalAuthTokenFilter bean. This circular dependency results
in an error when the gate application tries to start up. The application
fails with error:

BeanCurrentlyInCreationException: Error creating bean with name
'OAuth2SsoConfig': Bean with name 'OAuth2SsoConfig' has been injected
into other beans [externalAuthTokenFilter] in its raw version as part
of a circular reference, but has eventually been wrapped.

To fix this error, add the Component annotation to
ExternalAuthTokenFilter and remove the ExternalAuthTokenFilter bean
from OAuth2SsoConfig.

Co-authored-by: David Byron <dbyron@salesforce.com>

* fix(web): disable keel by default as it is an optional service (spinnaker#1453)

Co-authored-by: Justin Field <justin.field@armory.io>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* fix(api): fix movie quotes to match movie script (spinnaker#1423)

Co-authored-by: Justin Field <justin.field@armory.io>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* chore(build): gradle 6.8.1 (spinnaker#1413)

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* chore(dependencies): Autobump korkVersion (spinnaker#1493)

Co-authored-by: root <root@265713e22300>

* chore(dependencies): Autobump korkVersion (spinnaker#1495)

Co-authored-by: root <root@ca7e6659c3b3>

* chore(dependencies): Autobump korkVersion (spinnaker#1496)

Co-authored-by: root <root@7e927306ac3d>

* fix(vulnerability): avoid expose gate endpoints (spinnaker#1497)

* chore(dependencies): Autobump korkVersion (spinnaker#1501)

Co-authored-by: root <root@01b270ba5f94>

* chore(dependencies): Autobump fiatVersion (spinnaker#1504)

Co-authored-by: root <root@73790eeb50df>

* chore(dependencies): Autobump korkVersion (spinnaker#1502)

Co-authored-by: root <root@a229fbda21b3>
Co-authored-by: Matt <6519811+mattgogerly@users.noreply.github.com>

* feat(web): Expose experimental account storage API (spinnaker#1494)

* feat(web): Expose experimental account storage API

This adds some of the REST APIs introduced in the experimental account
storage API in Clouddriver to Gate. Initially, these APIs are only
available for admins.

* Combine account and credentials endpoints

* Add docs on AccountDefinition

* Add alpha annotations

* chore(build): update mergify config (spinnaker#1506)

Co-authored-by: Cameron Motevasselani <cameron@armory.io>

* fix(gate): Typos in Account Management API (spinnaker#1510)

* fix(gate/web): Fix typo in PreAuthorize annotation

It appears that some variables were renamed while I worked on the PR
and didn't update the annotations to match.

* fix(gate/core): Fix retrofit signature error

* fix(gate/web): Add explicit name property to AccountDefinition (spinnaker#1514)

This fixes an authorization check error where Jackson knows how to
handle the `name` property of an account definition, but SpEL does
not see the property. Now the PostFilter annotation should work
equivalently to the same filter check in Clouddriver.

* chore(dependencies): Autobump korkVersion (spinnaker#1515)

Co-authored-by: root <root@d38215f0a8da>

* fix(authn/oauth2): prevent oauth2 redirect loops (spinnaker#1517)

During setup of spinnaker authentication with oauth2 a common hurdle is a redirect loop.

For example:

spinnaker/spinnaker#5794
spinnaker/spinnaker#1630

Also, many threads in Slack discuss these problems. In fact this appears to be a common
pitfall for the spring-security-oauth2-autoconfigure library in general. A light refresher
on the ouath2 flow in play here seems worthwhile. The user is redirected from `/login` in gate
to the external auth provider (google, github, etc.) and after successfully authenticating
they are redirected back to the gate `/login` endpoint but this time with a code parameter
that is to be used to request an access token.

This request can fail for a variety of reasons, and if it does, the underlying spring library
triggers a redirect to the `/error` endpoint. What causes the redirect loop for gate in particular
(and for other users of the library in a similar fashion) is that the WebSecurityConfigurerAdapter
in play is treating `/error` as an authenticated path and so instead of just returning with a 401,
it re-redirects to `/login` and the redirect loop continues.

My thought is that instead of a redirect loop, simply allowing the 401 to be returned will be a stronger
more helpful signal as to what is going on. Hopefully it will save future first-time installers headaches.

Spinnaker docs have included several troubleshooting hints and tips for how where you terminate SSL
affects configuration etc. Even after following all of these and lots of spelunking through spinnaker
github issues and combing over threads in slack, I found myself still experiencing a redirect loop even
though I had applied all the combined wisdom that was applicable to my setup.

As it turns out, I had a bad copy/paste of my client secret in my configuration. So the request
to turn the code from google into an access token from google was failing with a 401. After much
debugging and deep diving into the spring security code I found that had I turned on DEBUG in gate
for these classes in gate-local.yml:

```
logging:
  level:
    org.springframework.security.web.authentication.SimpleUrlAuthenticationFailureHandler: DEBUG
    org.springframework.security.oauth2.client.filter.OAuth2ClientAuthenticationProcessingFilter: DEBUG
```

Then I would have seen in the logs that a 401 response was returned from google and perhaps it would have
caused me to look closer at my botched client secret configuration. I think perhaps we don't want to require
that all operators of spinnaker become spring-security-oauth2 experts. So I'm proposing adding `/error` to
the list of paths in gate that aren't treated as authenticated. Thus short-circuiting the redirect loop and
bringing to light helpful troubleshooting info that was previously more or less swallowed.

* chore(ci): update setup-java GHA to v2 (spinnaker#1518)

closes: spinnaker/spinnaker#6611

* fix(dependency): Issue with jackson-bom and kotlin-bom version conflict resolution while upgrading the spring-boot 2.3.x (spinnaker#1505)

* fix(dependency): Introducing spring dependency management gradle plugin

Spring boot has moved to gradle based dependency management from v2.3.x. This change has brought issue of conflict resolution failure of the Jackson-bom version and kotlin-bom version with gate service when it consumes the maven-bom generated by kork. The issue details are available in given link.
https://docs.google.com/document/d/1Ck4KeoB1ER0aQUTnf3e-x-M3i2Ur0It7YaaxEMiMXls/edit

To resolve this issue while upgrading gate service with spring v2.3.x, we must require the spring dependency management gradle plugin.

* Revert "fix(dependency): Introducing spring dependency management gradle plugin"

This reverts commit b3b2c9e.

* fix(dependency): Issue with jackson-bom and kotlin-bom version conflict resolution while upgrading the spring-boot 2.3.x.

The root cause of this issue is uncontrolled conflict resolution of jackson-bom and kotlin-bom dependency version imported from external maven BOM provided by kork-bom, as per the gradle documentation https://docs.gradle.org/6.9.1/userguide/platforms.html#sub:bom_import, we can use gradle enforcedPlatform closure as part of the implementation to strictly adhere the versions of direct and transitive dependencies imported BOM.

implementation(enforcedPlatform("io.spinnaker.kork:kork-bom:$korkVersion"))

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1519)

Co-authored-by: root <root@92ccf252161e>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1520)

Co-authored-by: root <root@8e952337838c>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1521)

Co-authored-by: root <root@e15a7ff336aa>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1522)

Co-authored-by: root <root@00e70a80e3dd>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1523)

Co-authored-by: root <root@4cdab225ba98>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1524)

Co-authored-by: root <root@2bf6cb9b4370>

* chore(dependencies): Autobump fiatVersion (spinnaker#1528)

Co-authored-by: root <root@69a0efe35201>

* chore(ci): GHA - container image and apt package build & push (spinnaker#1529)

to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* chore(ci): GHA - simplify build versioning (spinnaker#1530)

- collapse version info gathering steps into single `build_variables` step
- collapse version info parts into single string and use everywhere.
- use short git sha: `(git rev-parse --short HEAD)`

Note `build.yml` versioning is not compatible with Debian package
building as gradle plugin enforces `^[0-9]+`.
We don't publish master branch or release-* branches to GAR apt
repository though.
Prefixing the version with `<tag>-dev-` or something and publishing Debian
packages is possible but may pollute `apt-cache policy spinnaker-rosco` output
and overall be unnecessary with regular releases.

* fix(plugins-test): try harder for the version of versionNotSupportedPlugin to actually not be supported (spinnaker#1532)

Before this, a gate version >= 2.0.0 would cause versionNotSupportedPlugin to get used,
causing tests to fail, and making it impossible to e.g. release gate.

* chore(cit): GHA - plugin builds require SemVer (spinnaker#1531)

I wanted to avoid confusion between a git tag `X` building version `X` and a
master or release branch building version `X-dev-*` but it seems unavoidable.
SemVer is required by plugins.
See constraint: https://github.com/spinnaker/kork/blob/5dc6bb98615667f1b4f3e18445c1651d773c9f6b/kork-plugins/src/main/kotlin/com/netflix/spinnaker/kork/plugins/SpinnakerServiceVersionManager.kt#L47

changes:
- fetch full git repository so that we can access previous tag in branch.
  Convert `release.yml` to this method instead of `run: git.. --unshallow`.
- use previous git tag as start of version string. Cut the 'v' prefix from the
  tag, 'v1.2.3' -> '1.2.3' as required for Plugins (and Debians fwiw):
  `Caused by: Unexpected character 'LETTER(v)' at position '0', expecting '[DIGIT]'`
- append `-dev-<branch_name|'pr'>` to designate that it is not an
  official version. The short git SHA and date time are NOT present on
  release versions (eg: 1.2.3) so that also differs.
- do this version setting in `pr.yml` as well so we might pick up version
  issues in PR's and not just at merge.

* chore(dependencies): Autobump korkVersion (spinnaker#1536)

Co-authored-by: root <root@3f84b148af75>

* chore(dependencies): Autobump spinnakerGradleVersion (spinnaker#1537)

Co-authored-by: root <root@87e31b5c1a0c>

* feat(credentials): Update account type discriminator (spinnaker#1533)

This normalizes the type discriminator in account definitions to match that of account credentials instances (i.e., the type of CredentialsDefinition instance and Credentials instance are both specified through the "type" property).

This also removes redundant authorization annotations that are better enforced by AccountDefinitionService in Clouddriver.

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* resolved  AuthConfig file conflict.

Co-authored-by: Calvin Tse <sam.calvin.tse@gmail.com>
Co-authored-by: David Byron <dbyron@salesforce.com>
Co-authored-by: Emmanouil Katefidis <e.katefidis@outlook.com>
Co-authored-by: Justin Field <justin.field@armory.io>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Thomas Swanson <tomswanson4444@gmail.com>
Co-authored-by: Cameron Fieber <cfieber@netflix.com>
Co-authored-by: spinnakerbot <spinbot@spinnaker.io>
Co-authored-by: root <root@265713e22300>
Co-authored-by: root <root@ca7e6659c3b3>
Co-authored-by: root <root@7e927306ac3d>
Co-authored-by: Cristhian Castaneda <ccastanedarivera@gmail.com>
Co-authored-by: root <root@01b270ba5f94>
Co-authored-by: root <root@73790eeb50df>
Co-authored-by: root <root@a229fbda21b3>
Co-authored-by: Matt <6519811+mattgogerly@users.noreply.github.com>
Co-authored-by: Matt Sicker <boards@gmail.com>
Co-authored-by: Cameron Motevasselani <cmotevasselani@gmail.com>
Co-authored-by: Cameron Motevasselani <cameron@armory.io>
Co-authored-by: Matt Sicker <msicker@apple.com>
Co-authored-by: root <root@d38215f0a8da>
Co-authored-by: Chris Phillips <4722632+chris-h-phillips@users.noreply.github.com>
Co-authored-by: kskewes-sf <96093759+kskewes-sf@users.noreply.github.com>
Co-authored-by: Sandesh <sandeshjainhyd@gmail.com>
Co-authored-by: root <root@92ccf252161e>
Co-authored-by: root <root@8e952337838c>
Co-authored-by: root <root@e15a7ff336aa>
Co-authored-by: root <root@00e70a80e3dd>
Co-authored-by: root <root@4cdab225ba98>
Co-authored-by: root <root@2bf6cb9b4370>
Co-authored-by: root <root@69a0efe35201>
Co-authored-by: David Byron <82477955+dbyron-sf@users.noreply.github.com>
Co-authored-by: root <root@3f84b148af75>
Co-authored-by: root <root@87e31b5c1a0c>
osoriano pushed a commit to osoriano/keel that referenced this pull request Sep 2, 2023
…ker#2010)

to Google Artifact Registry

see: spinnaker/rosco#841

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants