Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deploy prow to k8s-infra-prow cluster #7141

Merged
merged 12 commits into from
Aug 22, 2024
Merged

deploy prow to k8s-infra-prow cluster #7141

merged 12 commits into from
Aug 22, 2024
+55,165 −0

Conversation

upodroid
Copy link
Member

@upodroid upodroid commented Aug 8, 2024

This is the prow specific piece of #7127

I copied the manifests from https://github.com/kubernetes/test-infra/tree/master/config/prow/cluster and adjusted a few things:

  1. I consolidated the prow compoment files in to a single file. so deck-service, deck-rbac and deck-deployment are now just deck.
  2. I commented out a few services that we don't use
  3. The metrics will be handled separately by using Managed Prometheus and visualising it in Grafana https://github.com/knative/infra/blob/002a2bd25f5b3488c258baeb93b260e34d8956ee/prow/cluster/control-plane/200-monitoring.yaml
  4. Prow will be deployed to the prow namespace instead of default. (I need to adjust the EKS IAM roles)
  5. The controllers need to be scaled down before this PR is merged.
  6. The secrets needs to be loaded into the project and their manifests rewritten

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Aug 8, 2024
@k8s-ci-robot k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. labels Aug 8, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 8, 2024
@upodroid upodroid mentioned this pull request Aug 8, 2024
Closed
@upodroid
deploy prow to k8s-infra-prow cluster
067c28c
@upodroid
fix secrets

Verified

This commit was signed with the committer’s verified signature.
targos Michaël Zasso
GPG key ID: 770F7A9A5AE15600
Verified
Learn about vigilant mode
d1a5dd0
@upodroid
add remaining secrets
Loading
Loading status checks…
5d75817
xmudrii
xmudrii reviewed Aug 15, 2024
View reviewed changes
kubernetes/apps/prow.yaml
- clusters:
selector:
matchLabels:
clusterType: 'prow'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have prow-control-plane and prow-build as Argo will be used for build clusters as well eventually. That way we avoid deploying stuff into wrong cluster by accident.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each cluster will have its own prow folder to deploy prow specific resources.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I see, then it's all good.

kubernetes/apps/prow.yaml Outdated
server: "{{ .server }}"
project: default
source:
path: kubernetes/{{ .name }}/prow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't path supposed to be gke-prow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{{.name}} is based on the clusterName defined in ArgoCD.

https://github.com/kubernetes/k8s.io/blob/main/kubernetes/gke-utility/argocd/clusters.yaml#L4

@upodroid upodroid changed the title [WIP] deploy prow to k8s-infra-prow cluster deploy prow to k8s-infra-prow cluster Aug 16, 2024
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2024
@upodroid
deploy gateway/httproute for prow
Loading
Loading status checks…
f938d0d
@michelle192837 michelle192837 mentioned this pull request Aug 21, 2024
Closed
44 tasks
@upodroid
use a dummy configmap
Loading
Loading status checks…
881ce07
@upodroid
delete halogen
71dcd3d
@upodroid
add monitoring and switch to default ns
Loading
Loading status checks…
c32a048
@upodroid
add the prow cluster kubeconfig
Loading
Loading status checks…
9ae2699
@upodroid
scale up prow
Loading
Loading status checks…
c26a864
xmudrii
xmudrii reviewed Aug 22, 2024
View reviewed changes
Copy link
Member

@xmudrii xmudrii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two final nits. Can't comment much on the Prow manifests, but given everything is up and running, I'm sure we're fine. :D

kubernetes/gke-prow/prow/hook.yaml
Comment on lines +55 to +61
# AWS_ variables needed to assume role to access the prow-build-cluster EKS cluster.
- name: AWS_ROLE_ARN
value: arn:aws:iam::468814281478:role/Prow-EKS-Admin
- name: AWS_WEB_IDENTITY_TOKEN_FILE
value: /var/run/secrets/aws-iam-token/serviceaccount/token
- name: AWS_REGION
value: us-east-2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Random note/blurb: we should look into if there's a better way to handle this. I'm wondering how we would handle it if we had multiple Prow build clusters on AWS.

Copy link
Member Author

@upodroid upodroid Aug 22, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have multiple AWS clusters. We allow this role arn:aws:iam::468814281478:role/Prow-EKS-Admin to access the cluster

kubernetes/gke-prow/prow/kustomization.yaml
kind: Kustomization
namespace: default

resources:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit: would be nice to sort this file A-Z, gateway.yaml and monitoring.yaml are in the wrong place.

xmudrii
xmudrii reviewed Aug 22, 2024
View reviewed changes
kubernetes/gke-prow/prow/crds/prowjob-crd.yaml Outdated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we apply this file? I don't see it being mentioned in kustomization.yaml unless I missed it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check the prow.yaml argo app

@upodroid upodroid added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Aug 22, 2024
@upodroid
update argocd config
Loading
Loading status checks…
4d3858b
@upodroid
Copy link
Member Author

This is ready to be merged

BenTheElder
BenTheElder reviewed Aug 22, 2024
View reviewed changes
Copy link
Member

@BenTheElder BenTheElder left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/hold

NOTE: we're going to have reduced availability to fix things for a bit, I am out tomorrow and probably some portion of the next two weeks, Arnaud is out and Dims is out already.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2024
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 22, 2024
@upodroid
Copy link
Member Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 22, 2024
@k8s-ci-robot k8s-ci-robot merged commit 258cfde into kubernetes:main Aug 22, 2024
3 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.32 milestone Aug 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BenTheElder BenTheElder

@xmudrii xmudrii

@cjwagner cjwagner

Assignees

@BenTheElder BenTheElder

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/k8s-infra Categorizes an issue or PR as relevant to SIG K8s Infra. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
v1.32
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@upodroid @k8s-ci-robot @BenTheElder @xmudrii