add prow build clusters #830

spiffxp · 2020-05-01T22:32:44Z

I tried to break this down into logical commits with details in each commit, so it may be easier to review that way. Or, I included README's for the modules and clusters, which try to dump current state and how I arrived there.

This sets up three modules in infra/gcp/clusters/modules:

k8s-infra-gke-project - provisions a project to host a build cluster, using kubernetes-public as a reference
k8s-infra-gke-cluster - provisions a gke cluster with node service account and usage info dumped into a bigquery dataset
k8s-infra-gke-nodepool - provisions a gke nodepool with some defaults that are k8s-infra specific

(note: they all depend on terraform ~> 0.12.20 and google ~> 3.19.0; I have not figured out the best way to non-destructively migrate aaa to this setup)

I used those modules to create two project+cluster+nodepool combos:

k8s-infra-prow-build/prow-build: the untrusted build cluster, and service accounts for pods and boskos-janitor
k8s-infra-prow-build-trusted/prow-build-trusted: the trusted build cluster, and service accounts for pods and gcb-builder (for jobs that push to releng/staging projects)
each of these have a resources/ dir containing kubernetes resource .yaml files destined for the cluster (I deployed these manually from a cloud-shell instance)

I added e2e projects intended for the untrusted build cluster:

k8s-infra-e2e-boskos-nnn: 40 projects for boskos to manage in a k8s-infra-gce-project pool
k8s-infra-e2e-gce-project: for pinning to an e2e job for development/debugging
k8s-infra-e2e-node-e2e-project: for pinning to a node e2e job for development/debugging

I updated some of the ensure-* scripts:

allow gcb-builder service account to push to releng/staging projects
allow k8s-auditor-gcp service account to run on prow-build-trusted

I hooked the clusters up to prow.k8s.io with on-call's help:

we discovered that gs://kubernetes-release-dev won't allow non-google.com accounts to be added to iam, which will prevent us from migrating some e2e's

I migrated some jobs over:

Migrate an e2e job to k8s-infra-prow-build test-infra#17486 - migrated an e2e job over to prow-build, it ran successfully (it ran all tests, some failed, but no more than before)
Use k8s-infra-prow-build-trusted for an image-pushing job test-infra#17487 - migrated an image-pushing job over to prow-build-trusted, Silly change to trigger an image pushing job #831 triggered a build which failed... after uncovering two typos and rerunning it ran successfully
Move some release-blocking/informing jobs to k8s-infra-prow-builds test-infra#17488 - migrated over some release-master-blocking and release-master-informing jobs to prow-build
Migrate image-pushing jobs to k8s-infra-prow-build-trusted test-infra#17489 - migrated over all of the image-pushing jobs to prow-build-trusted (thought I've had to revert one due to some missing serets ref: move post-kuberentes-push-images job back to test-infra-trusted test-infra#17534)
Migrate some release-branch jobs to k8s-infra-prow-build test-infra#17560 - migrated over some release-branch jobs
jobs: migrate some generated release-branch jobs to k8s-infra-prow-build test-infra#17562 - migrated over some generated release-branch jobs

I hooked prow-build's boskos instance up to monitoring.k8s.prow.io

Add k8s-infra-prow-build's boskos to monitoring test-infra#17523 - we can see the k8s-infra-gce-project pool here: https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1&fullscreen&panelId=7
Use instance template on boskos-http dashboard test-infra#17525 - we can select which boskos instance to view http metrics from here: https://monitoring.prow.k8s.io/d/vn8ROeuWz/boskos-server-dashboard?orgId=1

I removed the old kubernetes-public/prow-build-test cluster and spiffxp-* e2e projects

Followup work I've broken out into other issues:

dumped updated audit data (companion PR Update audit #848)
identify what it will take to migrate any remaining release-master-blocking jobs (Migrate release-master-blocking jobs to k8s-infra-prow-build #841)
setup greenhouse (bazel build cache) in prow-build (opened Setup greenhouse in k8s-infra-prow-build cluster #842)
setup ghproxy in prow-build-trusted (for tools like peribolos) (opened Setup ghproxy in k8s-infra-prow-build-trusted #843)
revisited access ACLs for these clusters (vs. re-using whatever aaa did) (opened Develop better ACLs for prow build clusters and e2e projects #844)
setup any kind of continuous deployment (will need to be done from cloud-shell for now) (opened Setup postsubmit jobs to deploy prow build cluster resources #845)
figured out what to do about jobs that use gs://kubernetes-release-dev (opened Migrate jobs away from gs://kubernetes-release-dev #846)

/cc @thockin @cblecker @bartsmykla

ameukam · 2020-05-01T23:23:12Z

infra/gcp/clusters/modules/k8s-infra-gke-project/main.tf

+resource "google_project" "project" {
+  name            = var.project_name
+  project_id      = var.project_name
+  org_id          = "758905017065"          // kubernetes.io


Could be a string variable with a default value.

I started with that, but moved to hardcodes because I don't think we want to give people a choice to use other orgs/billing (in much the same way our ensure_project bash sets these for everything)

infra/gcp/clusters/modules/k8s-infra-gke-project/main.tf

spiffxp · 2020-05-02T02:03:01Z

/hold
for comment

ameukam · 2020-05-04T20:03:55Z

infra/gcp/clusters/modules/k8s-infra-gke-cluster/main.tf

+    enabled = true
+  }
+}
+resource "google_container_cluster" "test_cluster" {


I'm not sure to understand this resource. Why not define a module test-k8s-infra-gke-cluster ?

I felt like copy-pasting resource definitions between modules was more likely to fall out-of-sync than copy-pasting within the same module. Copy-paste is the only approach I can use for any resource whose lifecycle depends on a flag/environment, since terraform doesn't allow these values to be derived from variables.

infra/gcp/clusters/k8s-infra-prow-build-trusted/prow-build-trusted/00-provider.tf

spiffxp · 2020-05-05T03:24:49Z

Turns out in order to support kind-ipv6 jobs the build cluster needs to be running an Ubuntu image, since COS doesn't provide an ipv6 stack. Using terraform apply to change the configuration of the node pool, I'm curious how this will roll out.

spiffxp · 2020-05-05T03:51:16Z

terraform apply timed out with

Error: Error waiting for updating GKE node pool: timeout while waiting for state to become 'DONE' (last state: 'RUNNING', timeout: 30m0s)

Running terraform plan again shows it wants to do the same update-in-place of the nodepool as before. Meanwhile I see the operation still ongoing via gcloud. Not going to bother re-running terraform.

# gcloud container operations describe operation-1588648687093-274e5e0a --region=us-central1
detail: 'Done with 3 out of 6 nodes (50.0%): 1 being processed, 3 succeeded'
name: operation-1588648687093-274e5e0a
operationType: UPGRADE_NODES
progress:
  metrics:
  - intValue: '6'
    name: NODES_TOTAL
  - intValue: '3'
    name: NODES_DONE
  - intValue: '0'
    name: NODES_DRAINING
  - intValue: '0'
    name: NODES_UPGRADING
  - intValue: '1'
    name: NODES_CREATING
  - intValue: '0'
    name: NODES_REGISTERING
  - intValue: '3'
    name: NODES_COMPLETE
selfLink: https://container.googleapis.com/v1/projects/773781448124/locations/us-central1/operations/operation-1588648687093-274e5e0a
startTime: '2020-05-05T03:18:07.093064626Z'
status: RUNNING
targetLink: https://container.googleapis.com/v1/projects/773781448124/locations/us-central1/clusters/prow-build/nodePools/pool1-20200430220922185300000001
zone: us-central1

ameukam · 2020-05-06T20:08:21Z

We could also add support for labels and in taints k8s-infra-gke-cluster. It could be useful for greenhouse.

created 2 manual projects and 40 boskos projects

actuated via: - terraform destroy before removing the .tf files - gcloud projects delete for the e2e projects - gcloud compute addresses delete for the boskos-metrics ip

spiffxp · 2020-05-07T00:31:03Z

@ameukam

We could also add support for labels and in taints k8s-infra-gke-cluster. It could be useful for greenhouse.

I think this is a great idea, but I'd like to cap this PR off here. I'm starting to get concerned about the amount of uncommitted infra I have running live.

If we agree we're fine with the terraform structured as is, I'll open a followup issue to support labels and would welcome the help.

spiffxp · 2020-05-07T00:37:15Z

Ping for reviews. I know it's a size/XXL, if there's something I can do to help make this easier to digest, let me know

cblecker

overall lgtm

cblecker · 2020-05-07T21:06:51Z

infra/gcp/clusters/modules/k8s-infra-gke-cluster/main.tf

+// to set lifecycle.prevent_destroy to false if is_prod_cluster
+// keep prod_ and test_ identical except for "unique to " comments
+resource "google_container_cluster" "prod_cluster" {
+  count     = var.is_prod_cluster == "true" ? 1 : 0


What happens if you toggle this on the same object?

I think going from not->prod with a bare terraform apply would nuke the test resources and create the prod resources.

I think going from prod->not would leave you with two copies of the resources actuated, even if terraform "forgot" about the prod resources.

I suspect you could get out of this and just have terraform start treating a resource differently if you used terraform state mv + terraform plan to verify there were no changes to actuate

spiffxp · 2020-05-11T23:16:24Z

ping @thockin I want to make sure we get your eyes on this

thockin

I understood most of this

infra/gcp/clusters/README.md

infra/gcp/clusters/kubernetes-public/prow-build-test/main.tf

infra/gcp/clusters/modules/k8s-infra-gke-cluster/main.tf

infra/gcp/clusters/k8s-infra-prow-build/prow-build/main.tf

thockin · 2020-05-15T22:58:56Z

infra/gcp/clusters/k8s-infra-prow-build/prow-build/main.tf

+
+locals {
+  project_id              = "k8s-infra-prow-build"
+  cluster_name            = "prow-build"           // The name of the cluster defined in this file


Are we ever going to need more than one? Should the be prow-build-1 or prow-build-aaa or something?

Historically the prow team hasn't ever had more than one build cluster per GCP project.

We have here a project named prow-build and a cluster named prow-build. I guess I am skeptical that we'll never want to use some GKE change that requires rebuilding the cluster, is all. If you're confident, I'll go with it, since it's likely you who has to deal with the mess if you are wrong :)

infra/gcp/clusters/k8s-infra-prow-build/prow-build/main.tf

thockin · 2020-05-15T23:10:12Z

infra/gcp/clusters/k8s-infra-prow-build-trusted/prow-build-trusted/main.tf

+*/
+
+locals {
+  project_id              = "k8s-infra-prow-build-trusted"


Can I pick on names? I like symmetry, so I'd expect to see

k8s-infra-prow-build + k8s-infra-prow-trusted

or

k8s-infra-prow-untrusted + k8s-infra-prow-trusted

or

k8s-infra-prow-build-untrusted + k8s-infra-prow-build-trusted

Is there a reason not to?

Well. I like all of these suggestions better than what I chose. I prefer the first since the names are shortest.

The reason not to would be that renaming the project id at this point is going to involve creating a new project/cluster/nodepool combo along with the requisite coordination with prow.k8s.io oncall. The blocker at the moment is our projects being capped at billing quota.

I can file an issue to redo the trusted cluster as k8s-infra-prow-trusted/prow-trusted and use it as an opportunity to have someone shadow, or someone else go through this while I watch.

If you think the rename is worth it I'll open a ticket, WDYT?

infra/gcp/clusters/k8s-infra-prow-build-trusted/prow-build-trusted/main.tf

infra/gcp/ensure-e2e-projects.sh

make a projects dir and move all projects into it, instead of having modules be "the one dir that isn't a project" update README to follow suit

update READMEs accordingly

this has been broken out into a separate issue

spiffxp · 2020-05-16T01:59:52Z

Update PTAL

spiffxp · 2020-05-21T19:22:56Z

ping for review

thockin

Thanks!

/lgtm
/approve

k8s-ci-robot · 2020-05-22T16:23:17Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: spiffxp, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [spiffxp,thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

spiffxp · 2020-05-26T18:13:59Z

/hold cancel

k8s-ci-robot requested review from bartsmykla, cblecker and thockin May 1, 2020 22:32

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. wg/k8s-infra approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 1, 2020

ameukam reviewed May 1, 2020

View reviewed changes

infra/gcp/clusters/modules/k8s-infra-gke-project/main.tf Outdated Show resolved Hide resolved

spiffxp changed the title ~~[wip] add prow build clusters~~ add prow build clusters May 2, 2020

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 2, 2020

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 2, 2020

ameukam reviewed May 4, 2020

View reviewed changes

infra/gcp/clusters/k8s-infra-prow-build-trusted/prow-build-trusted/00-provider.tf Outdated Show resolved Hide resolved

spiffxp force-pushed the prow-build-clusters branch 6 times, most recently from de7a266 to 0b1bc3c Compare May 6, 2020 17:51

spiffxp added 7 commits May 6, 2020 14:16

add k8s resources intended for prow-build-trusted

c1b8f4e

setup a gcb-builder svcacct for prow-build-trusted

8606717

create k8s-infra-e2e-projects and add to boskos

acb0f31

created 2 manual projects and 40 boskos projects

Add READMEs for prow-build and prow-build-trusted

179a7dc

enable stackdriver in prow-build projects

17ffbd5

Migrate prow-build nodepool to ubuntu-containerd

672d113

remove prow-build-test cluster

9d9483e

actuated via: - terraform destroy before removing the .tf files - gcloud projects delete for the e2e projects - gcloud compute addresses delete for the boskos-metrics ip

spiffxp force-pushed the prow-build-clusters branch from 0b1bc3c to 9d9483e Compare May 6, 2020 21:16

spiffxp mentioned this pull request May 7, 2020

Update audit #848

Merged

cblecker reviewed May 7, 2020

View reviewed changes

thockin reviewed May 15, 2020

View reviewed changes

spiffxp added 4 commits May 15, 2020 18:00

refactor: make a projects dir

3c2e795

make a projects dir and move all projects into it, instead of having modules be "the one dir that isn't a project" update README to follow suit

refactor: drop k8s-infra prefix from modules

b834ccd

update READMEs accordingly

drop TODOs about access

e8ea80d

this has been broken out into a separate issue

add scary comments for ugly hack

0d83cf8

spiffxp mentioned this pull request May 21, 2020

Add greenhouse to k8s-infra-prow-build #885

Merged

thockin reviewed May 22, 2020

View reviewed changes

k8s-ci-robot assigned thockin May 22, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 22, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 26, 2020

k8s-ci-robot merged commit e3db4e2 into kubernetes:master May 26, 2020

spiffxp deleted the prow-build-clusters branch May 27, 2020 17:20

spiffxp added the area/prow Setting up or working with prow in general, prow.k8s.io, prow build clusters label Aug 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add prow build clusters #830

add prow build clusters #830

spiffxp commented May 1, 2020 •

edited

Loading

ameukam May 1, 2020

spiffxp May 1, 2020

spiffxp commented May 2, 2020

ameukam May 4, 2020

spiffxp May 4, 2020

spiffxp commented May 5, 2020

spiffxp commented May 5, 2020 •

edited

Loading

ameukam commented May 6, 2020

spiffxp commented May 7, 2020

spiffxp commented May 7, 2020

cblecker left a comment

cblecker May 7, 2020

spiffxp May 8, 2020

spiffxp commented May 11, 2020

thockin left a comment

thockin May 15, 2020

spiffxp May 16, 2020

thockin May 22, 2020

thockin May 15, 2020

spiffxp May 16, 2020

spiffxp May 16, 2020

thockin May 22, 2020

spiffxp commented May 16, 2020

spiffxp commented May 21, 2020

thockin left a comment

k8s-ci-robot commented May 22, 2020

spiffxp commented May 26, 2020

add prow build clusters #830

add prow build clusters #830

Conversation

spiffxp commented May 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented May 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented May 5, 2020

spiffxp commented May 5, 2020 • edited Loading

ameukam commented May 6, 2020

spiffxp commented May 7, 2020

spiffxp commented May 7, 2020

cblecker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented May 11, 2020

thockin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spiffxp commented May 16, 2020

spiffxp commented May 21, 2020

thockin left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 22, 2020

spiffxp commented May 26, 2020

spiffxp commented May 1, 2020 •

edited

Loading

spiffxp commented May 5, 2020 •

edited

Loading