Migrate kettle to k8s-infra

[spiffxp]

Part of migrating away from gcp-project k8s-gubernator: #1308

• origin gcp-project: k8s-gubernator
• origin cluster-name: g8r
• apps:
  • kettle
• repo: https://github.com/kubernetes/test-infra/tree/master/kettle

My suggestions for target:

• project: kubernetes-public
• cluster: aaa
• namespace: kettle

/wg k8s-infra
/area cluster-infra
/sig testing

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
Kettle is still running there

[spiffxp]

Migrating kettle most likely looks something like

• migrate the bigquery database kettle writes to (Migrate k8s-gubernator:builds dataset to wg-k8s-infra #1307)
• make sure the permissions kettle needs against google.com resources work for non-google.com accounts (e.g. pubsub permissions in kubernetes-jenkins)
• migrate kettle

[spiffxp]

FYI @MushuEE
given that you've been modifying kettle lately, if you happen see things that could help inform a plan for this, drop 'em here

[MushuEE]

When you say

migrate the bigquery database kettle writes to

is that to a new project? What is the:
target project and target cluster?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale

[spiffxp]

/assign @MushuEE @spiffxp
to investigate possible approaches

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[BenTheElder]

@ixdy still works at Google and still had access, despite long since not working in cloud anymore ... myself and @liggitt now have owner access to k8s-gubernator project for continuity until we can migrate it. Thanks Jeff!

This still needs to happen before the prow default cluster shutdown in August and sooner is better.

[BenTheElder]

So we still have one "g8r" cluster on 1.26.11-gke.1055000 with 3 node pools, "pool-1" (e2-highmem-16, 1 node), "pool-highmem" (n1-highmem-8, 2 nodes), "pool-large" (n1-standard-8, 0 nodes).

It is running "kettle" and "kettle-staging" deployments with one pod each.

Each of those has a PD-SSD, 3001 and 201 GB respectively.

There are some bigquery datasets in this project, build/all is 1.67 TB.

[BenTheElder]

Given initially ingest this data from the prow GCS logs, I think we should probably look at cold-starting a new instance running in AAA, just overriding the cluster/project and deploying with the existing tooling.

There's a lot to be desired around auto deployment etc however

[BenTheElder]

I think @dims has this working, one remaining item will be when we're confident this is done let Googlers know and we'll see about turning down the old instance / GCP project ... (FYI @michelle192837 @cjwagner)

[dims]

@BenTheElder i want to watch it for a week before we can call it done!

[michelle192837]

Exciting stuff! :D Thanks y'all!

[BenTheElder]

[I scaled the old cluster down to zero this week, we'll check back next week]

[dims]

thanks @BenTheElder

[dims]

https://storage.googleapis.com/k8s-triage/index.html is being updated.

and the flakes json looks good as well

❯ gsutil ls -l gs://k8s-metrics


Updates are available for some Google Cloud CLI components.  To install them,
please run:
  $ gcloud components update

       114  2024-05-02T00:05:31Z  gs://k8s-metrics/build-stats-latest.json
     10040  2024-05-02T00:04:51Z  gs://k8s-metrics/failures-latest.json
    103224  2024-05-02T00:04:20Z  gs://k8s-metrics/flakes-daily-latest.json
    204024  2024-05-02T00:05:48Z  gs://k8s-metrics/flakes-latest.json
         5  2024-05-02T00:04:09Z  gs://k8s-metrics/job-flakes-latest.json
    376585  2024-05-02T00:05:08Z  gs://k8s-metrics/job-health-latest.json
         3  2024-05-02T00:05:20Z  gs://k8s-metrics/pr-consistency-latest.json
     83496  2024-05-02T00:04:36Z  gs://k8s-metrics/presubmit-health-latest.json
         3  2024-05-02T00:06:01Z  gs://k8s-metrics/weekly-consistency-latest.json
                                 gs://k8s-metrics/build-stats/
                                 gs://k8s-metrics/failures/
                                 gs://k8s-metrics/flakes-daily/
                                 gs://k8s-metrics/flakes/
                                 gs://k8s-metrics/istio-job-flakes/
                                 gs://k8s-metrics/job-flakes/
                                 gs://k8s-metrics/job-health/
                                 gs://k8s-metrics/pr-consistency/
                                 gs://k8s-metrics/presubmit-health/
                                 gs://k8s-metrics/weekly-consistency/
TOTAL: 9 objects, 777494 bytes (759.27 KiB)

We can turn down the old cluster early next week @BenTheElder

[BenTheElder]

SGTM. At some point I'd like to turn down the bigquery datasets and anything else lingering in that project as well.

[BenTheElder]

/assign
Will plan to turn down and delete everything in the old project this week.

[ameukam]

/assign Will plan to turn down and delete everything in the old project this week.

@BenTheElder also update #1308 and close it ? 🥺

[BenTheElder]

remaining follow up will be tracked in #1308