Moving k8s-artifacts-gcslogs bucket to its own project

[riaankleinhans]

ii.coop request access to the k8s-artifacts-gcslogs to work on K8s Infra WG projects
Granting access with the correct level of permission proof to be difficult.

If the bucket is in its own project it need not pick up projectOwners
Bindings is a cleaner solution to accomplish

Fixes issue: #1945
Refers to PR:
Read access to read access k8s-artifacts-gcslogs #1966
Enable GCS logs #904

[riaankleinhans]

/assign @ameukam @spiffxp
/cc @hh

[riaankleinhans]

k8s-artifacts-gcslogs bucket access available to ii
Thank you @spiffxp @ameukam

[ameukam]

/reopen
I'll consider this done when #2031 is merged. I plan to do the transition to the dedicated project next week.

[k8s-ci-robot]

@ameukam: Reopened this issue.

In response to this:

/reopen
I'll consider this done when #2031 is merged. I plan to do the transition to the dedicated project next week.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[spiffxp]

/assign @hh
I'm assigning @hh for awareness, but I think this is something that @ameukam and I can handle. We just might someone from ii to confirm that they are capable of polling/regenerating their reports at the end of this.

My sketch of a plan:

• switch all access log settings to point to the new bucket (gcp/bash/prod: ensure gcs logs for prod buckets #2680)
• copy the old access logs to the proper location in the new bucket (Moving k8s-artifacts-gcslogs bucket to its own project  #1968 (comment))
• ask ii to ensure polling/reporting works with the new bucket location
• drop old data, IAM resources (service accounts and/or bindings)
• verify that all PII data goes solely into the one project, and that only members authorized to handle PII have access

[ameukam]

We will need to migrate the BigQuery datasets used for images pulling reports.
@Riaankl @bernokl Can you provide us the list of datasets that need to be migrated ?

[riaankleinhans]

@ameukam I cleaned up the k8s-infra-ii-sandbox with only 2 data sets remaining:

• riaan_data_store
• etl_script_generated_set_prod
  These we need for reports

[ameukam]

@Riaankl Do you need those buckets to be migrated :

cloud alpha storage ls --project k8s-infra-ii-sandbox
gs://artifacts.k8s-infra-ii-sandbox.appspot.com/
gs://export-c2e4nmc5jmg9n5nacc60/
gs://ii_bq_scratch_dump/
gs://k8s-infra-ii-sandbox-bb-test/

Apart from all this, is there anything else that needs to be migrated ?
cc @bernokl

[riaankleinhans]

@Riaankl Do you need those buckets to be migrated :

cloud alpha storage ls --project k8s-infra-ii-sandbox
gs://artifacts.k8s-infra-ii-sandbox.appspot.com/
gs://export-c2e4nmc5jmg9n5nacc60/
gs://ii_bq_scratch_dump/
gs://k8s-infra-ii-sandbox-bb-test/

@ameukam Yes, please.

[spiffxp]

/milestone v1.23

[spiffxp]

/remove-priority important-longterm
/priority important-soon

[spiffxp]

Migrating files from the old bucket to the new bucket via

gsutil -m rsync gs://k8s-artifacts-gcslogs gs://k8s-infra-artifacts-gcslogs

This looks like it will take ~3h and transfer 2.2TB of data across ~325k files

[spiffxp]

Done migrating data over

[323.5k/323.5k files][  2.2 TiB/  2.2 TiB] 100% Done 671.9 MiB/s ETA 00:00:00
Operation completed over 323.5k objects/2.2 TiB.

[spiffxp]

@BobyMCbobs @Riaankl can you confirm you're reading from gs://k8s-infra-artifacts-gcslogs and that it's safe to delete gs://k8s-artifacts-gcslogs?

[BobyMCbobs]

@BobyMCbobs @Riaankl can you confirm you're reading from gs://k8s-infra-artifacts-gcslogs
Yes! Permissions are sorted, thank you 😃

and that it's safe to delete gs://k8s-artifacts-gcslogs?
I will have to defer back to @Riaankl for Monday (NZDT), to ensure things are all pulled across and referenced correctly.

Cheers, @spiffxp!

[riaankleinhans]

@BobyMCbobs now that you have pipeline off log data establish we do not need any older data.

[spiffxp]

I don't follow if the above comment means we're OK to delete the old bucket, can you please clarify?

[riaankleinhans]

I don't follow if the above comment means we're OK to delete the old bucket, can you please clarify?

From my side, yes you can delete, Thank you.

[spiffxp]

$ time gsutil -mq rm -r gs://k8s-artifacts-gcslogs

real	51m27.627s
user	28m47.162s
sys	12m52.428s

[spiffxp]

/close
Saw an audit PR confirm the bucket is gone. Calling this done.

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Saw an audit PR confirm the bucket is gone. Calling this done.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.