releng: Configure kubernetes-build prototype jobs to use the GCB builder #14788

justaugustus · 2019-10-15T21:18:36Z

With #14747 merged and kubernetes/release#900 in flight, we'll have the ability to submit Google Cloud Build (GCB) jobs, which mimic the kubernetes_build scenario that runs in the ci-kubernetes-build and ci-kubernetes-build-fast test jobs.

The intent here is to vet this new test setup, while also starting to seed build artifacts in the CNCF-sponsored k8s-staging-release-test GCP project.

/hold

Doing this so we can leverage credentials that actually have access to
write to the k8s-staging-release-test GCS buckets.

ref: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-shadow-build/1184196943560052740, https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-shadow-build-fast/1184206760995459074

W1015 20:56:48.455] Run: ('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--release-kind=kubernetes', '--bucket=k8s-staging-release-test', '--docker-registry=gcr.io/k8s-staging-release-test', '--extra-publish-file=k8s-master', '--allow-dup')
I1015 20:56:48.560] push-build.sh: BEGIN main on 23f1240c-ef86-11e9-90f3-324215531d4f Tue Oct 15 20:56:48 UTC 2019
I1015 20:56:48.560] 
I1015 20:56:48.604] 
I1015 20:56:48.609] push-build.sh is running a *REAL* push!!
I1015 20:56:48.615] 
I1015 20:56:48.618] 
I1015 20:56:48.621] ================================================================================
I1015 20:56:48.626] CHECK PREREQUISITES
I1015 20:56:48.629] ================================================================================
I1015 20:56:48.633] 
I1015 20:56:48.646] Checking/setting cloud tools: OK
I1015 20:56:50.750] Check release bucket k8s-staging-release-test: 
I1015 20:56:50.750] push-build.sh::main(): release::gcs::check_release_bucket k8s-staging-release-test
I1015 20:56:52.646] Checking write access to bucket k8s-staging-release-test: 
I1015 20:56:52.647] push-build.sh::release::gcs::check_release_bucket(): touch /tmp/push-build.sh-gcs-write.280970
I1015 20:56:52.652] 
I1015 20:56:52.653] push-build.sh::release::gcs::check_release_bucket(): /google-cloud-sdk/bin/gsutil cp /tmp/push-build.sh-gcs-write.280970 gs://k8s-staging-release-test
I1015 20:56:53.693] Copying file:///tmp/push-build.sh-gcs-write.280970 [Content-Type=application/octet-stream]...
I1015 20:56:53.784] / [0 files][    0.0 B/    0.0 B]                                                
AccessDeniedException: 403 pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com does not have storage.objects.create access to k8s-staging-release-test/push-build.sh-gcs-write.280970.
I1015 20:56:53.912] FAILED: You do not have access/write permission on k8s-staging-release-test. Unable
I1015 20:56:53.912] to continue.
I1015 20:56:53.915] FAILED
I1015 20:56:53.918] 
I1015 20:56:53.921] push-build.sh: DONE main on 23f1240c-ef86-11e9-90f3-324215531d4f Tue Oct 15 20:56:53 UTC 2019 in 5s
W1015 20:56:53.941] Traceback (most recent call last):
W1015 20:56:53.941]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 165, in <module>
W1015 20:56:53.942]     main(ARGS)
W1015 20:56:53.942]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 137, in main
W1015 20:56:53.942]     check(args.push_build_script, *push_build_args)
W1015 20:56:53.942]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 32, in check
W1015 20:56:53.942]     subprocess.check_call(cmd)
W1015 20:56:53.942]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W1015 20:56:53.942]     raise CalledProcessError(retcode, cmd)
W1015 20:56:53.942] subprocess.CalledProcessError: Command '('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--release-kind=kubernetes', '--bucket=k8s-staging-release-test', '--docker-registry=gcr.io/k8s-staging-release-test', '--extra-publish-file=k8s-master', '--allow-dup')' returned non-zero exit status 1
E1015 20:56:53.943] Command failed
I1015 20:56:53.943] process 507 exited with code 1 after 56.9m
E1015 20:56:53.943] FAIL: ci-kubernetes-shadow-build

Signed-off-by: Stephen Augustus saugustus@vmware.com

justaugustus · 2019-10-21T20:27:43Z

I think this is ready for review:
/assign @Katharine @tpepper @calebamiles
cc: @kubernetes/release-engineering

Katharine · 2019-10-21T20:40:56Z

/lgtm

I'll approve when the corresponding PR merges.

tpepper · 2019-10-21T21:25:20Z

config/jobs/image-pushing/k8s-staging-release-test.yaml

+  annotations:
+    fork-per-release: "true"
+    fork-per-release-generic-suffix: "true"
+    testgrid-dashboards: sig-release-shadow-master-blocking


The use of the word "shadow" seems a bit opaque once you come at this PR from the perspective of seeing it on testgrid. Is there a more plainly descriptive name it might have?

Maybe sig-release-prototype-master-blocking?

I like that. Will make the changes shortly.

tpepper · 2019-10-22T00:52:34Z

lgtm

The "image-builder" image now supports submitting "--no-source" GCB builds. This PR leverages the new builder image to mimic the kubernetes-build scenario via GCB instead of using the long-deprecated bootstrap image. Signed-off-by: Stephen Augustus <saugustus@vmware.com>

k8s-ci-robot · 2019-10-22T17:15:44Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justaugustus, Katharine

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/image-pushing/OWNERS~~ [Katharine]
~~config/jobs/kubernetes/sig-release/OWNERS~~ [Katharine,justaugustus]
~~config/testgrids/kubernetes/sig-release/OWNERS~~ [Katharine,justaugustus]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2019-10-22T17:18:39Z

@justaugustus: Updated the job-config configmap in namespace default using the following files:

key k8s-staging-release-test.yaml using file config/jobs/image-pushing/k8s-staging-release-test.yaml
key shadow-builds.yaml using file ``

In response to this:

With #14747 merged and kubernetes/release#900 in flight, we'll have the ability to submit Google Cloud Build (GCB) jobs, which mimic the kubernetes_build scenario that runs in the ci-kubernetes-build and ci-kubernetes-build-fast test jobs.

The intent here is to vet this new test setup, while also starting to seed build artifacts in the CNCF-sponsored k8s-staging-release-test GCP project.

/hold

Doing this so we can leverage credentials that actually have access to
write to the k8s-staging-release-test GCS buckets.

ref: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-shadow-build/1184196943560052740, https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-kubernetes-shadow-build-fast/1184206760995459074

W1015 20:56:48.455] Run: ('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--release-kind=kubernetes', '--bucket=k8s-staging-release-test', '--docker-registry=gcr.io/k8s-staging-release-test', '--extra-publish-file=k8s-master', '--allow-dup')
I1015 20:56:48.560] push-build.sh: BEGIN main on 23f1240c-ef86-11e9-90f3-324215531d4f Tue Oct 15 20:56:48 UTC 2019
I1015 20:56:48.560] 
I1015 20:56:48.604] 
I1015 20:56:48.609] push-build.sh is running a *REAL* push!!
I1015 20:56:48.615] 
I1015 20:56:48.618] 
I1015 20:56:48.621] ================================================================================
I1015 20:56:48.626] CHECK PREREQUISITES
I1015 20:56:48.629] ================================================================================
I1015 20:56:48.633] 
I1015 20:56:48.646] Checking/setting cloud tools: OK
I1015 20:56:50.750] Check release bucket k8s-staging-release-test: 
I1015 20:56:50.750] push-build.sh::main(): release::gcs::check_release_bucket k8s-staging-release-test
I1015 20:56:52.646] Checking write access to bucket k8s-staging-release-test: 
I1015 20:56:52.647] push-build.sh::release::gcs::check_release_bucket(): touch /tmp/push-build.sh-gcs-write.280970
I1015 20:56:52.652] 
I1015 20:56:52.653] push-build.sh::release::gcs::check_release_bucket(): /google-cloud-sdk/bin/gsutil cp /tmp/push-build.sh-gcs-write.280970 gs://k8s-staging-release-test
I1015 20:56:53.693] Copying file:///tmp/push-build.sh-gcs-write.280970 [Content-Type=application/octet-stream]...
I1015 20:56:53.784] / [0 files][    0.0 B/    0.0 B]                                                
AccessDeniedException: 403 pr-kubekins@kubernetes-jenkins-pull.iam.gserviceaccount.com does not have storage.objects.create access to k8s-staging-release-test/push-build.sh-gcs-write.280970.
I1015 20:56:53.912] FAILED: You do not have access/write permission on k8s-staging-release-test. Unable
I1015 20:56:53.912] to continue.
I1015 20:56:53.915] FAILED
I1015 20:56:53.918] 
I1015 20:56:53.921] push-build.sh: DONE main on 23f1240c-ef86-11e9-90f3-324215531d4f Tue Oct 15 20:56:53 UTC 2019 in 5s
W1015 20:56:53.941] Traceback (most recent call last):
W1015 20:56:53.941]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 165, in <module>
W1015 20:56:53.942]     main(ARGS)
W1015 20:56:53.942]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 137, in main
W1015 20:56:53.942]     check(args.push_build_script, *push_build_args)
W1015 20:56:53.942]   File "/workspace/./test-infra/jenkins/../scenarios/kubernetes_build.py", line 32, in check
W1015 20:56:53.942]     subprocess.check_call(cmd)
W1015 20:56:53.942]   File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
W1015 20:56:53.942]     raise CalledProcessError(retcode, cmd)
W1015 20:56:53.942] subprocess.CalledProcessError: Command '('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--release-kind=kubernetes', '--bucket=k8s-staging-release-test', '--docker-registry=gcr.io/k8s-staging-release-test', '--extra-publish-file=k8s-master', '--allow-dup')' returned non-zero exit status 1
E1015 20:56:53.943] Command failed
I1015 20:56:53.943] process 507 exited with code 1 after 56.9m
E1015 20:56:53.943] FAIL: ci-kubernetes-shadow-build

Signed-off-by: Stephen Augustus saugustus@vmware.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 15, 2019

k8s-ci-robot requested review from cblecker and chases2 October 15, 2019 21:18

justaugustus force-pushed the shadow-build branch 2 times, most recently from e427bc3 to ce58f9a Compare October 21, 2019 06:11

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 21, 2019

justaugustus changed the title ~~releng: Move shadow kubernetes builds to test-infra-trusted cluster~~ releng: Configure shadow kubernetes builds to use the GCB builder image Oct 21, 2019

justaugustus force-pushed the shadow-build branch from ce58f9a to d5a0c8c Compare October 21, 2019 20:11

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 21, 2019

justaugustus force-pushed the shadow-build branch from d5a0c8c to d89d1bb Compare October 21, 2019 20:26

k8s-ci-robot assigned calebamiles, Katharine and tpepper Oct 21, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 21, 2019

justaugustus mentioned this pull request Oct 21, 2019

Add support for running Kubernetes builds via Google Cloud Build kubernetes/release#900

Merged

tpepper suggested changes Oct 21, 2019

View reviewed changes

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 21, 2019

justaugustus force-pushed the shadow-build branch from d89d1bb to ec1094a Compare October 22, 2019 00:43

k8s-ci-robot added the area/testgrid label Oct 22, 2019

justaugustus changed the title ~~releng: Configure shadow kubernetes builds to use the GCB builder image~~ releng: Configure kubernetes-build prototype jobs to use the GCB builder Oct 22, 2019

justaugustus force-pushed the shadow-build branch from ec1094a to c2e1035 Compare October 22, 2019 17:10

Katharine approved these changes Oct 22, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 22, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 22, 2019

k8s-ci-robot merged commit 315f04b into kubernetes:master Oct 22, 2019

k8s-ci-robot added this to the v1.17 milestone Oct 22, 2019

This was referenced Oct 22, 2019

releng: Move prototype Kubernetes builds to test-infra-trusted cluster #14922

Merged

Establish Kubernetes build/stage/release process on k8s-infra kubernetes/release#911

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

releng: Configure kubernetes-build prototype jobs to use the GCB builder #14788

releng: Configure kubernetes-build prototype jobs to use the GCB builder #14788

Uh oh!

justaugustus commented Oct 15, 2019 •

edited

Loading

Uh oh!

justaugustus commented Oct 21, 2019

Uh oh!

Katharine commented Oct 21, 2019 •

edited

Loading

Uh oh!

tpepper Oct 21, 2019

Uh oh!

tpepper Oct 22, 2019

Uh oh!

justaugustus Oct 22, 2019

Uh oh!

tpepper commented Oct 22, 2019

Uh oh!

k8s-ci-robot commented Oct 22, 2019

Uh oh!

k8s-ci-robot commented Oct 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

releng: Configure kubernetes-build prototype jobs to use the GCB builder #14788

releng: Configure kubernetes-build prototype jobs to use the GCB builder #14788

Uh oh!

Conversation

justaugustus commented Oct 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

justaugustus commented Oct 21, 2019

Uh oh!

Katharine commented Oct 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tpepper Oct 21, 2019

Choose a reason for hiding this comment

Uh oh!

tpepper Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

justaugustus Oct 22, 2019

Choose a reason for hiding this comment

Uh oh!

tpepper commented Oct 22, 2019

Uh oh!

k8s-ci-robot commented Oct 22, 2019

Uh oh!

k8s-ci-robot commented Oct 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

justaugustus commented Oct 15, 2019 •

edited

Loading

Katharine commented Oct 21, 2019 •

edited

Loading