Print cloud build logs when the task fails

[Bobgy]

Fixes #1978

/area testing
/assign @Ark-kun

---

This change is 

[Ark-kun]

Is there any particular reason to poll the service instead of just running it synchronously? The timeout can be specified using the --timeout 600s argument or in the cloudbuild.yaml.
If you do not want to print the logs unless there is error, you can write them to a file and only print if there is a problem.

[Ark-kun]

/lgtm

[Bobgy]

@Ark-kun I didn't realize this wasn't clear, let me add a comment later. It's meant to save time. When we are waiting for cloud build, we can deploy a cluster already. That saves ~4min.

[Ark-kun]

/lgtm

[Ark-kun]

When we are waiting for cloud build, we can deploy a cluster already. That saves ~4min.

I see. I often use the shell parallelism for such purposes.
Example:

{gcoud clusters create && echo "$?" > /tmp/cluster_exit_code ;} &
cluster_pid=$!
gcloud builds submit ...
wait $cluster_pid
if [ "$(cat /tmp/cluster_exit_code)" != "0" ]; then
  exit 1
fi

[Bobgy]

Thanks, that's awesome. Learned something. I will change the implementation like that.

…

On Sat, Aug 31, 2019, 6:52 AM Alexey Volkov ***@***.***> wrote: When we are waiting for cloud build, we can deploy a cluster already. That saves ~4min. I see. I often use the shell parallelism for such purposes. Example: {gcoud clusters create && echo "$?" > /tmp/cluster_exit_code ;} & cluster_pid=$! gcloud builds submit ... wait $cluster_pid if [ "$(cat /tmp/cluster_exit_code)" != "0" ]; then exit 1 fi — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1989?email_source=notifications&email_token=ABF2LVLENW24WAXPWNOPQ3DQHGJBZA5CNFSM4IR7MZHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5S3ZSQ#issuecomment-526761162>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABF2LVPRBOIN5PPQDE3FL33QHGJBZANCNFSM4IR7MZHA> .

[Bobgy]

@Ark-kun Thanks for the advice, but I don't feel I have enough time on this at the moment. After thinking about the refactor for a while. I realized it isn't as trivial as I thought.

• There are two cloud build jobs + cluster setup that needs to run in parallel, so we have to make two of them async --- just making cluster setup async is not enough.
• When we try to make cloud build jobs async, we also need to wait for build_image.sh
• When I was implementing upgrade_tests: [Testing] KFP standalone test infra for upgradability #1971, it has a different workflow that makes it more complex to refactor.
• Like before, we don't want to print cloud build logs unless it fails.

For all of the above, I think having a current working version is enough. I don't quite want to spend the extra several hours to refactor now.

But I learned bash parallel features this time and I will start with this approach next time when I need to make something new or change something.

Thoughts?

[Ark-kun]

/lgtm

[Bobgy]

Thanks. Can you grant approval too?
I don't think I have permission.
/approve

[Bobgy]

@Ark-kun

[Ark-kun]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Ark-kun, Bobgy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• test/OWNERS [Ark-kun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Bobgy]

Thanks!

We got a chance to see how logs are printed when builds fail.

ERROR: Analysis of target '//backend/src/apiserver:apiserver' failed; build aborted: no such package '@com_github_spf13_viper//': failed to fetch com_github_spf13_viper: # cd .; git clone https://github.com/spf13/viper /root/.cache/bazel/_bazel_root/ff9dc353908781674f376ac4c88da873/external/com_github_spf13_viper
Cloning into '/root/.cache/bazel/_bazel_root/ff9dc353908781674f376ac4c88da873/external/com_github_spf13_viper'...
error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
2019/09/03 23:37:50 exit status 128
INFO: Elapsed time: 106.560s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (39 packages loaded, 6149 targets configured)
      
FAILED: Build did NOT complete successfully (39 packages loaded, 6149 targets configured)
The command '/bin/sh -c bazel build -c opt --action_env=PATH --define=grpc_no_ares=true backend/src/apiserver:apiserver' returned a non-zero code: 1
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1

/retest