Integrating GitHub and Cloudbuild

The monorepo pattern is commonly used alongside microservice architecture. While Cloudbuild itself works nicely with monorepos, Google's Cloudbuild GitHub Integration does not. It expects a single cloudbuild.yaml file in the repo root, while in a monorepo, services are parcelled off in subfolders each containing their own separate cloudbuild.yaml with the steps needed to build that service.

Without being able to use the existing cloudbuild integration we have two problems:

• getting notifications from GitHub to Cloudbuild
• getting notifications from Cloudbuild to GitHub

This sheds some light on how we solve these problems for Tracker.

From GitHub to Cloudbuild

We use a Cloud Source repository to mirror our GitHub repo inside GCP. This is done via a webhook which is created when setting up the mirror.

When we create triggers, those triggers fire based on events from the mirrored repository.

From Cloudbuild to GitHub

With build triggers firing based on events in the mirrored repo, we have to communicate the build status back to GitHub itself. Cloudbuild publishes build status changes to a Cloud Pub/Sub topic called cloud-builds.

Subscribing to that channel, converting the events to GitHub format and sending them is the job of the cloudbuild-status-reporter. It runs in Cloud Run so we don't have to worry that it will get overwhelmed with the number of events that happen.

The image

The image in our registry was created by building an image for our project and pushing it:

git clone https://github.com/sleepycat/cloudbuild-status-reporter.git && cd cloudbuild-status-reporter
docker build -t gcr.io/track-compliance/cloudbuild-status-reporter .
docker push gcr.io/track-compliance/cloudbuild-status-reporter

Deploying the image

Any updates to the env vars for the status-reporter will require you to redeploy. That's done like this:

gcloud beta run deploy --service-account=build-trigger-viewer@track-compliance.iam.gserviceaccount.com --platform=managed --region=us-central1 --update-env-vars=GCP_PROJECT=track-compliance,GITHUB_TOKEN=ghp_longghtoken,REPO_NAME=tracker,REPO_OWNER=canada-ca --allow-unauthenticated --image gcr.io/track-compliance/cloudbuild-status-reporter cloudbuild-status-reporter

Details

The annoying first time setup stuff is detailed in the cloudbuild-status-reporter repo and also covered in Google's documentation on triggering Cloud run from PubSub. These steps shouldn't need to be repeated.