Build a fully serverless APIs with Google Cloud Platform
Devfest Paris 2020 presentation https://docs.google.com/presentation/d/1cArhRnq9Q_YB2g-_-OGav5iP-TVv4d1BJtgRCBQAtf0
Steps
- Open Cloud Shell
- Setup environment variables
export GCP_PROJECT=[your-project-id]
export REGION=europe-west1
- Configure context for your project
gcloud config set project $GCP_PROJECT
gcloud config set compute/region $REGION
- Configure default Cloud Run mode (Fully managed mode)
gcloud config set run/platform managed
- Configure default Cloud Run and Functons deploy default region
gcloud config set run/region $REGION
gcloud config set functions/region $REGION
- Enable Apis
gcloud services enable run.googleapis.com \
cloudfunctions.googleapis.com \
endpoints.googleapis.com \
firebaserules.googleapis.com \
firestore.googleapis.com \
iap.googleapis.com \
cloudbuild.googleapis.com
- Pin the frequently used products in the codelab
- Cloud Run
- Cloud Functions
- App Engine
- Endpoints
- Datastore
- Storage
- Container Registry
- As App Engine has been activated, this has automatiquely created :
- Datastore instance
- Two buckets ($GCP_PROJECT.appspot.com and staging.$GCP_PROJECT.appspot.com) --> we will be using the first one
export GCP_BUCKET=$GCP_PROJECT.appspot.com
- Copy an existing datastore export into your project bucket
gsutil cp -r gs://serverless-apis-with-gcp-2020/export gs://$GCP_BUCKET
Import sample data into your datastore instance
gcloud datastore import gs://$GCP_BUCKET/export/export.overall_export_metadata
- Create the ESP Service Account
gcloud iam service-accounts create esp-sa --display-name='ESP Service Account'
- Add the required IAM roles the ESP Service Account
gcloud projects add-iam-policy-binding $GCP_PROJECT --member="serviceAccount:esp-sa@$GCP_PROJECT.iam.gserviceaccount.com" --role="roles/viewer"
gcloud projects add-iam-policy-binding $GCP_PROJECT --member="serviceAccount:esp-sa@$GCP_PROJECT.iam.gserviceaccount.com" --role="roles/servicemanagement.serviceController"
- Deploy the Extensible Service Proxy Container to Cloud Run with the previously created service account as identity
gcloud run deploy endpoints-runtime-serverless \
--image=gcr.io/endpoints-release/endpoints-runtime-serverless:1.44 \
--service-account="esp-sa@$GCP_PROJECT.iam.gserviceaccount.com" \
--allow-unauthenticated
Note the generated url looks like this: https://endpoints-runtime-serverless-[random]q-ew.a.run.app
- Clone Github Repository
git clone https://github.com/Sfeir/serverless-apis-with-gcp.git
- Explore the application file app.go and the associated docker file
cd ~/serverless-apis-with-gcp/run
cat app.go
cat Dockerfile
- Build the image and push it to the container registry
gcloud builds submit -t gcr.io/$GCP_PROJECT/app-inventory .
- Check that the image is stored to GCR TOOLS > Container Registry > Images
- (Optional) Enable Vulnerability Scanning TOOLS > Container Registry > Settings Click on the blue button 'Enable Vulnerability scanning' You should re-push the image to run the scanning
docker push gcr.io/$GCP_PROJECT/app-inventory
- Deploy the previously built image to Cloud Run
gcloud run deploy cloud-run-api \
--image=gcr.io/$GCP_PROJECT/app-inventory \
--allow-unauthenticated \
--set-env-vars PROJECT_ID=$GCP_PROJECT
Note the Cloud Run generated url looks like:
https://cloud-run-api-[random]-ew.a.run.app
https://cloud-run-api-[random]-ew.a.run.app?year=2018
https://cloud-run-api-[random]-ew.a.run.app?year=2019
https://cloud-run-api-[random]-ew.a.run.app?year=2020
- Remove public access to the Cloud Run API
gcloud run services remove-iam-policy-binding cloud-run-api \
--member="allUsers" \
--role="roles/run.invoker"
- Setup IAM Policy to allow only requests from ESP
gcloud run services add-iam-policy-binding cloud-run-api \
--member="serviceAccount:esp-sa@$GCP_PROJECT.iam.gserviceaccount.com" \
--role="roles/run.invoker"
- Deploy the cloud function code using the NodeJs 8 runtime
cd ~/serverless-apis-with-gcp/functions
gcloud functions deploy cloud-functions-api \
--runtime=nodejs8 \
--trigger-http \
--entry-point=appInventory \
--allow-unauthenticated
Note the Cloud Function urls that you can try to verify that everything is OK:
https://europe-west1-[GCP_PROJECT].cloudfunctions.net/cloud-functions-api
https://europe-west1-[GCP_PROJECT].cloudfunctions.net/cloud-functions-api?year=2018
https://europe-west1-[GCP_PROJECT].cloudfunctions.net/cloud-functions-api?year=2019
https://europe-west1-[GCP_PROJECT].cloudfunctions.net/cloud-functions-api?year=2020
- Remove public access to the Cloud Functions API
gcloud functions remove-iam-policy-binding cloud-functions-api \
--member="allUsers" \
--role="roles/cloudfunctions.invoker"
- Setup IAM to allow only requests from ESP
gcloud functions add-iam-policy-binding cloud-functions-api \
--member="serviceAccount:esp-sa@$GCP_PROJECT.iam.gserviceaccount.com" \
--role="roles/cloudfunctions.invoker"
- Deploy the App Engine code using the Python 3.7 runtime
cd ~/serverless-apis-with-gcp/appengine
gcloud app deploy
Note the App Engine url:
https://[GCP_PROJECT].appspot.com/
https://[GCP_PROJECT].appspot.com/?year=2018
https://[GCP_PROJECT].appspot.com/?year=2019
https://[GCP_PROJECT].appspot.com/?year=2020
- Setup Identity-Aware Proxy (IAP) to allow only requests from ESP
Burger Menu > Security > Identity-Aware Proxy
Configure Consent Screen
> Choose external
> Choose public
> Application name --> Application Inventory
> Optionaly you can upload a logo for the app
> Save your changes
Burger Menu > Security > Identity-Aware Proxy
Turn-on IAP by activating the radio button in the iap column
Try to access again and this time you will be denied!
Select 'App Engine App'
Click 'Add Member' button on the right
- New members : Your Email Address
- Select a role : Cloud IAP > IAP-secured Web App User
Try to access again (May take a few seconds to take effect) and this time you will be authorized!
Now grant the same access to the ESP service account and remove your account.
Note the App Engine url:
https://[GCP_PROJECT].appspot.com/
https://[GCP_PROJECT].appspot.com/?year=2018
https://[GCP_PROJECT].appspot.com/?year=2019
https://[GCP_PROJECT].appspot.com/?year=2020
- Update the specification file
cd ~/serverless-apis-with-gcp/swagger
nano app-inventory-api.yaml
Replace the following placeholders:
- ESP_URL : the cloud run service 'endpoints-runtime-serverless' url (be careful without https://)
- CLOUD_FUNCTION_URL : the cloud function 'cloud-functions-api' url
- CLOUD_RUN_URL : the cloud run service 'cloud-run-api' url
- APP_ENGINE_URL : The App Engine application url
- Client_ID :
- Burger Menu > APIs & Services > credentials
- Under the 'OAuth 2.0 Client IDs' copy the 'client id' of 'IAP-App-Engine-app'
(to save the file in nano use CTRL O, to quit CTRL X)
- Deploy the specification to Cloud Endpoints
gcloud endpoints services deploy app-inventory-api.yaml
If everything is okay you should get an output similar to this :
Service Configuration [2020-02-03r0] uploaded for service [endpoints-runtime-serverless-tpkdhd4z7q-uc.a.run.app]
Save the service name 'endpoints-runtime-serverless-[random]-ew.a.run.app' that we will be using in the next step.
export ENDPOINT_SERVICE_NAME=endpoints-runtime-serverless-[random]-uc.a.run.app
- Update the ESP with the Endpoint service name
gcloud run services update endpoints-runtime-serverless \
--set-env-vars ENDPOINTS_SERVICE_NAME=$ENDPOINT_SERVICE_NAME \
--project $GCP_PROJECT
- Test access to the different versions of API using the ESP runtime URL
https://endpoints-runtime-serverless-[random]-ew.a.run.app/run
https://endpoints-runtime-serverless-[random]-ew.a.run.app/functions
https://endpoints-runtime-serverless-[random]-ew.a.run.app/appengine
- Configures basic authentication with an API key by adding this section to the end of the spec file
security:
- api_key: []
securityDefinitions:
api_key:
type: "apiKey"
name: "key"
in: "query"
- Deploy the updated specification to Cloud Endpoints
gcloud endpoints services deploy app-inventory-api.yaml
- Try again to access the API and you should get this error
{
"code": 16,
"message": "Method doesn't allow unregistered callers (callers without established identity). Please use API Key or other form of API consumer identity to call this API.",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "service_control"
}
]
}
- Create an API key for your project
Burger Menu > APIs & Services > credentials
> + CREATE CREDENTIALS
> Select 'API Key'
> Copy The generated 'API Key' in order to use it in the url
https://endpoints-runtime-serverless-[random]-ew.a.run.app/run?key=[API Key]
> Close the popup
- Define an api metric
x-google-management:
metrics:
# Define a metric for read requests.
- name: "read-requests"
displayName: "Read requests"
valueType: INT64
metricKind: DELTA
- Define the metric limit
quota:
limits:
# Define the limit or the read-requests metric.
- name: "read-limit"
metric: "read-requests"
unit: "1/min/{project}"
values:
STANDARD: 100
- The final 'x-google-management' section should be :
x-google-management:
metrics:
# Define a metric for read requests.
- name: "read-requests"
displayName: "Read requests"
valueType: INT64
metricKind: DELTA
quota:
limits:
# Define the limit on the read-requests metric.
- name: "read-limit"
metric: "read-requests"
unit: "1/min/{project}"
values:
STANDARD: 100
- Apply the quota by adding 'x-google-quota' to the '/appengine' section in the same level with x-google-backend
x-google-quota:
metricCosts:
read-requests: 20
- The final '/appengine' section should be :
"/appengine":
get:
summary: "Get apps deployed to App Engine"
operationId: "app-inventory-appengine"
x-google-backend:
address: "https://codelab-project-XX-20200214.appspot.com"
jwt_audience: 671771450352-79a9ihrtgggm309n6n0g5ga64utctlq2.apps.googleusercontent.com
x-google-quota:
metricCosts:
read-requests: 20
- Deploy the updated specification to Cloud Endpoints
gcloud endpoints services deploy app-inventory-api.yaml
- Verify that the quota is working by calling many times until getting this error (After five tries) :
{
"code": 8,
"message": "Quota exceeded for quota metric 'get_requests' and limit 'get-limit' of service 'endpoints-runtime-serverless-tpkdhd4z7q-uc.a.run.app' for consumer 'project_number:671771450352'.",
"details": [
{
"@type": "type.googleapis.com/google.rpc.DebugInfo",
"stackEntries": [],
"detail": "internal"
}
]
}
- Burger Menu > Endpoints > Developer Portal
- Click on the blue button "Create portal" to generate a portal for the APIs in your project (this will take 5 to 10 minutes)
- When it's ready please visit https://endpointsportal.codelab-project-XX-20200214.cloud.goog/