Technical troubleshooting

Introduction

This page is for documenting technical solutions or tips to technical issues related to cloud.gov, databases, CircleCI, CMS, etc.

Table of Contents

• Restaging application updates
• Service account passwords
• Hasura password access
• Updating CMS
• CircleCi pipeline faliures
• Logging

Restaging application updates

Cloud.gov frequently updates the programming language buildpacks available to customers. Buildpack updates include programming language updates and often include security fixes. Cloud.gov will send an email notification when applications need to be restaged with the subject line of "Action required: restage your application".

A rolling restage operation is the quickest way to upgrade without incurring downtime. You may still want to leverage your deployment infrastructure to perform the upgrade if you have compliance requirements for redeployment operations.

You can restage your applications by opening the command line and entering the following commands:

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-nrrd

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-blog-nrrd

cf target -o doi-onrr -s dev ; cf restage --strategy rolling preview-data

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-onrr-frontend

cf target -o doi-onrr -s dev ; cf restage --strategy rolling dev-onrr-cms

cf target -o doi-onrr -s dev ; cf restage --strategy rolling preview-onrr-frontend

Service account passwords

The service account service creates unique cloud.gov credentials for each service key. Service account passwords expire every 90 days. If you see an error like:

Error Code: 403 Raw Response: {"error":"access_denied","error_description":"Your current password has expired. Please reset your password."}

Then you’ll need to delete the existing service key, recreate it, and update the username/password in your deployment scripts. For example:

cf delete-service-key my-service-account my-service-key

cf create-service-key my-service-account my-service-key

cf service-key my-service-account my-service-key

The last command will return the service account username/password pair. These steps can be used at any time to update/rotate credentials for service accounts.

It is desirable to have a single service account with access to multiple spaces. After creating a service key in the prod space, allow the same service key to access dev with the following command:

cf set-space-role <service key username> doi-onrr dev SpaceDeveloper

Hasura password access

The username and password for Hasura instances are stored in environment variables vi cf cli. To retrieve the Hasura credentials for the hasura-sandbox app, for example, take the following steps:

1. Target the dev space: cf target -s dev
2. Display environment variables: cf env hasura-sandbox
3. See HASURA_GRAPHQL_ADMIN_SECRET under the User-provided section of the output.

Updating CMS

Follow the instructions below in Circle Ci to trigger the CMS to update, this will need to be done with a new plugin or feature is added to the CMS.

1. In CircleCIi, select the main branch and click on trigger pipeline
2. Click add parameters
3. Select "boolean" for Parameter Type, in the "Name" write the command "copy-database", for value select "True"

Circleci Pipeline Failures

It can be very helpful and may be necessary to ssh to the circleci instance where a build pipeline failure occurs. Follow the guidance provided by circleci here and GitHub here to learn how to debug pipeline issues over ssh.

Logging

To enable logging of database operations in the CMS, the LOG_LEVEL environment variable must be trace. To enable, set an environment variable for the CMS application and restage the application:

cf target -s dev
cf set-env preview-onrr-cms LOG_LEVEL trace
cf restage --strategy=rolling preview-onrr-cms

When done, be sure to remove the environment variable to return the log level back to the default info.

cf unset-env preview-onrr-cms LOG_LEVEL
cf restage --strategy=rolling preview-onrr-cms