MEGA-Infrastructure

The MEGA infrastructure project.

Infrastructure

The MEGA Project is hosted in Openshift and split to the following 3 stages (Openshift Projects):

1. mega-dev
   The development stage, where usually the current develop state is deployed on.

2. mega-test
   The test environment where the acceptance tests are performed.

3. mega-prod
   The production system of the MEGA Project.

The cited image illustrates the system of the MEGA Project, its involved services, interactions and zones. Each zone is protected by OAuth whereby each zone uses its own clientId.

Figure 1. Mega System

Backup

The mega databaseis backed up by a Jenkins Pipeline and stored in an Openshift PVC.

Figure 2. Mega Backup (inactive)

Important

This type of backup is not working because we cannot get our Pod-Networks of the different OCP Projects merged.

Figure 3. Mega Backup via OCP CronJob (active)

Important

The CronJobs are working only within the project they are defined in.

Download backups

The backups are stored on a PVc within the hosting Project and can be synced locally via the mega-zep-db`service.

Sync Backups to local machine

oc rsync <POD_ID>:/mega-db-backup/ .

Restore a backup

The backups are located on PVC within a project, so actually we can only restore backups to the project they were made from.

Important

First modify /apps/mega-zep-db/restore-job.yml env var DUMPFILE with the filename you want to restore which you retrieved from the mega-db-service.

Restore via OCP Job

# Then create the job, which is executed right away
./apps/ocp.sh createMegaDbRestore

# Check the pod state
oc get pods

# Check the logs of the COMPLETED job
oc logs mega-db-restore-*

# Delete the restore OCP Job
./apps/ocp.sh deleteMegaDbRestore

Manually create backup

# login to the OpenShift Console
oc login https://manage.cloud.itandtel.at --token=<token>

# create the cronjob with the current date time. Format: yyyy-MM-dd-HH-mm-ss
kubectl create job --from=cronjob/mega-db-backup mega-db-backup-<date>-<time>

!!!does only work in prod!!! In dev and test there is no cronjob

Project Resources

Dockerfiles

The cited Dockerfiles are used to build the jenkins build agents for the MEGA Project.

• agent-nodejs/Dockerfile
  The Dockerfile for building the nodejs build agent.

• agent-jdk/Dockerfile
  The Dockerfile for building the OpenJDK build agent.

S2I

Jenkins S2i

We use a custom Jenkins which is based on the official Openshift-Jenkins-Image and which is build via the S2I mechanism.

• s2i/jenkins
  The directory holding the s2i sources.

• s2i/jenkins/plugins.txt
  The plugins.txt which defines the plugins and plugins version the Mega Jenkins uses.

Configuration files

Jenkins

Jenkins is configured via the configureation-as-code plugin which allows to configure Jenkins via YAML config files.

• /config/jenkins/jenkins-config.yaml
  The YAML configuration file which configures the Jenkins instance and the MEGA Job Definitions.

Tip	The YAML configuration file references secrets via environment variables, which are injected into the container by Openshift. Because the configuration files doesn’t contain any secrets, this configuration file is part of the repository.

MEGA

The configuration for MEGA is provided via a properties file and a secret file. Each used stage needs its own configuration file to separate the configurations per stage.

• application.<STAGE>.properties
  The staged configuration file. (Part of the repository)

• application.<STAGE>.properties
  The staged secret file. (In Cloud storage)

Tip	The secret file must be provided manually. See MEGA Repository for possible configuration values.

Openshift Templates

Jenkins

For the Jenkins service we use multiple Openshift Templates which are used to define an part of the Jenkins service and its dependencies.

1. /apps/jenkins/jenkins.yaml
   The Jenkins Openshift Template defining the Jenkins service.

2. /apps/jenkins/jenkins-agent-bc.yaml
   The Openshift Template holding the definitions of the by agents used Container Images and Openshift Build Configurations.

3. /apps/jenkins/jenkins-bc.yaml
   The Openshift Template holding the Openshift Build Configurations for the custom Jenkins build.

4. /apps/jenkins/jenkins-pvc.yaml
   The Openshift Template holding the definition for the persistence storage of Jenkins in Openshift.

5. /apps/jenkins/maven-pvc.yaml
   The Openshift Template holding the definition for the persistence storage for the caching of the maven repository used by the build agents.

6. /apps/jenkins/jenkins.properties
   The properties file holding the values for the jenkins.yaml Openshift Template defined parameters.

The Jenkins Openshift Templates can be configured via the cited configuration properties.

1. JENKINS_SERVICE_NAME=jenkins
   The name used for the Jenkins service and Openshift Resources

2. JENKINS_S2I_IMAGE=docker.io/openshift/jenkins-2-centos7:v3.11 The image the custom Jenkins Build is based on

3. JENKINS_IMAGE_STREAM_TAG=jenkins-custom:latest
   The name of the Openshift image Stream for the custom Jenkins Build Container Images

4. GIT_URL=https://github.com/Gepardec/mega-infrastructure.git
   The git repository url for the Jenkins related Openshift Build Configurations

5. GIT_REF=master
   The git reference used for the Jenkins related Openshift Build Configurations

6. GIT_SECRET=github-http
   The name of the Openshift secret providing the Openshift Secret of type kubernetes.io/basic-auth

7. MAVEN_VERSION=3.8.1
   The Maven version used by the JDK build agent Container Image

8. MEMORY_LIMIT=2Gi
   The Jenkins Service assgined RAM

9. MAVEN_VERSION=3.8.1
   The Maven Version to use for the agent-jdk Jenkins Agent Pod

10. MEGA_BACKUP_PVC
    The name of the backup pvc of mega

MEGA

The MEGA Project uses 2 Openshift Templates, one for the backend and one for the frontend to integrate the MEGA services in openshift. The templates are parametrized and whereby the parameter values are provided via properties files.

• mega-zep-backend.yaml
  The backend Openshift Template

• mega-zep-backend.<STAGE>.properties
  The staged backend configuration file

• mega-zep-frontend.yaml
  The backend Openshift Template

• mega-zep-frontend.<STAGE>.properties
  The staged frontend configuration file

The backend can be configured via the cited configuration properties.

• APP=mega-zep
  The value for the app label, which is add to each created Openshift Resource

• NAME=mega-zep-backend
  The value for the service and created Openshift Resources

• SECRET=mega
  The name of the secret holding the MEGA secrets for the specific stage

• CPU=500m
  The assigned CPU resources to the service

• MEMORY=256Mi
  The assigned RAM resources to the services

• TIMEZONE=Europe/Vienna
  The timezone to set on the container

The frontend can be configured via the cited configuration properties.

• APP=mega-zep
  The value for the app label, which is add to each created Openshift Resource

• NAME=mega-zep-frontend
  The value for the service and created Openshift Resources

• CPU=500m
  The assigned CPU resources to the service

• MEMORY=256Mi
  The assigned RAM resources to the services

• TIMEZONE=Europe/Vienna
  The timezone to set on the container

The db can be configured via the cited configuration properties.

• APP=mega-zep
  The value for the app label, which is add to each created Openshift Resource

• NAME=mega-zep-db
  The value for the service and created Openshift Resources

• VERSION=10
  The version of the postgres to use

• SECRET=mega
  The name of the secret holding the MEGA secrets for the specific stage

• PVC=mega-zep-db-10
  The name of persistence volume for the database

• CPU=500m
  The assigned CPU resources to the service

• MEMORY=256Mi
  The assigned RAM resources to the services

• TIMEZONE=Europe/Vienna
  The timezone to set on the container

• DATABASE_MAX_CONNECTIONS=100
  The maximal allowed connections to the database

• DATABASE_SHARED_BUFFERS=64MB
  The maximal shared buffer size

• PGCTLTIMEOUT=150
  The pgctl timeout in seconds

Pipelines

The cited Jenkinsfiles are provided by this repository.

• /jenkins/src/release/Jenkinsfile
  The release pipeline which releases a MEGA version to a specific stage.

Scripts

The cited scripts are provided by this repository.

• /apps/ocp.sh
  This scripts holds functions which can be used to managed Jenkins and Jenkins related resources.

Setup

Secret Files

The cited secret files have to be provided in the root of this repository.

jenkins-secrets.properties

emailUser=service@gepardec.com
emailPassword=****
megaBuildWebhookToken=***

This secret is used to configure the mailing and the github webhook.

git-http.properties

username=mega-dev
password=***

This secret is used for checkout of the mega Github repositories.

Important

The password must be a generated API Token and not the account password.

The MEGA Project configuration properties are described on the MEGA MEGA Repository.

• mega-secrets.dev.properties
  The DEV stage MEGA configuration

• mega-secrets.test.properties
  The TEST stage MEGA configuration

• mega-secrets.prod.properties
  The PROD stage MEGA configuration

Google OAuth2 client setup

Each stage uses its own OAuth clientId/clientSecret and therefore each stages has to be setup in Google. See here for a description how to setup a google oauth client.

Openshift Dev Project

Secrets

# Create secrets
STAGE=dev apps/ocp.sh createMegaSecrets
apps/ocp.sh createJenkinsSecrets

# Delete secrets
apps/ocp.sh deleteMegaSecrets
apps/ocp.sh deleteJenkinsSecrets

# Delete/Create secrets
STAGE=dev apps/ocp.sh recreateMegaSecrets
apps/ocp.sh recreateJenkinsSecrets

Build Configurations

# Create all build configs
apps/ocp.sh createBuildConfigs

# Delete all build configs
apps/ocp.sh deleteBuildConfigs

# Delete/Create all build configs
apps/ocp.sh recreateBuildConfigs

Mega Backup PVC

# Create Backup PVC
apps/ocp.sh createMegaBackupPvc

# Delete Backup PVC
apps/ocp.sh deleteMegaBackupPvc

# Recreate Backup PVC
apps/ocp.sh recreateMegaBackupPvc

Important

Take care that all backups are saved before deleting or recreating the backup persistence volume. Jenkins requires the backup pvc to exist because Jenkins uses it as a volume.

Jenkins Application

# Create jenkins
apps/ocp.sh createJenkins

# Delete jenkins
apps/ocp.sh deleteJenkins

# Delete/Create jenkins
apps/ocp.sh recreateJenkins

# Create jenkins pvc
apps/ocp.sh createJenkinsPvc

# Delete jenkins pvc
apps/ocp.sh deleteJenkinsPvc

# Delete/Create jenkins pvc
apps/ocp.sh recreateJenkinsPvc

# Create maven pvc
apps/ocp.sh createMavenPvc

# Delete maven pvc
apps/ocp.sh deleteMavenPvc

# Delete/Create maven pvc
apps/ocp.sh recreateMavenPvc

MEGA database

# Create the mega db image streams
apps/ocp.sh createMegaDbIs

# Delete the mega db image streams
apps/ocp.sh deleteMegaDbIs

# Delete/Create the mega db image streams
apps/ocp.sh recreateMegaDbIs

# Create the mega db persistence volume
apps/ocp.sh createMegaDbPvc

# Delete the mega db persistence volume
apps/ocp.sh deleteMegaDbPvc

# Delete/Create the mega db persistence volume
apps/ocp.sh recreateMegaDbPvc

# Create the mega db
apps/ocp.sh createMegaDb

# Delete the mega db
apps/ocp.sh deleteMegaDb

# Delete/Create the mega db
apps/ocp.sh recreateMegaDb

Important

Ensure that the properties in ocp/jenkins.properties and env vars in apps/ocp.sh are properly setup for your needs.

Openshift Test/Prod Project

Secrets

# Create secrets
STAGE=[prod|test] apps/ocp.sh createMegaSecrets

Important

Ensure that the configuration is properly setup for the selected stage and that you have set the proper project context

Know Issues

Jenkins Pipeline

First Repository Scan cannot consume credential

After Jenkins has newly deployed the build pipelines tries to scan the github repository for branches whereby it cannot consume the assigend credential the firt time the scan is executed. Therefore, after a deployment you need to skip the first execution of the scan and restart it. The second time the crednetial is consume correctly.