Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history 
commit 9206656
Author: David Parker <parkerda@uk.ibm.com>
Date:   Tue Dec 3 14:36:31 2024 +0000

    [major] Use new OLM functions in mas-devops (#1573)

commit 3ccdbef
Author: Harsh Tamakuwala <31183552+harsh42774@users.noreply.github.com>
Date:   Mon Dec 2 18:30:04 2024 +0530

    [patch] Fixes ibmcloud_api_key issue in COS deprovision for ibmcloud (#1581)

    Co-authored-by: Harsh Tamakuwala <Harsh.Tamakuwala@ibm.com>

commit 1283c03
Author: Terence Quinn <quinnt@us.ibm.com>
Date:   Fri Nov 29 03:52:51 2024 -0600

    [patch] create unique certificate for each db2 instance (#1579)

commit e04138a
Author: David Parker <parkerda@uk.ibm.com>
Date:   Tue Nov 26 21:40:39 2024 +0000

    [patch] get_default_storage_classes should always set failed=False (#1578)

commit 7014f49
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Tue Nov 26 16:32:42 2024 +0000

    [patch]  Add estimates for cpd in readme (#1577)

commit 9933bc2
Author: David Parker <parkerda@uk.ibm.com>
Date:   Tue Nov 26 13:46:05 2024 +0000

    [patch] Use correct variable for mongo extras (#1576)

commit bb720a3
Author: David Parker <parkerda@uk.ibm.com>
Date:   Tue Nov 26 13:32:30 2024 +0000

    [patch] Set cpd_product_version  from catalog metadata (#1575)

commit ba23e83
Author: Terence Quinn <quinnt@us.ibm.com>
Date:   Mon Nov 25 15:03:03 2024 -0600

    [patch] Fix missing catalog_digest (#1574)

    Co-authored-by: David Parker <parkerda@uk.ibm.com>

commit 0f60c61
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Fri Nov 22 17:33:20 2024 +0000

    [patch] Add back restore.yml.j2 (#1572)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>

commit 739c4d0
Author: David Parker <parkerda@uk.ibm.com>
Date:   Fri Nov 22 16:02:24 2024 +0000

    [major] Overhaul backup/restore capability (#1557)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>

commit d5ef410
Author: David Parker <parkerda@uk.ibm.com>
Date:   Fri Nov 22 12:39:48 2024 +0000

    [doc] Fix formatting

commit 61272c9
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Fri Nov 22 12:27:00 2024 +0000

    [patch] fix restore for manage role (#1571)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>

commit 0a13abd
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Thu Nov 21 10:40:05 2024 +0000

    [patch] Db2 backup/restore: fix copy local to pod script (#1569)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>

commit f474509
Author: David Parker <parkerda@uk.ibm.com>
Date:   Wed Nov 20 23:43:25 2024 +0000

    [major] Move catalog & storage class metadata to python package (#1561)

commit df4fe4c
Author: Lokesh <110647904+lokesh-sreedhara@users.noreply.github.com>
Date:   Wed Nov 20 08:49:28 2024 -0500

    [patch] remove MIRRORLOGPATH folder creation  (#1568)

commit 5668f33
Author: Lokesh <110647904+lokesh-sreedhara@users.noreply.github.com>
Date:   Wed Nov 20 07:49:43 2024 -0500

    [patch] fix conflicting DB config (#1567)

commit 5c595c1
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Tue Nov 19 17:45:03 2024 +0000

    [patch] can skip db2 in br_manage playbook (#1566)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>

commit 8c8f993
Author: stonepd <stonepd@uk.ibm.com>
Date:   Tue Nov 19 16:39:18 2024 +0000

    [patch] update CP4D README, remove outdated fix code (#1565)

    Co-authored-by: Sanjay Prabhakar <sanju7216@gmail.com>

commit 5eb2cd4
Author: Sanjay Prabhakar <sanju7216@gmail.com>
Date:   Tue Nov 19 14:42:36 2024 +0000

    [patch] fix restore for core and fix tar command in restore tasks  (#1564)

    Co-authored-by: Sanjay Prabhakar <sanjay.prabhakar@uk.ibm.com>
  • Loading branch information
@karol-czarnecki
karol-czarnecki committed Dec 3, 2024
1 parent 6214f29 commit 9c7c5a7
Show file tree
Hide file tree
Showing 177 changed files with 1,637 additions and 7,606 deletions.
1 change: 0 additions & 1 deletion build/bin/copy-role-docs.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ copyDoc gencfg_workspace
copyDoc grafana
copyDoc ibm_catalogs
copyDoc ibmcloud_resource_key
copyDoc install_operator
copyDoc kafka
copyDoc key_rotation
copyDoc kmodels
Expand Down
176 changes: 0 additions & 176 deletions docs/execution-environment.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -245,148 +245,6 @@ If you need to set an environment variable then you can do this in the playbook
- ibm.mas_devops.ocp_efs
```

### backup/restore
The backup and restore tasks and playbooks provided in the `ibm.mas_devops` collection has a number of differences with other roles/playbooks that can make creating the playbook difficult. The following playbook is an example of how a backup for core playbook can be written to work with AAP, along with a rclone.conf.j2 file that should be stored in the same location as the playbook in your SCM. The playbook needs to set both the `environment` so that the `ansible_env` var is set, and for task vars to be set to override the common_vars that are looking for environment variables on the local controller (note that `environment` does not update the local controller environment so lookups using the ansible builtin `env` won't pick up those values).

This playbook requires the following survey questions to be setup:

```yaml
{
"aws_access_key_id": "$encrypted$",
"aws_secret_access_key": "$encrypted$",
"ocp_token": "$encrypted$",
"ocp_server": "https://api.your_ocp_cluster:6443"
}
```

```yaml
---
- name: "Backup/Restore MAS Core"
hosts: localhost
any_errors_fatal: true
vars:
# Define the target for backup/restore
mas_instance_id: aap1
# Define what action to perform
masbr_action: backup
# Define storage type
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
# Define what to backup/restore
masbr_job_component:
name: "core"
instance: "{{ mas_instance_id }}"
namespace: "mas-{{ mas_instance_id }}-core"
# Configure path to backup_restore tasks that should be present in the execution environment
role_path: "/usr/share/ansible/collections/ansible_collections/ibm/mas_devops/roles/suite_backup_restore"
# Set here to get into ansible_env
environment:
MASBR_STORAGE_TYPE: cloud
MASBR_STORAGE_CLOUD_RCLONE_FILE: rclone.conf
MASBR_STORAGE_CLOUD_RCLONE_NAME: masbr
MASBR_STORAGE_CLOUD_BUCKET: mas-backup
pre_tasks:
- name: "Asserts aws_access_key_id secret defined"
assert:
that: aws_access_key_id is defined
fail_msg: "aws_access_key_id not defined"
- name: "Asserts aws_secret_access_key secret defined"
assert:
that: aws_secret_access_key is defined
fail_msg: "aws_secret_access_key not defined"
# Template out the rclone.conf so no credentials are stored in git
- name: "Create rclone.conf"
template:
src: "{{ masbr_storage_cloud_rclone_file }}.j2"
dest: "{{ masbr_storage_cloud_rclone_file }}"
roles:
- ibm.mas_devops.ocp_login
tasks:
# Common checks before run tasks
# -------------------------------------------------------------------------
- name: "Before run tasks"
include_tasks:
file: "{{ role_path }}/../../common_tasks/backup_restore/before_run_tasks.yml"
vars:
_job_type: "{{ masbr_action }}"
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
# Create k8s Job to run backup/restore tasks
# -------------------------------------------------------------------------
- name: "Create k8s Job to run {{ masbr_action }} tasks"
when: masbr_create_task_job
include_tasks:
file: "{{ role_path }}/../../common_tasks/backup_restore/create_run_tasks_job.yml"
vars:
_rt_playbook_name: "br_core"
_rt_env:
- name: "MASBR_ACTION"
value: "{{ masbr_action }}"
- name: "MASBR_JOB_VERSION"
value: "{{ masbr_job_version }}"
- name: "MAS_INSTANCE_ID"
value: "{{ mas_instance_id }}"
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
# Run backup/restore tasks locally
# -------------------------------------------------------------------------
- name: "Run {{ masbr_action }} tasks"
when: not masbr_create_task_job
block:
- name: "MongoDB: {{ masbr_action }}"
include_role:
name: ibm.mas_devops.mongodb
vars:
mongodb_action: "{{ masbr_action }}"
mas_app_id: "core"
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
- name: "MAS Core namespace: {{ masbr_action }}"
include_role:
name: ibm.mas_devops.suite_backup_restore
vars:
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
```

The following file is stored in your SCM and is a template to be injected with the credentials needed during the play. This allows the file to be stored in SCM without exposing credentials.

rclone.conf.j2
```
[masbr]
type = s3
provider = Minio
endpoint = http://minio-api.apps.mydomain.com
access_key_id = "{{ aws_access_key_id }}"
secret_access_key = "{{ aws_secret_access_key }}"
region = minio
```
Troubleshooting
-------------------------------------------------------------------------------

Expand Down Expand Up @@ -459,37 +317,3 @@ Example setting role vars:
vars:
kafka_version: 3.7.0
```
One caveat to the above is the `backup/restore` tasks as these use a combination of ansible vars and expecting environment variables to be set in the `ansible_env` variable. In order to work with `the backup/restore` tasks you need to set both `environment` and role/task vars (not play vars).

Example, showing `environment` and task vars set. See [backup exmaple](#examples-of-playbooks) for more complete play:
```yaml
- name: "Backup/Restore MAS Core"
hosts: localhost
any_errors_fatal: true
vars:
# Configure path to backup_restore tasks
role_path: "/usr/share/ansible/collections/ansible_collections/ibm/mas_devops/roles/suite_backup_restore"
# Set here to get into ansible_env
environment:
MASBR_STORAGE_TYPE: cloud
MASBR_STORAGE_CLOUD_RCLONE_FILE: rclone.conf
MASBR_STORAGE_CLOUD_RCLONE_NAME: masbr
MASBR_STORAGE_CLOUD_BUCKET: mas-backup
tasks:
# Common checks before run tasks
# -------------------------------------------------------------------------
- name: "Before run tasks"
include_tasks:
file: "{{ role_path }}/../../common_tasks/backup_restore/before_run_tasks.yml"
vars:
_job_type: "{{ masbr_action }}"
masbr_storage_type: cloud
masbr_storage_cloud_rclone_file: rclone.conf
masbr_storage_cloud_rclone_name: masbr
masbr_storage_cloud_bucket: mas-backup
```
Loading

0 comments on commit 9c7c5a7

Please sign in to comment.