Skip to content

Commit

Permalink
Feat recovery.mode=import (#53)
Browse files Browse the repository at this point in the history 
Signed-off-by: Philippe Noël <21990816+philippemnoel@users.noreply.github.com>
Co-authored-by: Philippe Noël <21990816+philippemnoel@users.noreply.github.com>
  • Loading branch information
@itay-grudev @philippemnoel
itay-grudev and philippemnoel committed Nov 8, 2024
1 parent 42d77f4 commit c9793cc
Show file tree
Hide file tree
Showing 22 changed files with 523 additions and 60 deletions.
50 changes: 35 additions & 15 deletions charts/paradedb/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -204,15 +204,15 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| cluster.monitoring.customQueries | list | `[]` | Custom Prometheus metrics Will be stored in the ConfigMap |
| cluster.monitoring.customQueriesSecret | list | `[]` | The list of secrets containing the custom queries |
| cluster.monitoring.disableDefaultQueries | bool | `false` | Whether the default queries should be injected. Set it to true if you don't want to inject default queries into the cluster. |
| cluster.monitoring.enabled | bool | `true` | Whether to enable monitoring |
| cluster.monitoring.enabled | bool | `false` | Whether to enable monitoring |
| cluster.monitoring.podMonitor.enabled | bool | `true` | Whether to enable the PodMonitor |
| cluster.monitoring.podMonitor.metricRelabelings | list | `[]` | The list of metric relabelings for the PodMonitor. Applied to samples before ingestion. |
| cluster.monitoring.podMonitor.relabelings | list | `[]` | The list of relabelings for the PodMonitor. Applied to samples before scraping. |
| cluster.monitoring.prometheusRule.enabled | bool | `true` | Whether to enable the PrometheusRule automated alerts |
| cluster.monitoring.prometheusRule.excludeRules | list | `[]` | Exclude specified rules |
| cluster.postgresGID | int | `-1` | The GID of the postgres user inside the image, defaults to 26 |
| cluster.postgresUID | int | `-1` | The UID of the postgres user inside the image, defaults to 26 |
| cluster.postgresql.parameters | object | `{"cron.database_name":"postgres"}` | PostgreSQL configuration options (postgresql.conf) |
| cluster.postgresql.parameters | object | `{}` | PostgreSQL configuration options (postgresql.conf) |
| cluster.postgresql.pg_hba | list | `[]` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| cluster.postgresql.pg_ident | list | `[]` | PostgreSQL User Name Maps rules (lines to be appended to the pg_ident.conf file) |
| cluster.postgresql.shared_preload_libraries | list | `[]` | Lists of shared preload libraries to add to the default ones |
Expand Down Expand Up @@ -251,7 +251,27 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| recovery.google.bucket | string | `""` | |
| recovery.google.gkeEnvironment | bool | `false` | |
| recovery.google.path | string | `"/"` | |
| recovery.method | string | `"backup"` | Available recovery methods: * `backup` - Recovers a CNPG cluster from a CNPG backup (PITR supported) Needs to be on the same cluster in the same namespace. * `object_store` - Recovers a CNPG cluster from a barman object store (PITR supported). * `pg_basebackup` - Recovers a CNPG cluster viaa streaming replication protocol. Useful if you want to migrate databases to CloudNativePG, even from outside Kubernetes. # TODO |
| recovery.import.databases | list | `[]` | Databases to import |
| recovery.import.postImportApplicationSQL | list | `[]` | List of SQL queries to be executed as a superuser in the application database right after is imported. To be used with extreme care. Only available in microservice type. |
| recovery.import.roles | list | `[]` | Roles to import |
| recovery.import.schemaOnly | bool | `false` | When set to true, only the pre-data and post-data sections of pg_restore are invoked, avoiding data import. |
| recovery.import.source.database | string | `"paradedb"` | |
| recovery.import.source.host | string | `""` | |
| recovery.import.source.passwordSecret.create | bool | `false` | Whether to create a secret for the password |
| recovery.import.source.passwordSecret.key | string | `"password"` | The key in the secret containing the password |
| recovery.import.source.passwordSecret.name | string | `""` | Name of the secret containing the password |
| recovery.import.source.passwordSecret.value | string | `""` | The password value to use when creating the secret |
| recovery.import.source.port | int | `5432` | |
| recovery.import.source.sslCertSecret.key | string | `""` | |
| recovery.import.source.sslCertSecret.name | string | `""` | |
| recovery.import.source.sslKeySecret.key | string | `""` | |
| recovery.import.source.sslKeySecret.name | string | `""` | |
| recovery.import.source.sslMode | string | `"verify-full"` | |
| recovery.import.source.sslRootCertSecret.key | string | `""` | |
| recovery.import.source.sslRootCertSecret.name | string | `""` | |
| recovery.import.source.username | string | `""` | |
| recovery.import.type | string | `"microservice"` | One of `microservice` or `monolith.` See: https://cloudnative-pg.io/documentation/1.24/database_import/#how-it-works |
| recovery.method | string | `"backup"` | Available recovery methods: * `backup` - Recovers a CNPG cluster from a CNPG backup (PITR supported) Needs to be on the same cluster in the same namespace. * `object_store` - Recovers a CNPG cluster from a barman object store (PITR supported). * `pg_basebackup` - Recovers a CNPG cluster viaa streaming replication protocol. Useful if you want to migrate databases to CloudNativePG, even from outside Kubernetes. * `import` - Import one or more databases from an existing Postgres cluster. |
| recovery.pgBaseBackup.database | string | `"paradedb"` | Name of the database used by the application. Default: `paradedb`. |
| recovery.pgBaseBackup.owner | string | `""` | Name of the secret containing the initial credentials for the owner of the user database. If empty a new secret will be created from scratch |
| recovery.pgBaseBackup.secret | string | `""` | Name of the owner of the database in the instance to be used by applications. Defaults to the value of the `database` key. |
Expand Down Expand Up @@ -282,18 +302,18 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| type | string | `"paradedb"` | Type of the CNPG database. Available types: * `paradedb` |
| version.paradedb | string | `"0.11.0"` | We default to v0.11.0 for testing and local development |
| version.postgresql | string | `"16"` | PostgreSQL major version to use |
| poolers[].name | string | `` | Name of the pooler resource |
| poolers[].instances | number | `1` | The number of replicas we want |
| poolers[].type | [PoolerType][PoolerType] | `rw` | Type of service to forward traffic to. Default: `rw`. |
| poolers[].poolMode | [PgBouncerPoolMode][PgBouncerPoolMode] | `session` | The pool mode. Default: `session`. |
| poolers[].authQuerySecret | [LocalObjectReference][LocalObjectReference] | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].authQuery | string | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].parameters | map[string]string | `{}` | Additional parameters to be passed to PgBouncer - please check the CNPG documentation for a list of options you can configure |
| poolers[].template | [PodTemplateSpec][PodTemplateSpec] | `{}` | The template of the Pod to be created |
| poolers[].template | [ServiceTemplateSpec][ServiceTemplateSpec] | `{}` | Template for the Service to be created |
| poolers[].pg_hba | []string | `{}` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| poolers[].monitoring.enabled | bool | `false` | Whether to enable monitoring for the Pooler. |
| poolers[].monitoring.podMonitor.enabled | bool | `true` | Create a podMonitor for the Pooler. |
| poolers[].name | string | `` | Name of the pooler resource |
| poolers[].instances | number | `1` | The number of replicas we want |
| poolers[].type | [PoolerType][PoolerType] | `rw` | Type of service to forward traffic to. Default: `rw`. |
| poolers[].poolMode | [PgBouncerPoolMode][PgBouncerPoolMode] | `session` | The pool mode. Default: `session`. |
| poolers[].authQuerySecret | [LocalObjectReference][LocalObjectReference] | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].authQuery | string | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].parameters | map[string]string | `{}` | Additional parameters to be passed to PgBouncer - please check the CNPG documentation for a list of options you can configure |
| poolers[].template | [PodTemplateSpec][PodTemplateSpec] | `{}` | The template of the Pod to be created |
| poolers[].template | [ServiceTemplateSpec][ServiceTemplateSpec] | `{}` | Template for the Service to be created |
| poolers[].pg_hba | []string | `{}` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| poolers[].monitoring.enabled | bool | `false` | Whether to enable monitoring for the Pooler. |
| poolers[].monitoring.podMonitor.enabled | bool | `true` | Create a podMonitor for the Pooler. |

## Maintainers

Expand Down
14 changes: 0 additions & 14 deletions charts/paradedb/README.md.gotmpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,20 +41,6 @@ helm upgrade --atomic --install cnpg \
cnpg/cloudnative-pg
```

### Installing the CloudNativePG Operator

Skip this step if the CloudNativePG operator is already installed in your cluster. If you do not wish to monitor your cluster, omit the `--set` commands.

```bash
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --atomic --install cnpg \
--create-namespace \
--namespace cnpg-system \
--set monitoring.podMonitorEnabled=true \
--set monitoring.grafanaDashboard.create=true \
cnpg/cloudnative-pg
```

### Setting up a ParadeDB CNPG Cluster

Create a `values.yaml` and configure it to your requirements. Here is a basic example:
Expand Down
57 changes: 29 additions & 28 deletions charts/paradedb/templates/_bootstrap.tpl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
bootstrap:
initdb:
{{- with .Values.cluster.initdb }}
{{- with (omit . "postInitSQL" "postInitApplicationSQL" "postInitTemplateSQL" "owner") }}
{{- with (omit . "postInitSQL" "postInitApplicationSQL" "postInitTemplateSQL" "owner" "import") }}
{{- . | toYaml | nindent 4 }}
{{- end }}
{{- end }}
Expand Down Expand Up @@ -70,33 +70,34 @@ bootstrap:
{{- end }}

externalClusters:
- name: pgBaseBackupSource
connectionParameters:
host: {{ .Values.recovery.pgBaseBackup.source.host | quote }}
port: {{ .Values.recovery.pgBaseBackup.source.port | quote }}
user: {{ .Values.recovery.pgBaseBackup.source.username | quote }}
dbname: {{ .Values.recovery.pgBaseBackup.source.database | quote }}
sslmode: {{ .Values.recovery.pgBaseBackup.source.sslMode | quote }}
{{- if .Values.recovery.pgBaseBackup.source.passwordSecret.name }}
password:
name: {{ default (printf "%s-pg-basebackup-password" (include "cluster.fullname" .)) .Values.recovery.pgBaseBackup.source.passwordSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.passwordSecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslKeySecret.name }}
sslKey:
name: {{ .Values.recovery.pgBaseBackup.source.sslKeySecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslKeySecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslCertSecret.name }}
sslCert:
name: {{ .Values.recovery.pgBaseBackup.source.sslCertSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslCertSecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslRootCertSecret.name }}
sslRootCert:
name: {{ .Values.recovery.pgBaseBackup.source.sslRootCertSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslRootCertSecret.key }}
{{- end }}
{{- include "cluster.externalSourceCluster" (list "pgBaseBackupSource" .Values.recovery.pgBaseBackup.source) | nindent 2 }}

{{- else if eq .Values.recovery.method "import" }}
initdb:
{{- with .Values.cluster.initdb }}
{{- with (omit . "owner" "import") }}
{{- . | toYaml | nindent 4 }}
{{- end }}
{{- end }}
{{- if .Values.cluster.initdb.owner }}
owner: {{ tpl .Values.cluster.initdb.owner . }}
{{- end }}
import:
source:
externalCluster: importSource
type: {{ .Values.recovery.import.type }}
databases: {{ .Values.recovery.import.databases | toJson }}
{{ with .Values.recovery.import.roles }}
roles: {{ . | toJson }}
{{- end }}
{{ with .Values.recovery.import.postImportApplicationSQL }}
postImportApplicationSQL:
{{- . | toYaml | nindent 6 }}
{{- end }}
schemaOnly: {{ .Values.recovery.import.schemaOnly }}

externalClusters:
{{- include "cluster.externalSourceCluster" (list "importSource" .Values.recovery.import.source) | nindent 2 }}

{{- else }}
recovery:
Expand Down
33 changes: 33 additions & 0 deletions charts/paradedb/templates/_external_source_cluster.tpl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{- define "cluster.externalSourceCluster" -}}
{{- $name := first . -}}
{{- $config := last . -}}
- name: {{ first . }}
connectionParameters:
host: {{ $config.host | quote }}
port: {{ $config.port | quote }}
user: {{ $config.username | quote }}
{{- with $config.database }}
dbname: {{ . | quote }}
{{- end }}
sslmode: {{ $config.sslMode | quote }}
{{- if $config.passwordSecret.name }}
password:
name: {{ $config.passwordSecret.name }}
key: {{ $config.passwordSecret.key }}
{{- end }}
{{- if $config.sslKeySecret.name }}
sslKey:
name: {{ $config.sslKeySecret.name }}
key: {{ $config.sslKeySecret.key }}
{{- end }}
{{- if $config.sslCertSecret.name }}
sslCert:
name: {{ $config.sslCertSecret.name }}
key: {{ $config.sslCertSecret.key }}
{{- end }}
{{- if $config.sslRootCertSecret.name }}
sslRootCert:
name: {{ $config.sslRootCertSecret.name }}
key: {{ $config.sslRootCertSecret.key }}
{{- end }}
{{- end }}
6 changes: 6 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-cluster-assert.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: source-paradedb
status:
readyInstances: 1
9 changes: 9 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-cluster.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
type: postgresql
mode: "standalone"
cluster:
instances: 1
superuserSecret: source-paradedb-superuser
storage:
size: 256Mi
backups:
enabled: false
8 changes: 8 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-superuser-password.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Secret
metadata:
name: source-paradedb-superuser
type: Opaque
data:
username: "cG9zdGdyZXM="
password: "cG9zdGdyZXM="
6 changes: 6 additions & 0 deletions charts/paradedb/test/postgresql-import/01-data_write-assert.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: batch/v1
kind: Job
metadata:
name: data-write
status:
succeeded: 1
Loading

0 comments on commit c9793cc

Please sign in to comment.