Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat recovery.mode=import #53

Merged
merged 12 commits into from
Nov 7, 2024
Merged

Feat recovery.mode=import #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
9849711
Implemented recovery.method=import
itay-grudev Nov 6, 2024
8d71aef
newline
philippemnoel Nov 6, 2024
e8673b0
newline
philippemnoel Nov 6, 2024
e66efcb
newline
philippemnoel Nov 6, 2024
0ba47d1
newline
philippemnoel Nov 6, 2024
b5175c2
newline
philippemnoel Nov 6, 2024
cef9e88
newline
philippemnoel Nov 6, 2024
4c16edc
newline
philippemnoel Nov 6, 2024
19c9a3b
Bug Fix: Readme and Schema
itay-grudev Nov 6, 2024
ae7d486
Bug Fix: cluster configuration test superuserSecret
itay-grudev Nov 6, 2024
375d5d9
Merge remote-tracking branch 'origin/feat/import' into feat/import
itay-grudev Nov 6, 2024
83cf0bc
Merge branch 'dev' into feat/import
itay-grudev Nov 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 35 additions & 15 deletions charts/paradedb/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,15 +204,15 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| cluster.monitoring.customQueries | list | `[]` | Custom Prometheus metrics Will be stored in the ConfigMap |
| cluster.monitoring.customQueriesSecret | list | `[]` | The list of secrets containing the custom queries |
| cluster.monitoring.disableDefaultQueries | bool | `false` | Whether the default queries should be injected. Set it to true if you don't want to inject default queries into the cluster. |
| cluster.monitoring.enabled | bool | `true` | Whether to enable monitoring |
| cluster.monitoring.enabled | bool | `false` | Whether to enable monitoring |
| cluster.monitoring.podMonitor.enabled | bool | `true` | Whether to enable the PodMonitor |
| cluster.monitoring.podMonitor.metricRelabelings | list | `[]` | The list of metric relabelings for the PodMonitor. Applied to samples before ingestion. |
| cluster.monitoring.podMonitor.relabelings | list | `[]` | The list of relabelings for the PodMonitor. Applied to samples before scraping. |
| cluster.monitoring.prometheusRule.enabled | bool | `true` | Whether to enable the PrometheusRule automated alerts |
| cluster.monitoring.prometheusRule.excludeRules | list | `[]` | Exclude specified rules |
| cluster.postgresGID | int | `-1` | The GID of the postgres user inside the image, defaults to 26 |
| cluster.postgresUID | int | `-1` | The UID of the postgres user inside the image, defaults to 26 |
| cluster.postgresql.parameters | object | `{"cron.database_name":"postgres"}` | PostgreSQL configuration options (postgresql.conf) |
itay-grudev marked this conversation as resolved.
Show resolved Hide resolved
| cluster.postgresql.parameters | object | `{}` | PostgreSQL configuration options (postgresql.conf) |
| cluster.postgresql.pg_hba | list | `[]` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| cluster.postgresql.pg_ident | list | `[]` | PostgreSQL User Name Maps rules (lines to be appended to the pg_ident.conf file) |
| cluster.postgresql.shared_preload_libraries | list | `[]` | Lists of shared preload libraries to add to the default ones |
Expand Down Expand Up @@ -251,7 +251,27 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| recovery.google.bucket | string | `""` | |
| recovery.google.gkeEnvironment | bool | `false` | |
| recovery.google.path | string | `"/"` | |
| recovery.method | string | `"backup"` | Available recovery methods: * `backup` - Recovers a CNPG cluster from a CNPG backup (PITR supported) Needs to be on the same cluster in the same namespace. * `object_store` - Recovers a CNPG cluster from a barman object store (PITR supported). * `pg_basebackup` - Recovers a CNPG cluster viaa streaming replication protocol. Useful if you want to migrate databases to CloudNativePG, even from outside Kubernetes. # TODO |
| recovery.import.databases | list | `[]` | Databases to import |
| recovery.import.postImportApplicationSQL | list | `[]` | List of SQL queries to be executed as a superuser in the application database right after is imported. To be used with extreme care. Only available in microservice type. |
| recovery.import.roles | list | `[]` | Roles to import |
| recovery.import.schemaOnly | bool | `false` | When set to true, only the pre-data and post-data sections of pg_restore are invoked, avoiding data import. |
| recovery.import.source.database | string | `"paradedb"` | |
| recovery.import.source.host | string | `""` | |
| recovery.import.source.passwordSecret.create | bool | `false` | Whether to create a secret for the password |
| recovery.import.source.passwordSecret.key | string | `"password"` | The key in the secret containing the password |
| recovery.import.source.passwordSecret.name | string | `""` | Name of the secret containing the password |
| recovery.import.source.passwordSecret.value | string | `""` | The password value to use when creating the secret |
| recovery.import.source.port | int | `5432` | |
| recovery.import.source.sslCertSecret.key | string | `""` | |
| recovery.import.source.sslCertSecret.name | string | `""` | |
| recovery.import.source.sslKeySecret.key | string | `""` | |
| recovery.import.source.sslKeySecret.name | string | `""` | |
| recovery.import.source.sslMode | string | `"verify-full"` | |
| recovery.import.source.sslRootCertSecret.key | string | `""` | |
| recovery.import.source.sslRootCertSecret.name | string | `""` | |
| recovery.import.source.username | string | `""` | |
| recovery.import.type | string | `"microservice"` | One of `microservice` or `monolith.` See: https://cloudnative-pg.io/documentation/1.24/database_import/#how-it-works |
| recovery.method | string | `"backup"` | Available recovery methods: * `backup` - Recovers a CNPG cluster from a CNPG backup (PITR supported) Needs to be on the same cluster in the same namespace. * `object_store` - Recovers a CNPG cluster from a barman object store (PITR supported). * `pg_basebackup` - Recovers a CNPG cluster viaa streaming replication protocol. Useful if you want to migrate databases to CloudNativePG, even from outside Kubernetes. * `import` - Import one or more databases from an existing Postgres cluster. |
| recovery.pgBaseBackup.database | string | `"paradedb"` | Name of the database used by the application. Default: `paradedb`. |
| recovery.pgBaseBackup.owner | string | `""` | Name of the secret containing the initial credentials for the owner of the user database. If empty a new secret will be created from scratch |
| recovery.pgBaseBackup.secret | string | `""` | Name of the owner of the database in the instance to be used by applications. Defaults to the value of the `database` key. |
Expand Down Expand Up @@ -282,18 +302,18 @@ refer to the [CloudNativePG Documentation](https://cloudnative-pg.io/documentat
| type | string | `"paradedb"` | Type of the CNPG database. Available types: * `paradedb` |
| version.paradedb | string | `"0.11.0"` | We default to v0.11.0 for testing and local development |
| version.postgresql | string | `"16"` | PostgreSQL major version to use |
| poolers[].name | string | `` | Name of the pooler resource |
| poolers[].instances | number | `1` | The number of replicas we want |
| poolers[].type | [PoolerType][PoolerType] | `rw` | Type of service to forward traffic to. Default: `rw`. |
| poolers[].poolMode | [PgBouncerPoolMode][PgBouncerPoolMode] | `session` | The pool mode. Default: `session`. |
| poolers[].authQuerySecret | [LocalObjectReference][LocalObjectReference] | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].authQuery | string | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].parameters | map[string]string | `{}` | Additional parameters to be passed to PgBouncer - please check the CNPG documentation for a list of options you can configure |
| poolers[].template | [PodTemplateSpec][PodTemplateSpec] | `{}` | The template of the Pod to be created |
| poolers[].template | [ServiceTemplateSpec][ServiceTemplateSpec] | `{}` | Template for the Service to be created |
| poolers[].pg_hba | []string | `{}` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| poolers[].monitoring.enabled | bool | `false` | Whether to enable monitoring for the Pooler. |
| poolers[].monitoring.podMonitor.enabled | bool | `true` | Create a podMonitor for the Pooler. |
| poolers[].name | string | `` | Name of the pooler resource |
| poolers[].instances | number | `1` | The number of replicas we want |
| poolers[].type | [PoolerType][PoolerType] | `rw` | Type of service to forward traffic to. Default: `rw`. |
| poolers[].poolMode | [PgBouncerPoolMode][PgBouncerPoolMode] | `session` | The pool mode. Default: `session`. |
| poolers[].authQuerySecret | [LocalObjectReference][LocalObjectReference] | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].authQuery | string | `{}` | The credentials of the user that need to be used for the authentication query. |
| poolers[].parameters | map[string]string | `{}` | Additional parameters to be passed to PgBouncer - please check the CNPG documentation for a list of options you can configure |
| poolers[].template | [PodTemplateSpec][PodTemplateSpec] | `{}` | The template of the Pod to be created |
| poolers[].template | [ServiceTemplateSpec][ServiceTemplateSpec] | `{}` | Template for the Service to be created |
| poolers[].pg_hba | []string | `{}` | PostgreSQL Host Based Authentication rules (lines to be appended to the pg_hba.conf file) |
| poolers[].monitoring.enabled | bool | `false` | Whether to enable monitoring for the Pooler. |
| poolers[].monitoring.podMonitor.enabled | bool | `true` | Create a podMonitor for the Pooler. |
itay-grudev marked this conversation as resolved.
Show resolved Hide resolved

## Maintainers

Expand Down
14 changes: 0 additions & 14 deletions charts/paradedb/README.md.gotmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,20 +41,6 @@ helm upgrade --atomic --install cnpg \
cnpg/cloudnative-pg
```

### Installing the CloudNativePG Operator

Skip this step if the CloudNativePG operator is already installed in your cluster. If you do not wish to monitor your cluster, omit the `--set` commands.

```bash
helm repo add cnpg https://cloudnative-pg.github.io/charts
helm upgrade --atomic --install cnpg \
--create-namespace \
--namespace cnpg-system \
--set monitoring.podMonitorEnabled=true \
--set monitoring.grafanaDashboard.create=true \
cnpg/cloudnative-pg
```

philippemnoel marked this conversation as resolved.
Show resolved Hide resolved
### Setting up a ParadeDB CNPG Cluster

Create a `values.yaml` and configure it to your requirements. Here is a basic example:
Expand Down
57 changes: 29 additions & 28 deletions charts/paradedb/templates/_bootstrap.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
bootstrap:
initdb:
{{- with .Values.cluster.initdb }}
{{- with (omit . "postInitSQL" "postInitApplicationSQL" "postInitTemplateSQL" "owner") }}
{{- with (omit . "postInitSQL" "postInitApplicationSQL" "postInitTemplateSQL" "owner" "import") }}
{{- . | toYaml | nindent 4 }}
{{- end }}
{{- end }}
Expand Down Expand Up @@ -70,33 +70,34 @@ bootstrap:
{{- end }}

externalClusters:
- name: pgBaseBackupSource
connectionParameters:
host: {{ .Values.recovery.pgBaseBackup.source.host | quote }}
port: {{ .Values.recovery.pgBaseBackup.source.port | quote }}
user: {{ .Values.recovery.pgBaseBackup.source.username | quote }}
dbname: {{ .Values.recovery.pgBaseBackup.source.database | quote }}
sslmode: {{ .Values.recovery.pgBaseBackup.source.sslMode | quote }}
{{- if .Values.recovery.pgBaseBackup.source.passwordSecret.name }}
password:
name: {{ default (printf "%s-pg-basebackup-password" (include "cluster.fullname" .)) .Values.recovery.pgBaseBackup.source.passwordSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.passwordSecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslKeySecret.name }}
sslKey:
name: {{ .Values.recovery.pgBaseBackup.source.sslKeySecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslKeySecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslCertSecret.name }}
sslCert:
name: {{ .Values.recovery.pgBaseBackup.source.sslCertSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslCertSecret.key }}
{{- end }}
{{- if .Values.recovery.pgBaseBackup.source.sslRootCertSecret.name }}
sslRootCert:
name: {{ .Values.recovery.pgBaseBackup.source.sslRootCertSecret.name }}
key: {{ .Values.recovery.pgBaseBackup.source.sslRootCertSecret.key }}
{{- end }}
{{- include "cluster.externalSourceCluster" (list "pgBaseBackupSource" .Values.recovery.pgBaseBackup.source) | nindent 2 }}

{{- else if eq .Values.recovery.method "import" }}
initdb:
{{- with .Values.cluster.initdb }}
{{- with (omit . "owner" "import") }}
{{- . | toYaml | nindent 4 }}
{{- end }}
{{- end }}
{{- if .Values.cluster.initdb.owner }}
owner: {{ tpl .Values.cluster.initdb.owner . }}
{{- end }}
import:
source:
externalCluster: importSource
type: {{ .Values.recovery.import.type }}
databases: {{ .Values.recovery.import.databases | toJson }}
{{ with .Values.recovery.import.roles }}
roles: {{ . | toJson }}
{{- end }}
{{ with .Values.recovery.import.postImportApplicationSQL }}
postImportApplicationSQL:
{{- . | toYaml | nindent 6 }}
{{- end }}
schemaOnly: {{ .Values.recovery.import.schemaOnly }}

externalClusters:
{{- include "cluster.externalSourceCluster" (list "importSource" .Values.recovery.import.source) | nindent 2 }}

{{- else }}
recovery:
Expand Down
33 changes: 33 additions & 0 deletions charts/paradedb/templates/_external_source_cluster.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{{- define "cluster.externalSourceCluster" -}}
{{- $name := first . -}}
{{- $config := last . -}}
- name: {{ first . }}
connectionParameters:
host: {{ $config.host | quote }}
port: {{ $config.port | quote }}
user: {{ $config.username | quote }}
{{- with $config.database }}
dbname: {{ . | quote }}
{{- end }}
sslmode: {{ $config.sslMode | quote }}
{{- if $config.passwordSecret.name }}
password:
name: {{ $config.passwordSecret.name }}
key: {{ $config.passwordSecret.key }}
{{- end }}
{{- if $config.sslKeySecret.name }}
sslKey:
name: {{ $config.sslKeySecret.name }}
key: {{ $config.sslKeySecret.key }}
{{- end }}
{{- if $config.sslCertSecret.name }}
sslCert:
name: {{ $config.sslCertSecret.name }}
key: {{ $config.sslCertSecret.key }}
{{- end }}
{{- if $config.sslRootCertSecret.name }}
sslRootCert:
name: {{ $config.sslRootCertSecret.name }}
key: {{ $config.sslRootCertSecret.key }}
{{- end }}
{{- end }}
6 changes: 6 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-cluster-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: source-paradedb
status:
readyInstances: 1
9 changes: 9 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-cluster.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
type: postgresql
mode: "standalone"
cluster:
instances: 1
superuserSecret: source-paradedb-superuser
storage:
size: 256Mi
backups:
enabled: false
8 changes: 8 additions & 0 deletions charts/paradedb/test/postgresql-import/00-source-superuser-password.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
apiVersion: v1
kind: Secret
metadata:
name: source-paradedb-superuser
type: Opaque
data:
username: "cG9zdGdyZXM="
password: "cG9zdGdyZXM="
6 changes: 6 additions & 0 deletions charts/paradedb/test/postgresql-import/01-data_write-assert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
apiVersion: batch/v1
kind: Job
metadata:
name: data-write
status:
succeeded: 1
Loading
Loading