Skip to content

MySQL ClickPipe: GCP doc #3735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Currently, ingesting data from MySQL to ClickHouse Cloud via ClickPipes is in Pr
:::


You can use ClickPipes to ingest data from your source MySQL database into ClickHouse Cloud. The source MySQL database can be hosted on-premises or in the cloud such as Amazon RDS or Aurora.
You can use ClickPipes to ingest data from your source MySQL database into ClickHouse Cloud. The source MySQL database can be hosted on-premises or in the cloud.

## Prerequisites {#prerequisites}

Expand All @@ -34,6 +34,8 @@ To get started, you first need to make sure that your MySQL database is set up c

2. [Amazon Aurora MySQL](./mysql/source/aurora)

3. [Cloud SQL for MySQL](./mysql/source/gcp)

3. [Amazon RDS MariaDB](./mysql/source/rds_maria)

Once your source MySQL database is set up, you can continue creating your ClickPipe.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
title: 'Aurora MySQL source setup guide'
---

import rds_backups from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/rds-backups.png';
import rds_backups from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/rds-backups.png';

Check notice on line 8 in docs/integrations/data-ingestion/clickpipes/mysql/source/aurora.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.SentenceLength

Suggestion: Improve readability by using fewer than 25 words in this sentence.
import parameter_group_in_blade from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/parameter_group_in_blade.png';
import security_group_in_rds_mysql from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/security-group-in-rds-mysql.png';
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
Expand Down
84 changes: 84 additions & 0 deletions docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
sidebar_label: 'Cloud SQL For MySQL '
description: 'Step-by-step guide on how to set up Cloud SQL for MySQL as a source for ClickPipes'
slug: /integrations/clickpipes/mysql/source/gcp
title: 'Cloud SQL for MySQL source setup guide'
---

import gcp_pitr from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/gcp-pitr.png';
import gcp_mysql_flags from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/gcp-mysql-flags.png';
import gcp_mysql_ip from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/gcp-mysql-ip.png';
import gcp_mysql_edit_button from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/gcp-mysql-edit-button.png';
import gcp_mysql_cert from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/gcp-mysql-cert.png';
import rootca from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/gcp/rootca.png';
import Image from '@theme/IdealImage';

# Cloud SQL for MySQL source setup guide

This is a step-by-step guide on how to configure your Cloud SQL for MySQL instance for replicating its data via the MySQL ClickPipe.

## Enable binary log retention {#enable-binlog-retention-gcp}
The binary log is a set of log files that contain information about data modifications made to an MySQL server instance, and binary log files are required for replication.

### Enable binary logging via PITR{#enable-binlog-logging-gcp}
The PITR feature determines whether binary logging is turned on or off for MySQL in Google Cloud. It can be set in the Cloud console, by editing your Cloud SQL instance and scrolling down to the below section.

<Image img={gcp_pitr} alt="Enabling PITR in Cloud SQL" size="lg" border/>

Setting the value to a reasonably long value depending on the replication use-case is advisable.

If not already configured, make sure to set these in the database flags section by editing the Cloud SQL:
1. `binlog_expire_logs_seconds` to a value >= `86400` (1 day).
2. `binlog_row_metadata` to `FULL`
3. `binlog_row_image` to `FULL`

To do this, click on the `Edit` button in the top right corner of the instance overview page.
<Image img={gcp_mysql_edit_button} alt="Edit button in GCP MySQL" size="lg" border/>

Then scroll down to the `Flags` section and add the above flags.

<Image img={gcp_mysql_flags} alt="Setting binlog flags in GCP" size="lg" border/>

## Configure a database user {#configure-database-user-gcp}

Connect to your Cloud SQL MySQL instance as the root user and execute the following commands:

1. Create a dedicated user for ClickPipes:

```sql
CREATE USER 'clickpipes_user'@'host' IDENTIFIED BY 'some-password';
```

2. Grant schema permissions. The following example shows permissions for the `clickpipes` database. Repeat these commands for each database and host you want to replicate:

```sql
GRANT SELECT ON `clickpipes`.* TO 'clickpipes_user'@'host';
```

3. Grant replication permissions to the user:

```sql
GRANT REPLICATION CLIENT ON *.* TO 'clickpipes_user'@'%';
GRANT REPLICATION SLAVE ON *.* TO 'clickpipes_user'@'%';
```

## Configure network access {#configure-network-access-gcp-mysql}

If you want to restrict traffic to your Cloud SQL instance, please add the [documented static NAT IPs](../../index.md#list-of-static-ips) to the whitelisted IPs of your Cloud SQL MySQL instance.
This can be done either by editing the instance or by heading over to the `Connections` tab in the sidebar in Cloud Console.

<Image img={gcp_mysql_ip} alt="IP whitelisting in GCP MySQL" size="lg" border/>

## Download and Use Root CA certificate {#download-root-ca-certificate-gcp-mysql}
To connect to your Cloud SQL instance, you need to download the root CA certificate.

1. Go to your Cloud SQL instance in the Cloud Console.
2. Click on `Connections` in the sidebar.
3. Click on the `Security` tab.
4. In the `Manage server CA certificates` section, click on the `DOWNLOAD CERTIFICATES` button at the bottom.

<Image img={gcp_mysql_cert} alt="Downloading GCP MySQL Cert" size="lg" border/>

5. In the ClickPipes UI, upload the downloaded certificate when creating a new MySQL ClickPipe.

<Image img={rootca} alt="Using GCP MySQL Cert" size="lg" border/>
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
title: 'RDS MySQL source setup guide'
---

import rds_backups from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/rds-backups.png';
import rds_backups from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/rds-backups.png';

Check notice on line 8 in docs/integrations/data-ingestion/clickpipes/mysql/source/rds.md

View workflow job for this annotation

GitHub Actions / vale

ClickHouse.SentenceLength

Suggestion: Improve readability by using fewer than 25 words in this sentence.
import parameter_group_in_blade from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/parameter_group_in_blade.png';
import security_group_in_rds_mysql from '@site/static/images/integrations/data-ingestion/clickpipes/mysql/source/rds/security-group-in-rds-mysql.png';
import edit_inbound_rules from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/rds/edit_inbound_rules.png';
Expand Down
4 changes: 4 additions & 0 deletions scripts/aspell-dict-file.txt
Original file line number Diff line number Diff line change
Expand Up @@ -1003,3 +1003,7 @@ clickpipes
clickpipes
--docs/integrations/data-ingestion/clickpipes/mysql/source/rds_maria.md--
clickpipes
--docs/integrations/data-ingestion/clickpipes/mysql/source/gcp.md--
PITR
pitr
rootca
3 changes: 2 additions & 1 deletion sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -729,7 +729,8 @@ const sidebars = {
items: [
"integrations/data-ingestion/clickpipes/mysql/source/rds",
"integrations/data-ingestion/clickpipes/mysql/source/aurora",
"integrations/data-ingestion/clickpipes/mysql/source/rds_maria"
"integrations/data-ingestion/clickpipes/mysql/source/rds_maria",
"integrations/data-ingestion/clickpipes/mysql/source/gcp"
],
},
"integrations/data-ingestion/clickpipes/mysql/datatypes"
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading