[FLINK-35634] Add CdcUp playground CLI scripts

README.md

@@ -28,14 +28,41 @@ and elegance of data integration via YAML to describe the data movement and tran
 The Flink CDC prioritizes efficient end-to-end data integration and offers enhanced functionalities such as 
 full database synchronization, sharding table synchronization, schema evolution and data transformation.
 
-![Flink CDC framework desigin](docs/static/fig/architecture.png)
+![Flink CDC framework design](docs/static/fig/architecture.png)
 
+### Quickstart Guide
 
+Flink CDC provides a CdcUp CLI utility to start a playground environment and run Flink CDC jobs.
+You will need to have a working Docker and Docker compose environment to use it.
+
+1. Run `git clone https://github.com/apache/flink-cdc.git --depth=1` to retrieve a copy of Flink CDC source code.
+2. Run `cd tools/cdcup/ && ./cdcup.sh init` to use the CdcUp tool to start a playground environment.
+3. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be ready.
+4. Run `./cdcup.sh mysql` to open a MySQL session, and create at least one table.
+
+```sql
+-- initialize db and table
+CREATE DATABASE cdc_playground;
+USE cdc_playground;
+CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+
+-- insert test data
+INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4, 'derrida');
+
+-- verify if it has been successfully inserted
+SELECT * FROM test_table;
+```
+
+5. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline job. You may also edit the pipeline definition file for further configurations.
+6. Run `./cdcup.sh flink` to access the Flink Web UI.
 
 ### Getting Started
 
 1. Prepare a [Apache Flink](https://nightlies.apache.org/flink/flink-docs-master/docs/try-flink/local_installation/#starting-and-stopping-a-local-cluster) cluster and set up `FLINK_HOME` environment variable.
 2. [Download](https://github.com/apache/flink-cdc/releases) Flink CDC tar, unzip it and put jars of pipeline connector to Flink `lib` directory.
+
+> If you're using macOS or Linux, you may use `brew install apache-flink-cdc` to install Flink CDC and compatible connectors quickly.
+
 3. Create a **YAML** file to describe the data source and data sink, the following example synchronizes all tables under MySQL app_db database to Doris :
   ```yaml
    source:
@@ -89,8 +116,6 @@ Try it out yourself with our more detailed [tutorial](docs/content/docs/get-star
 You can also see [connector overview](docs/content/docs/connectors/pipeline-connectors/overview.md) to view a comprehensive catalog of the
 connectors currently provided and understand more detailed configurations.
 
-
-
 ### Join the Community
 
 There are many ways to participate in the Apache Flink CDC community. The

docs/content.zh/docs/get-started/quickstart/cdc-up-quickstart-guide.md

@@ -0,0 +1,108 @@
+---
+title: "CdcUp Quickstart Guide"
+weight: 3
+type: docs
+aliases:
+- /get-started/quickstart/cdc-up-quickstart-guide
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Setting-up a CDC Pipeline Environment with CdcUp CLI
+
+Flink CDC now provides a CdcUp CLI utility to start a playground environment and run Flink CDC jobs quickly.
+
+You will need a working Docker and Docker Compose V2 environment to use it.
+
+## Step-by-step Guide
+
+1. Open a terminal and run `git clone https://github.com/apache/flink-cdc.git --depth=1` to retrieve a copy of Flink CDC source code.
+
+2. Run `cd flink-cdc/tools/cdcup/` to enter CDC directory and run `./cdcup.sh` to use the "cdc-up" tool to start a playground environment. You should see the following output:
+
+```
+Usage: ./cdcup.sh { init | up | pipeline <yaml> | flink | mysql | stop | down | help }
+
+Commands:
+    * init:
+        Initialize a playground environment, and generate configuration files.
+
+    * up:
+        Start docker containers. This may take a while before database is ready.
+
+    * pipeline <yaml>:
+        Submit a YAML pipeline job.
+
+    * flink:
+        Print Flink Web dashboard URL.
+
+    * mysql:
+        Open MySQL console.
+
+    * stop:
+        Stop all running playground containers.
+
+    * down:
+        Stop and remove containers, networks, and volumes.
+
+    * help:
+        Print this message.
+```
+
+3. Run `./cdcup.sh init`, waiting for pulling base docker image, and you should see the following output:
+
+```
+🎉 Welcome to cdc-up quickstart wizard!
+   There are a few questions to ask before getting started:
+🐿️ Which Flink version would you like to use? (Press ↑/↓ arrow to move and Enter to select)
+  1.17.2
+  1.18.1
+  1.19.1
+‣ 1.20.0
+```
+
+Use the arrow keys to navigate and press Enter to select specified Flink and CDC version, source, and sink connectors.
+
+4. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be ready. Some sink connectors (like Doris and StarRocks) need some time to initialize and ready for handling requests.
+
+5. If you're choosing MySQL as data source, you need to run `./cdcup.sh mysql` to open a MySQL session and create at least one table.
+
+For example, the following SQL commands create a database and a table, insert some test data, and verify the result:
+
+```sql
+-- initialize db and table
+CREATE DATABASE cdc_playground;
+USE cdc_playground;
+CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+
+-- insert test data
+INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4, 'derrida');
+
+-- verify if it has been successfully inserted
+SELECT * FROM test_table;
+```
+
+6. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline job. You may also edit the pipeline definition file for further configurations.
+
+7. Run `./cdcup.sh flink` and navigate to the printed URL to access the Flink Web UI.
+
+```
+$ ./cdcup.sh flink
+🚩 Visit Flink Dashboard at: http://localhost:33448
+```

docs/content/docs/get-started/quickstart/cdc-up-quickstart-guide.md

@@ -0,0 +1,108 @@
+---
+title: "CdcUp Quickstart Guide"
+weight: 3
+type: docs
+aliases:
+- /get-started/quickstart/cdc-up-quickstart-guide
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Setting-up a CDC Pipeline Environment with CdcUp CLI
+
+Flink CDC now provides a CdcUp CLI utility to start a playground environment and run Flink CDC jobs quickly.
+
+You will need a working Docker and Docker Compose V2 environment to use it.
+
+## Step-by-step Guide
+
+1. Open a terminal and run `git clone https://github.com/apache/flink-cdc.git --depth=1` to retrieve a copy of Flink CDC source code.
+
+2. Run `cd flink-cdc/tools/cdcup/` to enter CDC directory and run `./cdcup.sh` to use the "cdc-up" tool to start a playground environment. You should see the following output:
+
+```
+Usage: ./cdcup.sh { init | up | pipeline <yaml> | flink | mysql | stop | down | help }
+
+Commands:
+    * init:
+        Initialize a playground environment, and generate configuration files.
+
+    * up:
+        Start docker containers. This may take a while before database is ready.
+
+    * pipeline <yaml>:
+        Submit a YAML pipeline job.
+
+    * flink:
+        Print Flink Web dashboard URL.
+
+    * mysql:
+        Open MySQL console.
+
+    * stop:
+        Stop all running playground containers.
+
+    * down:
+        Stop and remove containers, networks, and volumes.
+
+    * help:
+        Print this message.
+```
+
+3. Run `./cdcup.sh init`, waiting for pulling base docker image, and you should see the following output:
+
+```
+🎉 Welcome to cdc-up quickstart wizard!
+   There are a few questions to ask before getting started:
+🐿️ Which Flink version would you like to use? (Press ↑/↓ arrow to move and Enter to select)
+  1.17.2
+  1.18.1
+  1.19.1
+‣ 1.20.0
+```
+
+Use the arrow keys to navigate and press Enter to select specified Flink and CDC version, source, and sink connectors.
+
+4. Run `./cdcup.sh up` to boot-up docker containers, and wait for them to be ready. Some sink connectors (like Doris and StarRocks) need some time to initialize and ready for handling requests.
+
+5. If you're choosing MySQL as data source, you need to run `./cdcup.sh mysql` to open a MySQL session and create at least one table.
+
+For example, the following SQL commands create a database and a table, insert some test data, and verify the result:
+
+```sql
+-- initialize db and table
+CREATE DATABASE cdc_playground;
+USE cdc_playground;
+CREATE TABLE test_table (id INT PRIMARY KEY, name VARCHAR(32));
+
+-- insert test data
+INSERT INTO test_table VALUES (1, 'alice'), (2, 'bob'), (3, 'cicada'), (4, 'derrida');
+
+-- verify if it has been successfully inserted
+SELECT * FROM test_table;
+```
+
+6. Run `./cdcup.sh pipeline pipeline-definition.yaml` to submit the pipeline job. You may also edit the pipeline definition file for further configurations.
+
+7. Run `./cdcup.sh flink` and navigate to the printed URL to access the Flink Web UI.
+
+```
+$ ./cdcup.sh flink
+🚩 Visit Flink Dashboard at: http://localhost:33448
+```

tools/cdcup/.gitignore

@@ -0,0 +1,63 @@
+*.gem
+*.rbc
+/.config
+/coverage/
+/InstalledFiles
+/pkg/
+/spec/reports/
+/spec/examples.txt
+/test/tmp/
+/test/version_tmp/
+/tmp/
+
+# Used by dotenv library to load environment variables.
+# .env
+
+# Ignore Byebug command history file.
+.byebug_history
+
+## Specific to RubyMotion:
+.dat*
+.repl_history
+build/
+*.bridgesupport
+build-iPhoneOS/
+build-iPhoneSimulator/
+
+## Specific to RubyMotion (use of CocoaPods):
+#
+# We recommend against adding the Pods directory to your .gitignore. However
+# you should judge for yourself, the pros and cons are mentioned at:
+# https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control
+#
+# vendor/Pods/
+
+## Documentation cache and generated files:
+/.yardoc/
+/_yardoc/
+/doc/
+/rdoc/
+
+## Environment normalization:
+/.bundle/
+/vendor/bundle
+/lib/bundler/man/
+
+# for a library or gem, you might want to ignore these files since the code is
+# intended to run in multiple environments; otherwise, check them in:
+# Gemfile.lock
+# .ruby-version
+# .ruby-gemset
+
+# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:
+.rvmrc
+
+# Used by RuboCop. Remote config files pulled in from inherit_from directive.
+# .rubocop-https?--*
+
+.idea/**
+
+# These are generated files
+cdc/**
+/docker-compose.yaml
+/pipeline-definition.yaml

tools/cdcup/Dockerfile

@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ruby:3.3-slim
+
+WORKDIR /src
+RUN apt-get update && apt-get install -y wget
+RUN gem install tty-prompt
+COPY src /src
+RUN chmod +x /src/app.rb
+ENTRYPOINT ["/src/app.rb"]

tools/cdcup/README.md

@@ -0,0 +1,30 @@
+# cdcup
+
+A `docker` (`compose`) environment on Linux / macOS is required to play with this. Ruby is **not** necessary.
+
+## `./cdcup.sh init`
+
+Initialize a playground environment, and generate configuration files.
+
+## `./cdcup.sh up`
+
+Start docker containers. Note that it may take a while before database is ready.
+
+## `./cdcup.sh pipeline <yaml>`
+
+Submit a YAML pipeline job. Before executing this, please ensure that:
+
+1. All container are running and ready for connections
+2. (For MySQL) You've created at least one database & tables to be captured
+
+## `./cdcup.sh flink`
+
+Print Flink Web dashboard URL.
+
+## `./cdcup.sh stop`
+
+Stop all running playground containers.
+
+## `./cdcup.sh down`
+
+Stop and remove containers, networks, and volumes.