-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
574bb6f
commit b3a965c
Showing
1 changed file
with
110 additions
and
12 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,30 +1,128 @@ | ||
| # my-component | ||
| # Project Migration - Tables Data | ||
|
|
||
| [](https://travis-ci.com/keboola/my-component) | ||
| [](https://github.com/keboola/app-project-migrate-tables-data/actions/workflows/push.yml) | ||
|
|
||
| > Fill in description | ||
| This application is used to migrate Storage data from one project to another. It is designed to be used in the Keboola Connection environment. | ||
|
|
||
| # Usage | ||
| ## Usage | ||
| The application migrates tables data from one project to another. If a bucket does not exist, it will be created. Similarly, if a table does not exist, it will be created. | ||
|
|
||
| > fill in usage instructions | ||
| ### SAPI mode | ||
| In SAPI mode, the application migrates data using the Storage API of Keboola Connection. | ||
|
|
||
| ### Database mode | ||
| In Database mode, the application migrates data directly between databases. This mode is significantly faster. Please note the following conditions: | ||
| - Replication from the source database must be allowed. | ||
| - Both Snowflake accounts must be in the same organization. | ||
| - The Snowflake user must have the `ACCOUNTADMIN` role. | ||
|
|
||
| For allowing replication, run the following SQL in source Snowflake account: | ||
| ```sql | ||
| ALTER DATABASE {{SOURCE_DATABASE_NAME}} ENABLE REPLICATION TO ACCOUNTS {{DESTINATION_ACCOUNT_REGION}}.{{DESTINATION_ACCOUNT_NAME}}; | ||
| ``` | ||
|
|
||
| If these conditions are not fulfilled, please use SAPI mode. | ||
|
|
||
| ## Configuration | ||
| The configuration `config.json` contains following properties: | ||
|
|
||
| - `parameters` - object (required): Configuration of the application | ||
| - `mode` - string (optional): `sapi`(default value) or `database` | ||
| - `dryRun` - boolean (optional): If set to `true`, the application will only simulate the migration. | ||
| - `sourceKbcUrl` - string (required): URL of the source project | ||
| - `sourceKbcToken` - string (required): Storage API token of the source project | ||
| - `tables` - array (optional): List of tables to migrate. If not set, all tables will be migrated. | ||
| - `db` - object (optional in `database` mode and extends `db` object in `image_parameters`): | ||
| - `host` - string (required): Snowflake host | ||
| - `username` - string (required): Snowflake username | ||
| - `#password` - string (required): Snowflake password | ||
| - `warehouse` - string (required): Snowflake warehouse | ||
| - `image_parameters` - object (require): Includes from Developer Portal | ||
| - `db` - object (required in `database` mode): | ||
| - `host` - string (required): Snowflake host | ||
| - `username` - string (required): Snowflake username | ||
| - `#password` - string (required): Snowflake password | ||
| - `warehouse` - string (required): Snowflake warehouse | ||
|
|
||
| ## Example of Configurations | ||
|
|
||
| ### DRY run | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "mode": "sapi", | ||
| "dryRun": true, | ||
| "sourceKbcUrl": "https://connection.keboola.com/", | ||
| "#sourceKbcToken": "SOURCE_KBC_TOKEN" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### SAPI mode | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "mode": "sapi", | ||
| "sourceKbcUrl": "https://connection.keboola.com/", | ||
| "#sourceKbcToken": "SOURCE_KBC_TOKEN" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Database mode | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "mode": "database", | ||
| "sourceKbcUrl": "https://connection.keboola.com/", | ||
| "#sourceKbcToken": "SOURCE_KBC_TOKEN" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Database mode with your own Snowflake | ||
|
|
||
| ```json | ||
| { | ||
| "parameters": { | ||
| "mode": "database", | ||
| "sourceKbcUrl": "https://connection.keboola.com/", | ||
| "#sourceKbcToken": "SOURCE_KBC_TOKEN", | ||
| "db": { | ||
| "host": "SNOWFLAKE_HOST", | ||
| "username": "SNOWFLAKE_USERNAME", | ||
| "#password": "SNOWFLAKE_PASSWORD", | ||
| "warehouse": "SNOWFLAKE_WAREHOUSE" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Development | ||
|
|
||
| Clone this repository and init the workspace with following command: | ||
|
|
||
| ``` | ||
| git clone https://github.com/keboola/my-component | ||
| cd my-component | ||
| git clone https://github.com/keboola/app-project-migrate-tables-data | ||
| cd app-project-migrate-tables-data | ||
| docker-compose build | ||
| docker-compose run --rm dev composer install --no-scripts | ||
| ``` | ||
|
|
||
| Run the test suite using this command: | ||
| Save .env file with following content: | ||
|
|
||
| ``` | ||
| docker-compose run --rm dev composer tests | ||
| SOURCE_CLIENT_URL= | ||
| SOURCE_CLIENT_TOKEN= | ||
| DESTINATION_CLIENT_URL= | ||
| DESTINATION_CLIENT_TOKEN= | ||
| ``` | ||
|
|
||
| # Integration | ||
|
|
||
| For information about deployment and integration with KBC, please refer to the [deployment section of developers documentation](https://developers.keboola.com/extend/component/deployment/) | ||
| Run the test suite using this command: | ||
|
|
||
| ``` | ||
| docker-compose run --rm dev composer tests | ||
| ``` |