Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PST-1956] README #17

Merged
merged 1 commit into from
Sep 25, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 110 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,128 @@
# my-component
# Project Migration - Tables Data

[![Build Status](https://travis-ci.com/keboola/my-component.svg?branch=master)](https://travis-ci.com/keboola/my-component)
[![GitHub Actions](https://github.com/keboola/app-project-migrate-tables-data/actions/workflows/push.yml/badge.svg)](https://github.com/keboola/app-project-migrate-tables-data/actions/workflows/push.yml)

> Fill in description
This application is used to migrate Storage data from one project to another. It is designed to be used in the Keboola Connection environment.

# Usage
## Usage
The application migrates tables data from one project to another. If a bucket does not exist, it will be created. Similarly, if a table does not exist, it will be created.

> fill in usage instructions
### SAPI mode
In SAPI mode, the application migrates data using the Storage API of Keboola Connection.

### Database mode
In Database mode, the application migrates data directly between databases. This mode is significantly faster. Please note the following conditions:
- Replication from the source database must be allowed.
- Both Snowflake accounts must be in the same organization.
- The Snowflake user must have the `ACCOUNTADMIN` role.

For allowing replication, run the following SQL in source Snowflake account:
```sql
ALTER DATABASE {{SOURCE_DATABASE_NAME}} ENABLE REPLICATION TO ACCOUNTS {{DESTINATION_ACCOUNT_REGION}}.{{DESTINATION_ACCOUNT_NAME}};
```

If these conditions are not fulfilled, please use SAPI mode.

## Configuration
The configuration `config.json` contains following properties:

- `parameters` - object (required): Configuration of the application
- `mode` - string (optional): `sapi`(default value) or `database`
- `dryRun` - boolean (optional): If set to `true`, the application will only simulate the migration.
- `sourceKbcUrl` - string (required): URL of the source project
- `sourceKbcToken` - string (required): Storage API token of the source project
- `tables` - array (optional): List of tables to migrate. If not set, all tables will be migrated.
- `db` - object (optional in `database` mode and extends `db` object in `image_parameters`):
- `host` - string (required): Snowflake host
- `username` - string (required): Snowflake username
- `#password` - string (required): Snowflake password
- `warehouse` - string (required): Snowflake warehouse
- `image_parameters` - object (require): Includes from Developer Portal
- `db` - object (required in `database` mode):
- `host` - string (required): Snowflake host
- `username` - string (required): Snowflake username
- `#password` - string (required): Snowflake password
- `warehouse` - string (required): Snowflake warehouse

## Example of Configurations

### DRY run

```json
{
"parameters": {
"mode": "sapi",
"dryRun": true,
"sourceKbcUrl": "https://connection.keboola.com/",
"#sourceKbcToken": "SOURCE_KBC_TOKEN"
}
}
```

### SAPI mode

```json
{
"parameters": {
"mode": "sapi",
"sourceKbcUrl": "https://connection.keboola.com/",
"#sourceKbcToken": "SOURCE_KBC_TOKEN"
}
}
```

### Database mode

```json
{
"parameters": {
"mode": "database",
"sourceKbcUrl": "https://connection.keboola.com/",
"#sourceKbcToken": "SOURCE_KBC_TOKEN"
}
}
```

### Database mode with your own Snowflake

```json
{
"parameters": {
"mode": "database",
"sourceKbcUrl": "https://connection.keboola.com/",
"#sourceKbcToken": "SOURCE_KBC_TOKEN",
"db": {
"host": "SNOWFLAKE_HOST",
"username": "SNOWFLAKE_USERNAME",
"#password": "SNOWFLAKE_PASSWORD",
"warehouse": "SNOWFLAKE_WAREHOUSE"
}
}
}
```

## Development

Clone this repository and init the workspace with following command:

```
git clone https://github.com/keboola/my-component
cd my-component
git clone https://github.com/keboola/app-project-migrate-tables-data
cd app-project-migrate-tables-data
docker-compose build
docker-compose run --rm dev composer install --no-scripts
```

Run the test suite using this command:
Save .env file with following content:

```
docker-compose run --rm dev composer tests
SOURCE_CLIENT_URL=
SOURCE_CLIENT_TOKEN=
DESTINATION_CLIENT_URL=
DESTINATION_CLIENT_TOKEN=
```

# Integration

For information about deployment and integration with KBC, please refer to the [deployment section of developers documentation](https://developers.keboola.com/extend/component/deployment/)
Run the test suite using this command:

```
docker-compose run --rm dev composer tests
```