Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate SQL templates #1

Merged
merged 38 commits into from
Oct 24, 2022
Merged

Generate SQL templates #1

Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
3fd394e
add generate command with parameters
martinjunger Sep 23, 2022
12fa7ab
add db-import-export-lib
martinjunger Sep 23, 2022
d4aba15
update readme
martinjunger Sep 23, 2022
bf38ba9
refactor action to separate class
martinjunger Oct 17, 2022
970cc18
move constants to Config
martinjunger Oct 17, 2022
6e752db
rename namespace
martinjunger Oct 18, 2022
2f6b316
add more options to config
martinjunger Oct 18, 2022
30ecb51
require phpunit
martinjunger Oct 18, 2022
c2f5d92
rename operations
martinjunger Oct 18, 2022
e04d91f
add tests for ConfigDefinition
martinjunger Oct 18, 2022
0e9fb6f
add Utils with replace methods and uniqid
martinjunger Oct 18, 2022
d340da2
add Snowflake/ImportFull/FromAbsGenerator
martinjunger Oct 18, 2022
e9bbd8a
use generator in action
martinjunger Oct 18, 2022
c6f63b7
replace supports other quoters
martinjunger Oct 18, 2022
70ae3c4
use given columns and primaryKeys
martinjunger Oct 18, 2022
8391cc9
add Synapse/ImportFull/FromAbsGenerator
martinjunger Oct 18, 2022
7529154
add Synapse/ImportFull/FromTableGenerator
martinjunger Oct 19, 2022
2547abd
replace Synapse distribution and index methods
martinjunger Oct 19, 2022
e35f9cf
remove create final table
martinjunger Oct 19, 2022
9ac45a8
update names of operation parameter
martinjunger Oct 19, 2022
612ae36
reformat output queries
martinjunger Oct 19, 2022
8a81f0b
change replace identifier for prefix
martinjunger Oct 19, 2022
22ef58b
add replace identifier for suffix
martinjunger Oct 19, 2022
c6fab4e
replace Synapse SECRET prefixed value
martinjunger Oct 19, 2022
a4fe6c0
replace dedup tables usign Twig syntax
martinjunger Oct 19, 2022
8972790
update composer
martinjunger Oct 19, 2022
15d6752
fix cs
martinjunger Oct 19, 2022
b9d1d1b
fix stan
martinjunger Oct 19, 2022
d24933d
update readme
martinjunger Oct 19, 2022
8846d00
fix rename backendType to backend
martinjunger Oct 19, 2022
1511c13
add component test
martinjunger Oct 19, 2022
c1fb666
fix cs
martinjunger Oct 19, 2022
d0a1eda
add Synapse/ImportFull/FromTableGenerator test
martinjunger Oct 20, 2022
1e984ae
add GeneratorFactory and test for GenerateAction
martinjunger Oct 20, 2022
7a4dee3
add more tests for component
martinjunger Oct 20, 2022
fe32ecb
change Synapse identity to Managed
martinjunger Oct 21, 2022
5189c92
ignore ext-odbc
martinjunger Oct 24, 2022
444f00d
remove debug dumps
martinjunger Oct 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM php:8-cli
FROM php:7.4-cli
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

proč 7.4 jestli kvůli csv-options tak bysme měli vydat novů verzi nebo by šlo ignorovat ten requirement

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zatim to nejspis nepujde... Je tam totiz zavislost:

  • php-db-import-export - php8 ready
    • php-csv-options - php8 ready
    • php-csv-db-import - ^7.1

Muzeme to forknout a upravit (nejaky takovy forky uz jsou)...


ARG COMPOSER_FLAGS="--prefer-dist --no-interaction"
ARG DEBIAN_FRONTEND=noninteractive
Expand All @@ -14,10 +14,22 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
git \
locales \
unzip \
unixodbc-dev \
&& rm -r /var/lib/apt/lists/* \
&& sed -i 's/^# *\(en_US.UTF-8\)/\1/' /etc/locale.gen \
&& locale-gen \
&& chmod +x /tmp/composer-install.sh \
&& locale-gen

# ODBC
RUN set -x \
&& docker-php-source extract \
&& cd /usr/src/php/ext/odbc \
&& phpize \
&& sed -ri 's@^ *test +"\$PHP_.*" *= *"no" *&& *PHP_.*=yes *$@#&@g' configure \
&& ./configure --with-unixODBC=shared,/usr \
&& docker-php-ext-install odbc \
&& docker-php-source delete
martinjunger marked this conversation as resolved.
Show resolved Hide resolved

RUN chmod +x /tmp/composer-install.sh \
&& /tmp/composer-install.sh

ENV LANGUAGE=en_US.UTF-8
Expand Down
43 changes: 38 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,51 @@
# my-component
# Custom Query manager component

> Fill in description
Generates templates for Custom Query feature in Keboola Connection platform.

# Usage

> fill in usage instructions
Create configuration in `data/config.json` with:
- `parameters`:
- `backend`: one of supported backend: `synapse`, `snowflake`; required
- `operation`: one of supported operation: `importFull`, `importIncremental`; required
- `source`: one of supported operation: `table`, `fileAbs`; required
- `columns`: list of columns; required
- `primaryKeys`: list of primary keys; optional
- `action`: one of supported sync action to be run: `generate`; required

for example:
```json
{
"parameters": {
"backend": "synapse",
"operation": "importFull",
"source": "fileAbs",
"columns": [
"column1",
"column2"
],
"primaryKeys": [
"column1"
]
},
"action": "generate"
}
```

Run component:
```shell
docker-compose run --rm dev
```

Will return JSON with query templates.

## Development

Clone this repository and init the workspace with following command:

```
git clone https://github.com/keboola/my-component
cd my-component
git clone https://github.com/keboola/component.app-custom-query-manager
cd component.app-custom-query-manager
docker-compose build
docker-compose run --rm dev composer install --no-scripts
```
Expand Down
17 changes: 9 additions & 8 deletions composer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,29 +1,30 @@
{
"name": "keboola/my-component",
"description": "MyComponent description",
"name": "keboola/app-custom-query-manager",
"description": "Application to generate SQL queries",
"license": "MIT",
"require": {
"php": "^8.1",
"keboola/php-component": "^9.0"
"php": "^7.4",
"keboola/db-import-export": "^1.10",
"keboola/php-component": "^9.0",
"phpunit/phpunit": "^9.5"
},
"require-dev": {
"php-parallel-lint/php-parallel-lint": "^1.3",
"keboola/coding-standard": ">=7.0.2",
"keboola/datadir-tests": "^5.3",
"keboola/php-temp": "^2.0",
"phpstan/phpstan": "^1.4",
"phpunit/phpunit": "^9.5",
"symfony/process": "^5.0"
},
"autoload": {
"psr-4": {
"MyComponent\\": "src/"
"Keboola\\CustomQueryManagerApp\\": "src/"
}
},
"autoload-dev": {
"psr-4": {
"MyComponent\\Tests\\": "tests/phpunit/",
"MyComponent\\FunctionalTests\\": "tests/functional/"
"Keboola\\CustomQueryManagerApp\\Tests\\": "tests/phpunit/",
"Keboola\\CustomQueryManagerApp\\FunctionalTests\\": "tests/functional/"
}
},
"scripts": {
Expand Down
Loading