Skip to content

Remove Snowflake engine #84

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 0 additions & 3 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,4 @@ composer.phar
composer.lock
vendor
set-env.sh
snowflake_linux_x8664_odbc.tgz
snowflake-odbc.deb
snowflake_odbc_generic0.log
.env
24 changes: 0 additions & 24 deletions .travis.yml

This file was deleted.

40 changes: 0 additions & 40 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,58 +3,18 @@ FROM php:7.1
ENV DEBIAN_FRONTEND noninteractive
ENV COMPOSER_ALLOW_SUPERUSER 1
ENV COMPOSER_PROCESS_TIMEOUT 3600
# snowflake - charset settings
ENV LANG en_US.UTF-8

ARG SNOWFLAKE_ODBC_VERSION=2.21.1
ARG SNOWFLAKE_GPG_KEY=EC218558EABB25A1

RUN apt-get update \
&& apt-get install -y unzip \
git \
unixodbc \
unixodbc-dev \
libpq-dev \
gpg \
debsig-verify \
dirmngr \
gpg-agent \
&& rm -r /var/lib/apt/lists/*

RUN echo "memory_limit = -1" >> /usr/local/etc/php/php.ini

RUN docker-php-ext-install pdo_pgsql pdo_mysql

# Snowflake ODBC
# https://github.com/docker-library/php/issues/103#issuecomment-353674490
RUN set -ex; \
docker-php-source extract; \
{ \
echo '# https://github.com/docker-library/php/issues/103#issuecomment-353674490'; \
echo 'AC_DEFUN([PHP_ALWAYS_SHARED],[])dnl'; \
echo; \
cat /usr/src/php/ext/odbc/config.m4; \
} > temp.m4; \
mv temp.m4 /usr/src/php/ext/odbc/config.m4; \
docker-php-ext-configure odbc --with-unixODBC=shared,/usr; \
docker-php-ext-install odbc; \
docker-php-source delete

## install snowflake drivers
COPY ./docker/snowflake/generic.pol /etc/debsig/policies/$SNOWFLAKE_GPG_KEY/generic.pol
ADD https://sfc-repo.snowflakecomputing.com/odbc/linux/$SNOWFLAKE_ODBC_VERSION/snowflake-odbc-$SNOWFLAKE_ODBC_VERSION.x86_64.deb /tmp/snowflake-odbc.deb
COPY ./docker/snowflake/simba.snowflake.ini /usr/lib/snowflake/odbc/lib/simba.snowflake.ini

RUN mkdir -p ~/.gnupg \
&& chmod 700 ~/.gnupg \
&& echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf \
&& mkdir -p /usr/share/debsig/keyrings/$SNOWFLAKE_GPG_KEY \
&& gpg --keyserver hkp://keys.gnupg.net --recv-keys $SNOWFLAKE_GPG_KEY \
&& gpg --export $SNOWFLAKE_GPG_KEY > /usr/share/debsig/keyrings/$SNOWFLAKE_GPG_KEY/debsig.gpg \
&& debsig-verify /tmp/snowflake-odbc.deb \
&& gpg --batch --delete-key --yes $SNOWFLAKE_GPG_KEY \
&& dpkg -i /tmp/snowflake-odbc.deb

RUN cd \
&& curl -sS https://getcomposer.org/installer | php \
&& ln -s /root/composer.phar /usr/local/bin/composer
28 changes: 1 addition & 27 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
# Keboola Database table importer
[![Build Status](https://travis-ci.org/keboola/php-db-import.svg?branch=master)](https://travis-ci.org/keboola/php-db-import)

Handling of large bulk data into database tables.

### Supported engines:
- `AWS Redshift`, `Snowflake`
- `AWS Redshift`
- Load data from CSV stored in AWS S3
- Load data from another Redshift table in same database

Expand All @@ -20,7 +19,6 @@ Handling of large bulk data into database tables.

- Create AWS S3 bucket and IAM user using `aws-services.json` cloudformation template.
- Create Redshift cluster
- Download Snowflake driver (.deb) and place it into root of repository (`./snowflake-odbc.deb`)
- Create `.env` file. Use output of `aws-services` cloudfront stack to fill the variables and your Redshift credentials.
```
REDSHIFT_HOST=
Expand All @@ -29,13 +27,6 @@ REDSHIFT_USER=
REDSHIFT_DATABASE=
REDSHIFT_PASSWORD=

SNOWFLAKE_HOST=
SNOWFLAKE_PORT=
SNOWFLAKE_USER=
SNOWFLAKE_PASSWORD=
SNOWFLAKE_DATABASE=
SNOWFLAKE_WAREHOUSE=

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_S3_BUCKET=
Expand All @@ -55,23 +46,6 @@ CREATE DATABASE keboola_db_import;
GRANT ALL ON DATABASE keboola_db_import TO keboola_db_import;
```


#### Snowflake settings
Role, user, database and warehouse are required for tests. You can create them:
```
CREATE ROLE "KEBOOLA_DB_IMPORT";
CREATE DATABASE "KEBOOLA_DB_IMPORT";
GRANT ALL PRIVILEGES ON DATABASE "KEBOOLA_DB_IMPORT" TO ROLE "KEBOOLA_DB_IMPORT";

CREATE WAREHOUSE "KEBOOLA_DB_IMPORT" WITH WAREHOUSE_SIZE = 'XSMALL' WAREHOUSE_TYPE = 'STANDARD' AUTO_SUSPEND = 3600 AUTO_RESUME = TRUE;
GRANT USAGE ON WAREHOUSE "KEBOOLA_DB_IMPORT" TO ROLE "KEBOOLA_DB_IMPORT" WITH GRANT OPTION;

CREATE USER "KEBOOLA_DB_IMPORT"
PASSWORD = "YOUR_PASSWORD"
DEFAULT_ROLE = "KEBOOLA_DB_IMPORT";

GRANT ROLE "KEBOOLA_DB_IMPORT" TO USER "KEBOOLA_DB_IMPORT";
```
#### Tests Execution
Run tests with following command.

Expand Down
4 changes: 1 addition & 3 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,13 @@
"tests-load-s3": "php ./tests/loadS3.php",
"tests-redshift": "phpunit tests/Redshift",
"tests-redshift-legacy": "export REDSHIFT_LEGACY_IMPORT=1 && phpunit tests/Redshift",
"tests-snowflake": "phpunit tests/Snowflake",
"build": [
"@composer install",
"@phpcs",
"@phpstan",
"@tests-load-s3",
"@tests-redshift",
"@tests-redshift-legacy",
"@tests-snowflake"
"@tests-redshift-legacy"
],
"ci": [
"@composer validate --strict",
Expand Down
6 changes: 0 additions & 6 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,6 @@ services:
- REDSHIFT_DATABASE
- REDSHIFT_PASSWORD
- REDSHIFT_LEGACY_IMPORT
- SNOWFLAKE_HOST
- SNOWFLAKE_PORT
- SNOWFLAKE_USER
- SNOWFLAKE_PASSWORD
- SNOWFLAKE_DATABASE
- SNOWFLAKE_WAREHOUSE
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_S3_BUCKET
Expand Down
11 changes: 0 additions & 11 deletions docker/snowflake/generic.pol

This file was deleted.

9 changes: 0 additions & 9 deletions docker/snowflake/simba.snowflake.ini

This file was deleted.

1 change: 0 additions & 1 deletion phpstan.neon
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
parameters:
ignoreErrors:
- '#Property Keboola\\DbImportTest\\Snowflake\\ImportTest::\$connection \(Keboola\\Db\\Import\\Snowflake\\Connection\) does not accept null.#'
45 changes: 0 additions & 45 deletions src/MessageTransformation.php

This file was deleted.

Loading