Skip to content

Commit

Permalink
Rename destination to Redshift No DBT
Browse files Browse the repository at this point in the history
  • Loading branch information
mohamagdy committed Apr 8, 2022
1 parent 59203b7 commit d3594c4
Show file tree
Hide file tree
Showing 22 changed files with 55 additions and 55 deletions.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
*
!Dockerfile
!main.py
!destination_redshift_py
!destination_redshift_no_dbt
!setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ RUN apk --no-cache add bash \

# copy payload code only
COPY main.py ./
COPY destination_redshift_py ./destination_redshift_py
COPY destination_redshift_no_dbt ./destination_redshift_no_dbt

ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.0
LABEL io.airbyte.name=airbyte/destination-redshift-py
LABEL io.airbyte.name=airbyte/destination-redshift-no-dbt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Redshift Py Destination
# Redshift No Dbt Destination

This is the repository for the Redshift Py destination connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/destinations/redshift-py).
This is the repository for the Redshift No Dbt destination connector, written in Python.
For information about how to use this connector within Airbyte, see [the documentation](https://docs.airbyte.io/integrations/destinations/redshift-no-dbt).

## Local development

Expand Down Expand Up @@ -32,16 +32,16 @@ should work as you expect.
#### Building via Gradle
From the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:destination-redshift-py:build
./gradlew :airbyte-integrations:connectors:destination-redshift-no-dbt:build
```

#### Create credentials
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/destinations/redshift-py)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `destination_redshift_py/spec.json` file.
**If you are a community contributor**, follow the instructions in the [documentation](https://docs.airbyte.io/integrations/destinations/redshift-no-dbt)
to generate the necessary credentials. Then create a file `secrets/config.json` conforming to the `destination_redshift_no_dbt/spec.json` file.
Note that the `secrets` directory is gitignored by default, so there is no danger of accidentally checking in sensitive information.
See `integration_tests/sample_config.json` for a sample config file.

**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `destination redshift-py test creds`
**If you are an Airbyte core member**, copy the credentials in Lastpass under the secret name `destination redshift-no-dbt test creds`
and place them into `secrets/config.json`.

### Locally running the connector
Expand All @@ -57,23 +57,23 @@ python main.py read --config secrets/config.json --catalog integration_tests/con
#### Build
First, make sure you build the latest Docker image:
```
docker build . -t airbyte/destination-redshift-py:dev
docker build . -t airbyte/destination-redshift-no-dbt:dev
```

You can also build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:destination-redshift-py:airbyteDocker
./gradlew :airbyte-integrations:connectors:destination-redshift-no-dbt:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/destination-redshift-py:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-redshift-py:dev check --config /secrets/config.json
docker run --rm airbyte/destination-redshift-no-dbt:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-redshift-no-dbt:dev check --config /secrets/config.json
# messages.jsonl is a file containing line-separated JSON representing AirbyteMessages
cat messages.jsonl | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-redshift-py:dev write --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
cat messages.jsonl | docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-redshift-no-dbt:dev write --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```
## Testing
Make sure to familiarize yourself with [pytest test discovery](https://docs.pytest.org/en/latest/goodpractices.html#test-discovery) to know how your test files and methods should be named.
Expand Down Expand Up @@ -101,11 +101,11 @@ Coming soon:
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:destination-redshift-py:unitTest
./gradlew :airbyte-integrations:connectors:destination-redshift-no-dbt:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:destination-redshift-py:integrationTest
./gradlew :airbyte-integrations:connectors:destination-redshift-no-dbt:integrationTest
```

## Dependency Management
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ plugins {
}

airbytePython {
moduleDirectory 'destination_redshift_py'
moduleDirectory 'destination_redshift_no_dbt'
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
#
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
#


from .destination import DestinationRedshiftNoDbt

__all__ = ["DestinationRedshiftNoDbt"]
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

from dotmap import DotMap

from destination_redshift_py.table import Table
from destination_redshift_no_dbt.table import Table

CSV_EXTENSION = "csv"
GZIP_EXTENSION = "gzip"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
from typing import List, Union

from destination_redshift_py.field import DataType
from destination_redshift_no_dbt.field import DataType

VARCHAR = "VARCHAR"
TIMESTAMP_WITHOUT_TIME_ZONE = "TIMESTAMP WITHOUT TIME ZONE"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,16 @@
from psycopg2._psycopg import connection as Connection
from psycopg2.pool import ThreadedConnectionPool, SimpleConnectionPool

from destination_redshift_py.csv_writer import CSVWriter
from destination_redshift_py.jsonschema_to_tables import JsonToTables, PARENT_CHILD_SPLITTER
from destination_redshift_py.s3_objects_manager import S3ObjectsManager
from destination_redshift_py.stream import Stream
from destination_redshift_py.table import AIRBYTE_EMITTED_AT, AIRBYTE_ID_NAME, Table
from destination_redshift_no_dbt.csv_writer import CSVWriter
from destination_redshift_no_dbt.jsonschema_to_tables import JsonToTables, PARENT_CHILD_SPLITTER
from destination_redshift_no_dbt.s3_objects_manager import S3ObjectsManager
from destination_redshift_no_dbt.stream import Stream
from destination_redshift_no_dbt.table import AIRBYTE_EMITTED_AT, AIRBYTE_ID_NAME, Table

logger = logging.getLogger("airbyte")


class DestinationRedshiftPy(Destination):
class DestinationRedshiftNoDbt(Destination):
def __init__(self):
self.streams: Dict[str, Stream] = dict()

Expand Down Expand Up @@ -157,7 +157,7 @@ def get_records(parents: Union[DotMap, List[DotMap]], method: str) -> List[DotMa
@staticmethod
def _assign_id_and_emitted_at(records: List[DotMap], emitted_at: int, hashing_keys: List[str]):
for record in records:
DestinationRedshiftPy._assign_id(record=record, hashing_keys=hashing_keys)
DestinationRedshiftNoDbt._assign_id(record=record, hashing_keys=hashing_keys)
record[AIRBYTE_EMITTED_AT.name] = datetime.utcfromtimestamp(emitted_at / 1000).isoformat(timespec="seconds")

@staticmethod
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
from collections import OrderedDict
from typing import Optional, List

from destination_redshift_py.data_type_converter import DataTypeConverter, FALLBACK_DATATYPE
from destination_redshift_py.field import Field
from destination_redshift_py.table import Table
from destination_redshift_no_dbt.data_type_converter import DataTypeConverter, FALLBACK_DATATYPE
from destination_redshift_no_dbt.field import Field
from destination_redshift_no_dbt.table import Table

PARENT_CHILD_SPLITTER = "."

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/redshift-py",
"documentationUrl": "https://docs.airbyte.io/integrations/destinations/redshift-no-dbt",
"supported_destination_sync_modes": ["overwrite", "append", "append_dedup"],
"supportsIncremental": true,
"supportsDBT": false,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from typing import Dict

from airbyte_cdk.models import DestinationSyncMode
from destination_redshift_py.table import Table
from destination_redshift_no_dbt.table import Table
from pydantic.main import BaseModel


Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from typing import List, Optional

from destination_redshift_py.data_type_converter import VARCHAR, TIMESTAMP_WITHOUT_TIME_ZONE
from destination_redshift_py.field import Field, DataType
from destination_redshift_no_dbt.data_type_converter import VARCHAR, TIMESTAMP_WITHOUT_TIME_ZONE
from destination_redshift_no_dbt.field import Field, DataType

AIRBYTE_ID_NAME = "_airbyte_ab_id"
AIRBYTE_EMITTED_AT_NAME = "_airbyte_emitted_at"
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
#
# Copyright (c) 2021 Airbyte, Inc., all rights reserved.
#


import sys

from destination_redshift_no_dbt import DestinationRedshiftNoDbt

if __name__ == "__main__":
DestinationRedshiftNoDbt().run(sys.argv[1:])
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
]

setup(
name="destination_redshift_py",
description="Destination implementation for Redshift Py.",
name="destination_redshift_no_dbt",
description="Destination implementation for Redshift No Dbt.",
author="Airbyte",
author_email="contact@airbyte.io",
packages=find_packages(),
Expand Down

This file was deleted.

11 changes: 0 additions & 11 deletions airbyte-integrations/connectors/destination-redshift-py/main.py

This file was deleted.

0 comments on commit d3594c4

Please sign in to comment.