Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[non-forked] 🐛 Destination Databend: upgrade databend-sqlalchemy dependency #24399

Merged
merged 11 commits into from
Mar 27, 2023
Original file line number Diff line number Diff line change
Expand Up @@ -434,7 +434,7 @@
- name: Databend
destinationDefinitionId: 302e4d8e-08d3-4098-acd4-ac67ca365b88
dockerRepository: airbyte/destination-databend
dockerImageTag: 0.1.1
dockerImageTag: 0.1.2
icon: databend.svg
documentationUrl: https://docs.airbyte.com/integrations/destinations/databend
releaseStage: alpha
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7260,7 +7260,7 @@
supported_destination_sync_modes:
- "overwrite"
- "append"
- dockerImage: "airbyte/destination-databend:0.1.1"
- dockerImage: "airbyte/destination-databend:0.1.2"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/destinations/databend"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,5 +34,5 @@ COPY destination_databend ./destination_databend
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.1.1
LABEL io.airbyte.version=0.1.2
LABEL io.airbyte.name=airbyte/destination-databend
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,6 @@ def __init__(self, host: str, port: int, database: str, table: str, username: st
self.password = password

def open(self):
handle = connector.connect(f"https://{self.username}:{self.password}@{self.host}:{self.port}").cursor()
handle = connector.connect(f"https://{self.username}:{self.password}@{self.host}:{self.port}/{self.database}").cursor()

return handle
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
from destination_databend.client import DatabendClient


@pytest.fixture(name="config")
@pytest.fixture(name="databendConfig")
def config_fixture() -> Mapping[str, Any]:
with open("secrets/config.json", "r") as f:
return json.loads(f.read())
Expand All @@ -49,20 +49,20 @@ def configured_catalog_fixture() -> ConfiguredAirbyteCatalog:


@pytest.fixture(autouse=True)
def teardown(config: Mapping):
def teardown(databendConfig: Mapping):
yield
client = DatabendClient(**config)
client = DatabendClient(**databendConfig)
cursor = client.open()
cursor.close()


@pytest.fixture(name="client")
def client_fixture(config) -> DatabendClient:
return DatabendClient(**config)
def client_fixture(databendConfig) -> DatabendClient:
return DatabendClient(**databendConfig)


def test_check_valid_config(config: Mapping):
outcome = DestinationDatabend().check(logging.getLogger('airbyte'), config)
def test_check_valid_config(databendConfig: Mapping):
outcome = DestinationDatabend().check(logging.getLogger('airbyte'), databendConfig)
assert outcome.status == Status.SUCCEEDED


Expand Down Expand Up @@ -103,7 +103,7 @@ def retrieve_all_records(client: DatabendClient) -> List[AirbyteRecordMessage]:
return overwrite_out + append_out


def test_write(config: Mapping, configured_catalog: ConfiguredAirbyteCatalog, client: DatabendClient):
def test_write(databendConfig: Mapping, configured_catalog: ConfiguredAirbyteCatalog, client: DatabendClient):
"""
This test verifies that:
1. writing a stream in "overwrite" mode overwrites any existing data for that stream
Expand All @@ -124,7 +124,7 @@ def test_write(config: Mapping, configured_catalog: ConfiguredAirbyteCatalog, cl
expected_states = [first_state_message, second_state_message]
output_states = list(
destination.write(
config, configured_catalog, [*first_record_chunk, first_state_message, *second_record_chunk, second_state_message]
databendConfig, configured_catalog, [*first_record_chunk, first_state_message, *second_record_chunk, second_state_message]
)
)
assert expected_states == output_states, "Checkpoint state messages were expected from the destination"
Expand All @@ -139,7 +139,7 @@ def test_write(config: Mapping, configured_catalog: ConfiguredAirbyteCatalog, cl
_record(overwrite_stream, str(i), i) for i in range(10, 15)
]

output_states = list(destination.write(config, configured_catalog, [*third_record_chunk, third_state_message]))
output_states = list(destination.write(databendConfig, configured_catalog, [*third_record_chunk, third_state_message]))
assert [third_state_message] == output_states

records_in_destination = retrieve_all_records(client)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

from setuptools import find_packages, setup

MAIN_REQUIREMENTS = ["airbyte-cdk", "requests", "databend-sqlalchemy"]
MAIN_REQUIREMENTS = ["airbyte-cdk", "requests", "databend-sqlalchemy==0.1.6"]

TEST_REQUIREMENTS = ["pytest~=6.1"]
setup(
Expand Down
2 changes: 1 addition & 1 deletion connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@
| **Clickhouse** | <img alt="Clickhouse icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/clickhouse.svg" height="30" height="30"/> | Destination | airbyte/destination-clickhouse:0.2.2 | alpha | [link](https://docs.airbyte.com/integrations/destinations/clickhouse) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-clickhouse) | <small>`ce0d828e-1dc4-496c-b122-2da42e637e48`</small> |
| **Cloudflare R2** | <img alt="Cloudflare R2 icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/cloudflare-r2.svg" height="30" height="30"/> | Destination | airbyte/destination-r2:0.1.0 | alpha | [link](https://docs.airbyte.com/integrations/destinations/r2) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-r2) | <small>`0fb07be9-7c3b-4336-850d-5efc006152ee`</small> |
| **Convex** | <img alt="Convex icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/convex.svg" height="30" height="30"/> | Destination | airbyte/destination-convex:0.1.0 | alpha | [link](https://docs.airbyte.io/integrations/destinations/convex) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-convex) | <small>`3eb4d99c-11fa-4561-a259-fc88e0c2f8f4`</small> |
| **Databend** | <img alt="Databend icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/databend.svg" height="30" height="30"/> | Destination | airbyte/destination-databend:0.1.1 | alpha | [link](https://docs.airbyte.com/integrations/destinations/databend) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-databend) | <small>`302e4d8e-08d3-4098-acd4-ac67ca365b88`</small> |
| **Databend** | <img alt="Databend icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/databend.svg" height="30" height="30"/> | Destination | airbyte/destination-databend:0.1.2 | alpha | [link](https://docs.airbyte.com/integrations/destinations/databend) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-databend) | <small>`302e4d8e-08d3-4098-acd4-ac67ca365b88`</small> |
| **Databricks Lakehouse** | <img alt="Databricks Lakehouse icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/databricks.svg" height="30" height="30"/> | Destination | airbyte/destination-databricks:1.0.0 | alpha | [link](https://docs.airbyte.com/integrations/destinations/databricks) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-databricks) | <small>`072d5540-f236-4294-ba7c-ade8fd918496`</small> |
| **DuckDB** | <img alt="DuckDB icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/duckdb.svg" height="30" height="30"/> | Destination | airbyte/destination-duckdb:0.1.0 | alpha | [link](https://docs.airbyte.io/integrations/destinations/duckdb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-duckdb) | <small>`94bd199c-2ff0-4aa2-b98e-17f0acb72610`</small> |
| **DynamoDB** | <img alt="DynamoDB icon" src="https://raw.githubusercontent.com/airbytehq/airbyte/master/airbyte-config/init/src/main/resources/icons/dynamodb.svg" height="30" height="30"/> | Destination | airbyte/destination-dynamodb:0.1.7 | alpha | [link](https://docs.airbyte.com/integrations/destinations/dynamodb) | [code](https://github.com/airbytehq/airbyte/tree/master/airbyte-integrations/connectors/destination-dynamodb) | <small>`8ccd8909-4e99-4141-b48d-4984b70b2d89`</small> |
Expand Down
11 changes: 7 additions & 4 deletions docs/integrations/destinations/databend.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,11 +51,14 @@ You should now have all the requirements needed to configure Databend as a desti
* **Password**
* **Database**

## Compatibility
If your databend version >= v0.9.0 or later, you need to use databend-sqlalchemy version >= v0.1.0. And the [Databend Cloud](https://app.databend.com/) will only support databend version > 0.9.0.

## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:-----------------------------------------|
| 0.1.1 | 2022-01-09 | [21182](https://github.com/airbytehq/airbyte/pull/21182) | Remove protocol option and enforce HTTPS |
| 0.1.0 | 2022-01-09 | [20909](https://github.com/airbytehq/airbyte/pull/20909) | Destination Databend |
| Version | Date | Pull Request | Subject |
|:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------|
| 0.1.2 | 2023-02-11 | [22855](https://github.com/airbytehq/airbyte/pull/22855) | Fix compatibility issue with databend-query 0.9 |
| 0.1.1 | 2022-01-09 | [21182](https://github.com/airbytehq/airbyte/pull/21182) | Remove protocol option and enforce HTTPS |
| 0.1.0 | 2022-01-09 | [20909](https://github.com/airbytehq/airbyte/pull/20909) | Destination Databend |