Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Source File: Fix special characters bug #21012

Merged
merged 6 commits into from
Jan 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -489,7 +489,7 @@
- name: File
sourceDefinitionId: 778daa7c-feaf-4db6-96f3-70fd645acc77
dockerRepository: airbyte/source-file
dockerImageTag: 0.2.32
dockerImageTag: 0.2.33
documentationUrl: https://docs.airbyte.com/integrations/sources/file
icon: file.svg
sourceType: file
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4041,7 +4041,7 @@
supportsNormalization: false
supportsDBT: false
supported_destination_sync_modes: []
- dockerImage: "airbyte/source-file:0.2.32"
- dockerImage: "airbyte/source-file:0.2.33"
spec:
documentationUrl: "https://docs.airbyte.com/integrations/sources/file"
connectionSpecification:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,5 @@ RUN pip install .
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.32
LABEL io.airbyte.version=0.2.33
LABEL io.airbyte.name=airbyte/source-file-secure
2 changes: 1 addition & 1 deletion airbyte-integrations/connectors/source-file/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ COPY source_file ./source_file
ENV AIRBYTE_ENTRYPOINT "python /airbyte/integration_code/main.py"
ENTRYPOINT ["python", "/airbyte/integration_code/main.py"]

LABEL io.airbyte.version=0.2.32
LABEL io.airbyte.version=0.2.33
LABEL io.airbyte.name=airbyte/source-file
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ def is_ssh_ready(ip, port):
ip,
port=port,
username="user1",
password="pass1",
password="abc123@456#",
)
return True
except (SSHException, socket.error):
Expand All @@ -93,9 +93,9 @@ def ssh_service(docker_ip, docker_services):
def provider_config(ssh_service):
def lookup(name):
providers = {
"ssh": dict(storage="SSH", host=ssh_service, user="user1", password="pass1", port=2222),
"scp": dict(storage="SCP", host=ssh_service, user="user1", password="pass1", port=2222),
"sftp": dict(storage="SFTP", host=ssh_service, user="user1", password="pass1", port=100),
"ssh": dict(storage="SSH", host=ssh_service, user="user1", password="abc123@456#", port=2222),
"scp": dict(storage="SCP", host=ssh_service, user="user1", password="abc123@456#", port=2222),
"sftp": dict(storage="SFTP", host=ssh_service, user="user1", password="abc123@456#", port=100),
"gcs": dict(storage="GCS"),
"s3": dict(storage="S3"),
"azure": dict(storage="AzBlob"),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@ services:
- "2222:22"
volumes:
- ./sample_files:/home/user1/files
command: user1:pass1:1001
command: user1:abc123@456#:1001
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import json
import tempfile
import traceback
import urllib
from os import environ
from typing import Iterable
from urllib.parse import urlparse
Expand Down Expand Up @@ -108,8 +109,11 @@ def _open(self):
port = self._provider["port"]
return smart_open.open(f"webhdfs://{host}:{port}/{url}", **self.args)
elif storage in ("ssh://", "scp://", "sftp://"):
user = self._provider["user"]
host = self._provider["host"]
# We need to quote parameters to deal with special characters
# https://bugs.python.org/issue18140
user = urllib.parse.quote(self._provider["user"])
host = urllib.parse.quote(self._provider["host"])
url = urllib.parse.quote(url)
# TODO: Remove int casting when https://github.com/airbytehq/airbyte/issues/4952 is addressed
# TODO: The "port" field in spec.json must also be changed
_port_value = self._provider.get("port", 22)
Expand All @@ -120,7 +124,7 @@ def _open(self):
# Explicitly turn off ssh keys stored in ~/.ssh
transport_params = {"connect_kwargs": {"look_for_keys": False}, "timeout": SSH_TIMEOUT}
if "password" in self._provider:
password = self._provider["password"]
password = urllib.parse.quote(self._provider["password"])
uri = f"{storage}{user}:{password}@{host}:{port}/{url}"
else:
uri = f"{storage}{user}@{host}:{port}/{url}"
Expand Down
3 changes: 2 additions & 1 deletion docs/integrations/sources/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,8 @@ In order to read large files from a remote location, this connector uses the [sm
## Changelog

| Version | Date | Pull Request | Subject |
|:--------|:-----------| :------------------------------------------------------- |:---------------------------------------------------------|
|:--------|:-----------|:---------------------------------------------------------|:---------------------------------------------------------|
| 0.2.33 | 2023-01-04 | [21012](https://github.com/airbytehq/airbyte/pull/21012) | Fix special characters bug |
| 0.2.32 | 2022-12-21 | [20740](https://github.com/airbytehq/airbyte/pull/20740) | Source File: increase SSH timeout to 60s |
| 0.2.31 | 2022-11-17 | [19567](https://github.com/airbytehq/airbyte/pull/19567) | Source File: bump 0.2.31 |
| 0.2.30 | 2022-11-10 | [19222](https://github.com/airbytehq/airbyte/pull/19222) | Use AirbyteConnectionStatus for "check" command |
Expand Down